GenBank Release 226.0 Available : June 20 2018

[Cavanaugh, Mark (NIH/NLM/NCBI) [E]]

Greetings GenBank Users,

GenBank Release 226.0 is now available via FTP from the National Center
for Biotechnology Information (NCBI):

Ftp Site Directory Contents
---------------- --------- ---------------------------------------
ftp.ncbi.nih.gov genbank GenBank Release 226.0 flatfiles
ncbi-asn1 ASN.1 data used to create Release 226.0

Close-of-data for GenBank 226.0 occurred on 06/17/2018. Uncompressed,
the Release 226.0 flatfiles require roughly 894 GB (sequence files only).
The ASN.1 data require approximately 735 GB.

Recent statistics for 'traditional' sequences (including non-bulk-oriented
TSA, and excluding WGS, bulk-oriented TSA, TLS, and the CON-division):

Release Date Base Pairs Entries

225 Apr 2018 260189141631 208452303
226 Jun 2018 263957884539 209775348

Recent statistics for WGS sequencing projects:

Release Date Base Pairs Entries

225 Apr 2018 2784740996536 621379029
226 Jun 2018 2944617324086 639804105

Recent statistics for bulk-oriented TSA sequencing projects:

Release Date Base Pairs Entries

225 Apr 2018 205232396043 227364990
226 Jun 2018 216556686631 238788334

Recent statistics for bulk-oriented TLS sequencing projects:

Release Date Base Pairs Entries

225 Apr 2018 5612769448 14782654
226 Jun 2018 5896511468 15393041

During the 64 days between the close dates for GenBank Releases 225.0
and 226.0, the 'traditional' portion of GenBank grew by 3,768,742,908
basepairs and by 1,323,045 sequence records. During that same period,
874,605 records were updated. An average of 34,338 'traditional' records
were added and/or updated per day.

Between releases 225.0 and 226.0, the WGS component of GenBank grew by
159,876,327,550 basepairs and by 18,425,076 sequence records.

Between releases 225.0 and 226.0, the TSA component of GenBank grew by
11,324,290,588 basepairs and by 11,423,344 sequence records.

Between releases 225.0 and 226.0, the TLS component of GenBank grew by
283,742,020 basepairs and by 610,387 sequence records.

For additional release information, see the README files in either of
the directories mentioned above, and the release notes (gbrel.txt) in
the genbank directory. Sections 1.3 and 1.4 of the release notes
(Changes in Release 226.0 and Upcoming Changes) have been appended
below for your convenience.

* * * Important Note * * *

Section 1.4.1 of the GenBank release notes describes future accession
format changes for traditional nucleotide sequences, WGS/TSA/TLS
sequences, and for protein sequences. These important changes are likely
to be of interest to many GenBank users, and we encourage a review of
the section.

Release 226.0 data, and subsequent updates, are available now via
NCBI's Entrez and Blast services.

As a general guideline, we suggest first transferring the GenBank
release notes (gbrel.txt) whenever a release is being obtained. Check
to make sure that the date and release number in the header of the
release notes are current (eg: June 15 2018, 226.0). If they are
not, interrupt the remaining transfers and then request assistance from
the NCBI Service Desk.

A comprehensive check of the headers of all release files after your
transfers are complete is also suggested. Here's how one might go about
this on a Unix or Linux platform, using csh/tcsh :

set files = `ls gb*.*`
foreach i ($files)
head -10 $i | grep Release
end

Or, if the files are compressed, perhaps:

gzcat $i | head -10 | grep Release

If you encounter problems while ftp'ing or uncompressing Release
226.0, please send email outlining your difficulties to:

in...@ncbi.nlm.nih.gov

Mark Cavanaugh, Michael Kimelman, Ilya Dondoshansky
GenBank
NCBI/NLM/NIH/HHS


1.3 Important Changes in Release 226.0

1.3.1 Organizational changes

The total number of sequence data files increased by 46 with this release:

- the BCT division is now composed of 492 files (+18)
- the ENV division is now composed of 103 files (+1)
- the GSS division is now composed of 306 files (+2)
- the INV division is now composed of 168 files (+5)
- the PAT division is now composed of 335 files (+5)
- the PHG division is now composed of 5 files (+1)
- the PLN division is now composed of 174 files (+2)
- the ROD division is now composed of 31 files (+1)
- the VRL division is now composed of 56 files (+2)
- the VRT division is now composed of 94 files (+9)

1.3.2 New source feature qualifier : /submitter_seqid

Data submitters typically have their own identifiers for genomic contigs and
scaffolds of Whole Genome Shotgun (WGS) sequencing projects, the RNA sequences
of Transcriptome Shotgun Assembly (TSA) sequencing projects, and the genomic
loci of Targeted Locus Study (TLS) sequencing projects. These identifiers can
be very simple (contig01, contig02, etc.), or they can have a bit more
meaning/structure. Examples of the latter include:

gcontig_1106166512749 (ABDU01000001)
CCB157_001 (BDDQ01000001)
Lo7_v2_contig_2306 (CCJQ010001199)

The INSDC has decided that it would be helpful to provide these identifiers
in a formalized way, since they may be known to, or used by, parties other than
the submitters themselves. For example, if a submitter had made them public
in some sort of data products, or displayed in genome browsers, cited in an
analysis, or mentioned on websites.

The definition for this new qualifier is as follows:

Qualifier /submitter_seqid=
Definition identifier attributed to each sequence within an assembly.
This identifier is appropriate for WGS, TSA, TLS and CON records.
The submitter_seqid should be unique within the context of a
single set of assembled sequences.
Value format "text"
Example /submitter_seqid="NODE_1"
Comment The length of the value should be limited to <51 charaters.
Spaces, greater than (>), left/right square brackets ([ ])
and vertical bar (|) in addition to double quotation marks
(") can not be used for the value of /submitter_seqid qualifier.

The new qualifier became legal for sequence records as of April 15 2018.
However, GenBank is still in the process of modifying its software to
accommodate the new qualifier, and it may be several more weeks before
it appears within sequence records.

1.3.3 New /gap_type value : "contamination"

When contamination is discovered in a sequence record, removing the
bases from the sequence data can be problematic (especially at the 5' end)
because the length of the sequence changes. If there exist higher-level
scaffold/CON-division records (possibly chromosomes), the resulting change
in length requires an adjustment to the coordinate system of the scaffold/
chromosome, and the features annotated on it. The impact of such a change
on both data submitters and users can impose quite a burden.

To address this, the INSDC has decided to introduce a new Gap Type for
the assembly_gap feature : contamination . When sequence contamination
is discovered, the submitter will have the option of replacing the affected
base pairs with Ns, via a terminal assembly_gap feature. For example:

assembly_gap 1..2956
/estimated_length=2956
/gap_type="contamination"
/note="contamination masked with Ns"

The updated definition of the /gap_type qualifier is as follows:

Qualifier /gap_type=
Definition type of gap connecting components in records of a genome assembly,
or the type of biological gap in a record that is part of a genome
assembly;
Value format "between scaffolds", "within scaffold", "telomere", "centromere",
"short arm", "heterochromatin", "repeat within scaffold",
"repeat between scaffolds", "contamination", "unknown"
Example /gap_type="between scaffolds"
/gap_type="within scaffold"
Comment This qualifier is used only for assembly_gap features and its values
are controlled by the AGP Specification version 2.0:
https://www.ncbi.nlm.nih.gov/assembly/agp/AGP_Specification/
Please also visit: http://www.insdc.org/controlled-vocabulary-gaptype-qualifier

The new value for /gap_type became legal for sequence records as of
April 15 2018. However, GenBank is still in the process of modifying
its software to accommodate "contamination" gaps, and it may be several
more weeks before it appears within sequence records.

1.3.4 GSS File Header Problem

GSS sequences at GenBank are maintained in two different systems, depending
on their origin, and the dumps from those systems occur in parallel. Because
the second dump (for example) has no prior knowledge of exactly how many GSS
files will be dumped by the first, it does not know how to number its own
output files.

There is thus a discrepancy between the filenames and file headers for 130
of the GSS flatfiles in Release 226.0. Consider gbgss177.seq :

GBGSS1.SEQ Genetic Sequence Data Bank
June 15 2018

NCBI-GenBank Flat File Release 226.0

GSS Sequences (Part 1)

87375 loci, 64103840 bases, from 87375 reported sequences

Here, the filename and part number in the header is "1", though the file
has been renamed as "177" based on the number of files dumped from the other
system. Files gbgss177.seq.gz through gbgss306.seq.gz are affected. We hope
to resolve this discrepancy at some point, but the priority is certainly
much lower than many other tasks.

1.4 Upcoming Changes

1.4.1 Changes to nucleotide and protein accession formats

The accession format used for Whole Genome Shotgun (WGS), Transcriptome
Shotgun Assembly (TSA), and Targeted Locus Study (TLS) sequencing projects
consists of a four-letter Project Code prefix, a two-digit Assembly-Version
number, followed by 6, 7, or 8 digits (depending on the number of sequences
in the project).

By the end of 2018 the INSDC members plan to expand this format, using
a six-letter Project Code prefix, two-digit Assembly-Version number, followed
by 7, 8, or 9 digits. An example of such an accession is AAAAAA020000001 .

Non-WGS/TLS/TSA nucleotide sequences currently make use of a "2+6"
accession format, consisting of a two-letter prefix followed by six digits.
This format will be expanded to make use of eight digits, allowing accession
ranges that have been exhausted (eg: JG000001-JG999999) to be "re-opened"
(eg: JG00000001-JG99999999).

Protein sequences currently make use of a "3+5" accession format, consisting
of a three-letter prefix followed by five digits. This format will be expanded
to make use of seven digits, allowing exhausted protein accession ranges such
as EZZ00001-EZZ99999 to be re-opened as EZZ0000001-EZZ9999999.

The earliest date on which these new accessions will appear for GenBank
sequences is December 15th 2018.