GenBank Release 222.0 Available : October 17 2017

[Cavanaugh, Mark (NIH/NLM/NCBI) [E]]

Greetings GenBank Users,

GenBank Release 222.0 is now available via FTP from the National Center
for Biotechnology Information (NCBI):

Ftp Site Directory Contents
---------------- --------- ---------------------------------------
ftp.ncbi.nih.gov genbank GenBank Release 222.0 flatfiles
ncbi-asn1 ASN.1 data used to create Release 222.0

Close-of-data for GenBank 222.0 occurred on 10/14/2017. Uncompressed,
the Release 222.0 flatfiles require roughly 850 GB (sequence files only).
The ASN.1 data require approximately 704 GB.

Recent statistics for 'traditional' sequences (including non-bulk-oriented
TSA, and excluding WGS, bulk-oriented TSA, TLS, and the CON-division):

Release Date Base Pairs Entries

221 Aug 2017 240343378258 203180606
222 Oct 2017 244914705468 203953682

Recent statistics for WGS sequencing projects:

Release Date Base Pairs Entries

221 Aug 2017 2242294609510 499965722
222 Oct 2017 2318156361999 508825331

Recent statistics for bulk-oriented TSA sequencing projects:

Release Date Base Pairs Entries

221 Aug 2017 167045663417 186777106
222 Oct 2017 172909268535 192754804

Recent statistics for bulk-oriented TLS sequencing projects:

Release Date Base Pairs Entries

221 Aug 2017 824191338 1628475
222 Oct 2017 2993818315 9479460

During the 62 days between the close dates for GenBank Releases 221.0
and 222.0, the 'traditional' portion of GenBank grew by 4,571,327,210
basepairs and by 773,076 sequence records. During that same period,
151,221 records were updated. An average of 14,908 'traditional' records
were added and/or updated per day.

Between releases 221.0 and 222.0, the WGS component of GenBank grew by
75,861,752,489 basepairs and by 8,859,609 sequence records.

Between releases 221.0 and 222.0, the TSA component of GenBank grew by
5,863,605,118 basepairs and by 5,977,698 sequence records.

Between releases 221.0 and 222.0, the TLS component of GenBank grew by
2,169,626,977 basepairs and by 7,850,985 sequence records.

For additional release information, see the README files in either of
the directories mentioned above, and the release notes (gbrel.txt) in
the genbank directory. Sections 1.3 and 1.4 of the release notes
(Changes in Release 222.0 and Upcoming Changes) have been appended
below for your convenience.

Release 222.0 data, and subsequent updates, are available now via
NCBI's Entrez and Blast services.

As a general guideline, we suggest first transferring the GenBank
release notes (gbrel.txt) whenever a release is being obtained. Check
to make sure that the date and release number in the header of the
release notes are current (eg: October 15 2017, 222.0). If they are
not, interrupt the remaining transfers and then request assistance from
the NCBI Service Desk.

A comprehensive check of the headers of all release files after your
transfers are complete is also suggested. Here's how one might go about
this on a Unix or Linux platform, using csh/tcsh :

set files = `ls gb*.*`
foreach i ($files)
head -10 $i | grep Release
end 1.3 Important Changes in Release 222.0

1.3.1 Organizational changes

The total number of sequence data files increased by 33 with this release:

- the BCT division is now composed of 407 391 files (+16)
- the CON division is now composed of 360 359 files (+1)
- the INV division is now composed of 157 155 files (+2)
- the PAT division is now composed of 301 294 files (+7)
- the PLN division is now composed of 157 150 files (+7)

1.3.2 GSS File Header Problem

GSS sequences at GenBank are maintained in two different systems, depending
on their origin, and the dumps from those systems occur in parallel. Because
the second dump (for example) has no prior knowledge of exactly how many GSS
files will be dumped by the first, it does not know how to number its own
output files.

There is thus a discrepancy between the filenames and file headers for 130
of the GSS flatfiles in Release 222.0. Consider gbgss175.seq :

GBGSS1.SEQ Genetic Sequence Data Bank
October 15 2017

NCBI-GenBank Flat File Release 222.0

GSS Sequences (Part 1)

87375 loci, 64103840 bases, from 87375 reported sequences

Here, the filename and part number in the header is "1", though the file
has been renamed as "175" based on the number of files dumped from the other
system. Files gbgss175.seq.gz through gbgss304.seq.gz are affected. We hope
to resolve this discrepancy at some point, but the priority is certainly
much lower than many other tasks.

1.4 Upcoming Changes

1.4.1 New source feature qualifier : /submitter_seqid

Data submitters typically have their own identifiers for genomic contigs and
scaffolds of Whole Genome Shotgun (WGS) sequencing projects, the RNA sequences
of Transcriptome Shotgun Assembly (TSA) sequencing projects, and the genomic
loci of Targeted Locus Study (TLS) sequencing projects. These identifiers can
be very simple (contig01, contig02, etc.), or they can have a bit more
meaning/structure. Examples of the latter include:

gcontig_1106166512749 (ABDU01000001)
CCB157_001 (BDDQ01000001)
Lo7_v2_contig_2306 (CCJQ010001199)

The INSDC has decided that it would be helpful to provide these identifiers
in a formalized way, since they may be known-to or used-by parties other than
the submitters themselves. For example, if a submitter had made them public
in some sort of data products, or used by genome browsers, or cited in an
analysis, or mentioned on websites.

These submitter identifiers will be provided via a new qualifier of the
source feature : /submitter_seqid . The value format for the qualifier will
be free text.

A complete definition of the qualifier will be provided when it becomes
available. The earliest implementation date would be as of GenBank Release
223.0 on December 15th 2017. But a more realistic timeframe is January or
February of 2018.

1.4.2 New /gap_type value : "contamination"

When contamination is discovered in a sequence record, removing the
bases from the sequence data can be problematic (especially at the 5' end)
because the length of the sequence changes. If there exist higher-level
scaffold/CON-division records (possibly chromosomes), the resulting change
in length requires an adjustment to the coordinate system of the scaffold/
chromosome, and the features annotated on it. The impact of such a change
on both data submitters and users can impose quite a burden.

To address this, the INSDC has decided to introduce a new Gap Type for
the assembly_gap feature : contamination . When sequence contamination
is discovered, the submitter will have the option of replacing the affected
base pairs with Ns, via a terminal assembly_gap feature. For example:

assembly_gap 1..2956
/estimated_length=2956
/gap_type="contamination"
/note="contamination masked with Ns"

An updated definition of the /gap_type qualifier will be provided when
it becomes available. The earliest implementation date would be as of
GenBank Release 223.0 on December 15th 2017. But a more realistic timeframe
is January or February of 2018.

Or, if the files are compressed, perhaps:

gzcat $i | head -10 | grep Release

If you encounter problems while ftp'ing or uncompressing Release
222.0, please send email outlining your difficulties to:

in...@ncbi.nlm.nih.gov

Mark Cavanaugh, Michael Kimelman, Ilya Dondoshansky
GenBank
NCBI/NLM/NIH/HHS