Historical newspapers dataset released

[Eben English]

In collaboration with Boston Public Library, Harvard Law Library's Institutional Data Initiative has released a data set on HuggingFace derived from the BPL's public domain newspapers collection.

https://huggingface.co/collections/institutional/institutional-newspapers

This data set includes 1,473,635 newspaper scans from issues published between 1795 and 1930, which have been processed using ML and AI techniques to enhance the OCR and segment each article into distinct semantic units. This output has produced 83,147,041 individual crops segmented from those scans, over 16 billion o200k_base tokens of VLM OCR text, as well as bounding box coordinates, raw OCR, text analysis, crop type classification, language detection, named-entity recognition, subject classification, reading order detection, and text + image vector embeddings for each segment.

The publication of this data set will be of significant use for computational linguistics, AI model training, and historical research. The data processing pipeline created through this project can also serve as a model for other organizations that want to enhance collections of historical newspapers with improved full-text searching and subject analysis.

The dataset, pipeline code, and technical documentation are all openly available:

•  Dataset: https://huggingface.co/collections/institutional/institutional-newspapers
  
  
• Technical report: "Institutional Newspapers Pipeline: Deriving billions of high quality tokens from historical newspapers": https://doi.org/10.48550/arXiv.2608.18972
  
  
• Code: https://github.com/institutional/institutional-newspapers-pipeline

Read more here: https://institutional.org/posts/institutional-newspapers-boston-public-library/

Thanks,

Eben English
Boston Public Library