works_topics issue
66 views
Skip to first unread message
Jérémie Roussel
Oct 2, 2026, 10:51:29 AMOct 2
to OpenAlex Community
Hi, i am working on the openalex database (from the snapshot with
last update on the 26/06/2026) and i discovered a hidden problem on the
works_topics.
Quick summary: 22 million works are affected. The same 3 topics are
assigned to all of these works with the exact same scores. It is also
worth noting that these topics are totally unrelated to the content of
the publications and are mostly non-english (99.2%).
With
the help of Claude Pro, i produced the below bug report as accurate as
possible to explain my discovery and help you resolve it (the below is
also attached).
# Constant default topic triple assigned to works without English-language text
## Summary
A constant triple of topics, with constant scores, is assigned to works the
topic classifier apparently cannot process — in practice, works whose title
and abstract are not in English. The first of the three is also written to
`primary_topic`, so these works are presented as confidently classified when
no classification took place.
The triple is always, in this order:
| Rank | Topic | Score | Subfield | Field |
|---|---|---|---|---|
| 1 (`primary_topic`) | `T14423` Military Technology and Strategies | 0.0184 | Aerospace Engineering | Engineering |
| 2 | `T13154` Legal and Regulatory Analysis | 0.0105 | Transportation | Social Sciences |
| 3 | `T14235` Linguistic, Cultural, and Literary Studies | 0.0086 | Sociology and Political Science | Social Sciences |
Affected works: 22,045,803, or 4.32% of the corpus. For the single
year 2018, restricted to `type = article`, the pattern covers 515,705
works — 6.2% of that year's articles.
## Reproducible evidence
Two works with nothing in common return byte-identical topic arrays:
- `W3130247736` — Japanese, 2018, trauma care delivered through Médecins Sans
Frontières field missions.
- `W2894336679` — Chinese, 2018, personalised learning in chemistry teaching
under China's new college entrance examination.
Both return the three topics above, in that order, with scores 0.0184 /
0.0105 / 0.0086, and both have `primary_topic` = *Military Technology and
Strategies*. Neither paper has any connection to military technology,
transport regulation, or literary studies.
Checked against `api.openalex.org/works/...` on 2 October 2026, so this is
not an artefact of the snapshot or of our own loading pipeline.
Ten further examples, each verifiable through the API:
| ID | Title | Lang | Year | Citations |
|---|---|---|---|---|
| `W2239339950` | 3．COPD 急性増悪患者の呼吸リハビリテーションプログラム | ja | 2011 | 0 |
| `W629819862` | ハイウェイホテルの存在意義とSA・PAの役割 | ja | 2008 | 0 |
| `W617677911` | 媼昔語り(上)難儀な目におうた | ja | 2009 | 0 |
| `W4392591077` | سير عمل النظام الدولي لمراقبة المخدر | ar | 2023 | 0 |
| `W1099338339` | 外国船舶監督官 サブスタンダード船の排除を通じ… | ja | 2015 | 0 |
| `W376953663` | 遊休・荒廃桑園活用のために--遊休桑園活用対策検討から | ja | 1986 | 0 |
| `W617933229` | 中央アジア・テンギス(Tengiz)油田 | ja | 2004 | 0 |
| `W4299875180` | Військовослужбовці вермахту у сприйнятті населення окупованого Донбасу | uk | 2018 | 0 |
| `W3171314581` | 초고층 건축물 피난안전구역의 비상방송음 음성명료도 예측 및 평가 | ko | 2020 | 0 |
| `W980268294` | 问卷调查中的"假卷"现象和解决方法 | zh | 1991 | 0 |
The dates span 1986 to 2023, so this is not confined to one ingest vintage.
## Why we read this as a classifier fallback rather than a bad prediction
**The scores are identical to four decimal places across millions of
unrelated works.** A model producing predictions yields varying confidence; a
constant value is a prior, not a prediction.
**The three topics always appear together, in the same rank order.** The
cleanest demonstration is at subfield level. Counting 2018 articles assigned
to each of the three subfields, and then recounting with the triple excluded:
| Subfield | Works assigned | Works after excluding the triple | Difference |
|---|---|---|---|
| Transportation | 582,595 | 66,890 | 515,705 |
| Aerospace Engineering | 765,580 | 249,875 | 515,705 |
| Sociology and Political Science | 1,351,834 | 836,129 | 515,705 |
The difference is identical to the unit across all three. The same 515,705
works land in all three subfields simultaneously, which is only possible if
the three assignments are emitted as one block rather than predicted
independently.
The affected works are 99.2% non-English. By language:
| Language | Works | Share |
|---|---|---|
| ja | 10,522,174 | 47.7% |
| zh | 4,059,950 | 18.4% |
| ru | 2,629,121 | 11.9% |
| ko | 2,106,654 | 9.6% |
| ar | 628,972 | 2.9% |
| uk | 606,016 | 2.7% |
| (null) | 593,120 | 2.7% |
| fa | 433,168 | 2.0% |
| en | 176,593 | 0.8% |
| others | — | 1.3% |
The trigger is the absence of usable English text, not the subject matter.
Japanese, Chinese, Russian and Korean alone account for 87.6%.
The three topics themselves are legitimate and also receive genuine
high-confidence assignments — their scores span 0.0004 to 0.9999. The defect
is specifically the mass of assignments carrying the constant triple scores,
not the topics.
## Impact
primary_topic` is populated from a 0.018-confidence guess. Most
downstream consumers read `primary_topic` without inspecting the score, so a
Japanese clinical paper is distributed to them as aerospace engineering. A
null would be more useful than a wrong value, and far more useful than a
wrong value that looks authoritative.
Subfield work counts are inflated beyond plausibility. 2018 had 8,262,869
articles in total. Transportation, Aerospace Engineering and Sociology and
Political Science together account for roughly 31% of them. Transportation
alone is credited with 582,595 articles — 7% of world output for one year, in
a subfield whose genuine share is a small fraction of that. After removing
the triple, the three counts fall to 66,890, 249,875 and 836,129, which are
plausible figures.
Mean citation counts per subfield are depressed by large factors. Because
the affected works are almost never cited, they drag down any average computed
within the subfields they land in. Taking the mean `cited_by_count` of 2018
articles per subfield — a plain aggregate over OpenAlex fields, reproducible
by anyone:
| Subfield | Mean citations, as published | Mean citations, triple excluded | Factor |
|---|---|---|---|
| Transportation | 0.978 | 7.990 | 8.2× |
| Aerospace Engineering | 2.553 | 7.681 | 3.0× |
| Sociology and Political Science | 3.082 | 4.940 | 1.6× |
For reference, the mean across all 8,262,869 articles of 2018 is 8.62. Once
the triple is excluded, Transportation and Aerospace Engineering return to
ordinary values; as published, they are implausible outliers.
## Suggested remedies, in order of preference
Leave `topics` empty and `primary_topic` null when the classifier cannot
produce a prediction above a confidence floor. An explicit absence is
actionable; a confident-looking wrong value is not.
Failing that, expose the fallback explicitly — a flag, a sentinel topic, or a
documented reserved score value — so consumers can exclude it with one
condition rather than having to identify the fingerprint themselves.
Independently of either: document in the API reference that `topics` scores
are not calibrated probabilities and that `primary_topic` may carry
arbitrarily low confidence. We found no warning to that effect, and the field
name invites the opposite assumption.
Extending the classifier to non-English titles and abstracts would address the
root cause. We assume that is a larger undertaking and are not asking for it
here — but we note that the current behaviour systematically mislabels
scholarship published in Japanese, Chinese, Russian and Korean, which is a
coverage equity issue as much as a data quality one.
## What we are not asking for
We are not requesting record-by-record corrections. This is a single
systematic pattern covering 22 million works, and the fix belongs in the
classification pipeline, not in the correction queue.
---
*Reported by Roussel Jérémie, Organisation currently being set up. Figures computed from the OpenAlex Parquet
snapshot loaded into PostgreSQL, using only OpenAlex fields
(`works_topics.score`, `topics.subfield_id`, `works.cited_by_count`,
`works.language`, `works.type`, `works.publication_year`). SQL available on
request; every individual example is verifiable through the public API.*
## Summary
A constant triple of topics, with constant scores, is assigned to works the
topic classifier apparently cannot process — in practice, works whose title
and abstract are not in English. The first of the three is also written to
`primary_topic`, so these works are presented as confidently classified when
no classification took place.
The triple is always, in this order:
| Rank | Topic | Score | Subfield | Field |
|---|---|---|---|---|
| 1 (`primary_topic`) | `T14423` Military Technology and Strategies | 0.0184 | Aerospace Engineering | Engineering |
| 2 | `T13154` Legal and Regulatory Analysis | 0.0105 | Transportation | Social Sciences |
| 3 | `T14235` Linguistic, Cultural, and Literary Studies | 0.0086 | Sociology and Political Science | Social Sciences |
Affected works: 22,045,803, or 4.32% of the corpus. For the single
year 2018, restricted to `type = article`, the pattern covers 515,705
works — 6.2% of that year's articles.
## Reproducible evidence
Two works with nothing in common return byte-identical topic arrays:
- `W3130247736` — Japanese, 2018, trauma care delivered through Médecins Sans
Frontières field missions.
- `W2894336679` — Chinese, 2018, personalised learning in chemistry teaching
under China's new college entrance examination.
Both return the three topics above, in that order, with scores 0.0184 /
0.0105 / 0.0086, and both have `primary_topic` = *Military Technology and
Strategies*. Neither paper has any connection to military technology,
transport regulation, or literary studies.
Checked against `api.openalex.org/works/...` on 2 October 2026, so this is
not an artefact of the snapshot or of our own loading pipeline.
Ten further examples, each verifiable through the API:
| ID | Title | Lang | Year | Citations |
|---|---|---|---|---|
| `W2239339950` | 3．COPD 急性増悪患者の呼吸リハビリテーションプログラム | ja | 2011 | 0 |
| `W629819862` | ハイウェイホテルの存在意義とSA・PAの役割 | ja | 2008 | 0 |
| `W617677911` | 媼昔語り(上)難儀な目におうた | ja | 2009 | 0 |
| `W4392591077` | سير عمل النظام الدولي لمراقبة المخدر | ar | 2023 | 0 |
| `W1099338339` | 外国船舶監督官 サブスタンダード船の排除を通じ… | ja | 2015 | 0 |
| `W376953663` | 遊休・荒廃桑園活用のために--遊休桑園活用対策検討から | ja | 1986 | 0 |
| `W617933229` | 中央アジア・テンギス(Tengiz)油田 | ja | 2004 | 0 |
| `W4299875180` | Військовослужбовці вермахту у сприйнятті населення окупованого Донбасу | uk | 2018 | 0 |
| `W3171314581` | 초고층 건축물 피난안전구역의 비상방송음 음성명료도 예측 및 평가 | ko | 2020 | 0 |
| `W980268294` | 问卷调查中的"假卷"现象和解决方法 | zh | 1991 | 0 |
The dates span 1986 to 2023, so this is not confined to one ingest vintage.
## Why we read this as a classifier fallback rather than a bad prediction
**The scores are identical to four decimal places across millions of
unrelated works.** A model producing predictions yields varying confidence; a
constant value is a prior, not a prediction.
**The three topics always appear together, in the same rank order.** The
cleanest demonstration is at subfield level. Counting 2018 articles assigned
to each of the three subfields, and then recounting with the triple excluded:
| Subfield | Works assigned | Works after excluding the triple | Difference |
|---|---|---|---|
| Transportation | 582,595 | 66,890 | 515,705 |
| Aerospace Engineering | 765,580 | 249,875 | 515,705 |
| Sociology and Political Science | 1,351,834 | 836,129 | 515,705 |
The difference is identical to the unit across all three. The same 515,705
works land in all three subfields simultaneously, which is only possible if
the three assignments are emitted as one block rather than predicted
independently.
The affected works are 99.2% non-English. By language:
| Language | Works | Share |
|---|---|---|
| ja | 10,522,174 | 47.7% |
| zh | 4,059,950 | 18.4% |
| ru | 2,629,121 | 11.9% |
| ko | 2,106,654 | 9.6% |
| ar | 628,972 | 2.9% |
| uk | 606,016 | 2.7% |
| (null) | 593,120 | 2.7% |
| fa | 433,168 | 2.0% |
| en | 176,593 | 0.8% |
| others | — | 1.3% |
The trigger is the absence of usable English text, not the subject matter.
Japanese, Chinese, Russian and Korean alone account for 87.6%.
The three topics themselves are legitimate and also receive genuine
high-confidence assignments — their scores span 0.0004 to 0.9999. The defect
is specifically the mass of assignments carrying the constant triple scores,
not the topics.
## Impact
primary_topic` is populated from a 0.018-confidence guess. Most
downstream consumers read `primary_topic` without inspecting the score, so a
Japanese clinical paper is distributed to them as aerospace engineering. A
null would be more useful than a wrong value, and far more useful than a
wrong value that looks authoritative.
Subfield work counts are inflated beyond plausibility. 2018 had 8,262,869
articles in total. Transportation, Aerospace Engineering and Sociology and
Political Science together account for roughly 31% of them. Transportation
alone is credited with 582,595 articles — 7% of world output for one year, in
a subfield whose genuine share is a small fraction of that. After removing
the triple, the three counts fall to 66,890, 249,875 and 836,129, which are
plausible figures.
Mean citation counts per subfield are depressed by large factors. Because
the affected works are almost never cited, they drag down any average computed
within the subfields they land in. Taking the mean `cited_by_count` of 2018
articles per subfield — a plain aggregate over OpenAlex fields, reproducible
by anyone:
| Subfield | Mean citations, as published | Mean citations, triple excluded | Factor |
|---|---|---|---|
| Transportation | 0.978 | 7.990 | 8.2× |
| Aerospace Engineering | 2.553 | 7.681 | 3.0× |
| Sociology and Political Science | 3.082 | 4.940 | 1.6× |
For reference, the mean across all 8,262,869 articles of 2018 is 8.62. Once
the triple is excluded, Transportation and Aerospace Engineering return to
ordinary values; as published, they are implausible outliers.
## Suggested remedies, in order of preference
Leave `topics` empty and `primary_topic` null when the classifier cannot
produce a prediction above a confidence floor. An explicit absence is
actionable; a confident-looking wrong value is not.
Failing that, expose the fallback explicitly — a flag, a sentinel topic, or a
documented reserved score value — so consumers can exclude it with one
condition rather than having to identify the fingerprint themselves.
Independently of either: document in the API reference that `topics` scores
are not calibrated probabilities and that `primary_topic` may carry
arbitrarily low confidence. We found no warning to that effect, and the field
name invites the opposite assumption.
Extending the classifier to non-English titles and abstracts would address the
root cause. We assume that is a larger undertaking and are not asking for it
here — but we note that the current behaviour systematically mislabels
scholarship published in Japanese, Chinese, Russian and Korean, which is a
coverage equity issue as much as a data quality one.
## What we are not asking for
We are not requesting record-by-record corrections. This is a single
systematic pattern covering 22 million works, and the fix belongs in the
classification pipeline, not in the correction queue.
---
*Reported by Roussel Jérémie, Organisation currently being set up. Figures computed from the OpenAlex Parquet
snapshot loaded into PostgreSQL, using only OpenAlex fields
(`works_topics.score`, `topics.subfield_id`, `works.cited_by_count`,
`works.language`, `works.type`, `works.publication_year`). SQL available on
request; every individual example is verifiable through the public API.*
Have a good day,
Cordialy,
Roussel Jérémie
Reply all
Reply to author
Forward
0 new messages