LangTrend

How does this work?

Full pipeline description, language classes composition, limitations, and references.

Pipeline diagram

LangTrend data pipeline from arXiv fetch through extraction, cleaning, LLM judge verification, manifest build, and dashboard deploy

Language classes

ClassDescription1Count2Examples2
Class 0Have exceptionally limited resources, and have rarely been considered in language technologies.6134Pinyin, Irarutu, Himarimã
Class 1Have some unlabelled data; however, collecting labelled data is challenging.130Hawaiian, Frisian, Nahuatl
Class 2A small set of labelled datasets has been collected, and language support communities are there to support the language.96Sinhala, Irish, Zulu
Class 3Has a strong web presence, and a cultural community that backs it. Have been highly benefited by unsupervised pre-training.30Hindi, Tamil, Urdu
Class 4Have a large amount of unlabelled data, and lesser, but still a significant amount of labelled data. have dedicated NLP communities researching these languages.22Indonesian, Russian, Italian
Class 5Have a dominant online presence. There have been massive investments in the development of resources and technologies.7Arabic, Chinese, English

1 Class descriptions are adapted from Joshi et al. [3] (opens in new tab) (project site: Linguistic Diversity (opens in new tab), which has been a great foundation for LangTrend in tracking language inclusion in the NLP world).

2 Counts and example languages are updated from the Some Languages are More Equal than Others (opens in new tab) repository, which follows the work from Ranathunga and de Silva [2] (opens in new tab).

Limitations

  • Language detection is based on explicit mentions in titles, abstracts, section text, and cleaned PDF body text, so indirect references (e.g., citing a multilingual dataset without naming the languages) can be missed. Furthermore, the regex pass may produce false positives (shorter language names in particular can appear as acronyms, author names, or technical terms). Text cleaning and acronym filtering reduce these, but don't eliminate them entirely.
  • Detected languages are also checked by an LLM judge, which classifies each one as studied, mentioned-only, or a false positive with a one-line reason. The judge is not infallible — like any LLM, it can hallucinate a plausible-sounding reason, misread context, or judge near-identical cases inconsistently across different papers, and it only sees a bounded context (the title, abstract, and snippets around each match) rather than the full paper. Every verdict is shown with its reason so it can be checked against the source paper, and nothing it flags is ever deleted — a judged false positive or mentioned-only detection stays visible, just excluded from (or dimmed in) the counts.
  • HTML extraction is preferred for accuracy, but the pipeline depends on the availability of arXiv HTML pages. If the HTML version is missing or incomplete, the pipeline falls back to the PDF (via Docling). If the PDF is also unavailable or has been withdrawn, only the abstract text is analysed.
  • The pipeline only covers papers submitted to the cs.CL arXiv category (It may include multiple categories, which include cs.CL). Multilingual NLP papers appearing in adjacent categories (cs.AI, cs.LG, cs.CV, etc.), which exclude cs.CL are not captured.
  • Papers are processed at fetch time. If an author updates a paper with new or removed language mentions, these are not reflected unless the pipeline re-runs for the same date window.
  • The pipeline runs once a week (every Tuesday), covering the previous week's arXiv announcement window. Papers arXiv announces later than their submission week (held in moderation, or delayed by a holiday) are added to the week they were actually announced in, so a week's paper list can include some with older submission dates. A paper cross-listed into cs.CL after its week was already fetched is not picked up.

Privacy

  • Page views are counted with GoatCounter, which sets no cookies and stores no personal data; the totals are shown in the footer.
  • Digest subscriptions are held by Buttondown: your email address and the languages you picked. Neither is stored in the LangTrend repository or read by its pipeline, and every email has an unsubscribe link.

References

  1. S. Ranathunga, N. De Silva, D. Jayakody, and A. Fernando, “Shoulders of Giants: A Look at the Degree and Utility of Openness in NLP Research,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), L.-W. Ku, A. Martins, and V. Srikumar, Eds., Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, pp. 519–529. doi: 10.18653/v1/2024.acl-short.48.
  2. S. Ranathunga and N. de Silva, “Some Languages are More Equal than Others: Probing Deeper into the Linguistic Disparity in the NLP World,” in Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Y. He, H. Ji, S. Li, Y. Liu, and C.-H. Chang, Eds., Online only: Association for Computational Linguistics, Nov. 2022, pp. 823–848. doi: 10.18653/v1/2022.aacl-main.62.
  3. P. Joshi, S. Santy, A. Budhiraja, K. Bali, and M. Choudhury, “The State and Fate of Linguistic Diversity and Inclusion in the NLP World,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, Eds., Online: Association for Computational Linguistics, Jul. 2020, pp. 6282–6293. doi: 10.18653/v1/2020.acl-main.560.
  4. C. Auer et al., “Docling Technical Report,” 2024, arXiv. doi: 10.48550/ARXIV.2408.09869.