Text and Data Mining in Scholarly Publishing: A Guide
Text and data mining (TDM) means running software across many papers at once to find patterns no one could spot reading them one by one. In scholarly publishing, access is set by publisher licenses, platform APIs, and library agreements. Open corpora offer another route under stated reuse terms.
What is text and data mining?
Text and data mining applies computer programs to a collection of documents, not to one paper read in isolation. In scholarly publishing, the collection may be article metadata, abstracts, or full-text files, and the program searches for patterns such as word frequencies, co-occurring terms, topic clusters, or change over time.
A researcher might process a corpus of abstracts to see which terms appear together more often in a given decade. The method becomes useful only when the corpus is large enough to reveal a signal, so the first practical question is what the software is allowed to read. Publishers use similar techniques internally for search, recommendation, and quality control.
How do publishers and libraries handle TDM access?
Access normally follows one of four routes. A publisher license may allow or limit mining for subscribers. A platform API may deliver text under technical limits and reuse rules. A library agreement may add mining rights for an institution, and an open corpus may invite anyone to process the text under a stated license.
| Pathway | Who grants access | What to check |
|---|---|---|
| Publisher license | Journal or platform | The terms attached to the subscription or site |
| Platform API | Publisher or data provider | Allowed fields and reuse terms |
| Library agreement | Library and publisher consortium | Whether mining is included or needs an addendum |
| Open corpus | Repository, funder, or author | The open license or reuse statement |
Many scholarly platforms expose a public API for metadata. Crossref publishes a public API for article metadata, and metadata access does not replace full-text access. Read access and mining access are not the same thing. A library can check whether a subscription includes mining, and a publisher usually sets out its position in the license or on the platform. Because the same article may be available through more than one route, the permitted use can differ from one system to another.
What did the PRC-era guide and survey cover?
The Publishing Research Consortium (PRC) produced a 2013 guide on text mining and scholarly publishing, followed by a 2016 survey report on text and data mining. Virginia Tech’s VTechWorks repository holds the full documents.
| Document | Organization | Year | What it asked or concluded | Where to read |
|---|---|---|---|---|
| PRC Text Mining and Scholarly Publishing guide | Publishing Research Consortium | 2013 | Explains how text mining interacts with journal publishing and where access terms are set | VTechWorks repository |
| PRC text and data mining survey report | Publishing Research Consortium | 2016 | Reports survey responses about text and data mining in scholarly publishing | VTechWorks repository |
These documents date from 2013 and 2016, so their examples and practices may have changed. They remain useful as records of the questions publishers and libraries were asking, not as current legal guidance.
Where does this connect on the site?
Text mining feeds journal metrics because usage, citation, and topic patterns often come from mined article data. The broader effects appear in how metrics and analytics reshape publishing, where scholarly publishing is heading, library and publisher consortia, digital publishing of research, and ethics in scientific publishing.
FAQ
What is text and data mining?
Text and data mining applies computer programs to large collections of documents to identify patterns, frequencies, or relationships that manual reading would not easily reveal.
Can I mine journal articles I can read?
Not automatically. Access rights depend on the publisher's license and the platform terms for that journal or article. A librarian can help check what is allowed in your institution.
Is scraping a publisher site allowed?
Usually the site's terms of use restrict automated scraping. Publishers often provide an API or a negotiated agreement for large-scale mining projects instead.
Sources
Publishing Research Consortium. PRC text mining and scholarly publishing documents, 2013 and 2016. Virginia Tech VTechWorks repository. Crossref. Public metadata API cited in the access section.