Open Scholarly Metrics for the Journal of Open Source Software

The Journal of Open Source Software (JOSS) is a developer friendly, open access journal for research software packages published as open source software. JOSS started publishing in 2016, and has published more than 1,500 articles so far. After five years and that many articles, it is time to have a closer look at how JOSS content is reused. Given the context of publishing open access articles about open source software, it is only natural to look at reuse using open (scholarly) metrics, taking advantage of the fact that JOSS is registering all its articles as Crossref DOIs, and Crossref keeps track of citation counts and makes them openly available in the Crossref REST API (via the is-referenced-by-count property). It is straightforward to collect this information from Crossref using a Jupyter notebook and visualize it:

The most obvious finding in this chart is the small number of JOSS papers with very high citation counts, including 2,094 citations for the most-cited paper published in November 2019 (Wickham et al. 2019, Welcome to the Tidyverse), and currently 19 papers with at least 19 citations – see all JOSS papers with DOI, title, publication date and citation count here.

While we typically are most interested in citations, there are other dimensions of reuse, summarized for example in this table:

Tracking views and downloads of scholarly articles is a significant effort as it requires a standardized approach, but some of the other dimensions are made easily available via the Crossref Event Data service and can be collected via a Jupyter notebook.

Here Crossref is the combined number of citations collected via the REST API, DataCite is the number of citations of JOSS articles by DataCite DOIs, typically for the software described in the JOSS article archived via GitHub in the Zenodo software archive, Twitter is the number of tweets referencing the JOSS article, Wikipedia is the number of Wikipedia edits referencing the JOSS article, Newsfeed is the number of blog posts or news articles mentioning a JOSS article, and Hypothesis is the number of publicly available Hypothesis.is annotations mentioning a JOSS article.

Using the Jupyter notebook there are several paths one could follow to go into more detail. Or one can expand on the work done for JOSS by Charlotte Soneson (JOSS Submission Analysis) addressing very similar questions and in much more detail (sans the social media usage), for example the software licenses used for the code repositories described in the JOSS papers:

Note: updated October 26, 2021 with a link to the JOSS analysis done by Charlotte Soneson.

References

Wickham H, Averick M, Bryan J, et al. Welcome to the Tidyverse. JOSS. 2019;4(43):1686. doi:10.21105/joss.01686

Fenner M. What Can Article-Level Metrics Do for You? PLoS Biol. 2013;11(10):e1001687. doi:10.1371/journal.pbio.1001687