Rogue Scholar upgrade pains

Martin Fenner -

This week the Rogue Scholar science blog archive experienced major upgrade pains, and Rogue Scholar search became unavailable from Tuesday until Thursday. I tried to upgrade to a pre-release version (13.0.0b4.dev0) of the InvenioRDM repository software, and ran into multiple issues. Going back to the previously installed v12.1.0 took longer than anticipated, mainly because of issues with the Opensearch index.

This morning Rogue Scholar is almost working normally again, except for the blog communities, which will take until Monday to be fixed.

The primary reason for upgrading the InvenioRDM software was so that I could integrate Crossref DOI registration, based on work I completed last week. The experience with installing a pre-release version of InvenioRDM told me a few things:

  • install InvenioRDM to production only after extensive testing (as I did last September/October),
  • service stability is more important than new features, and I am adjusting my deployment strategy and tooling,
  • observability is critical when running infrastructure, and this can be improved for Rogue Scholar.

Postpone upgrading to InvenioRDM v13.0

Integrating Crossref DOI registration into InvenioRDM requires a current development version of InvenioRDM, which is currently v13.x, ahead of the last released version v12.1​. As InvenioRDM v13.0 will be released in a few weeks, I will wait with that work until v13.0 is released and Rogue Scholar is updated to that version. There is additional DOI registration work needed, as Rogue Scholar not only registers DOIs with Crossref, but uses multiple DOI prefixes (not yet supported in InvenioRDM) and also accepts blog posts with DOIs registered externally with DataCite.

Upgrade Rogue Scholar Infrastructure

After growing to more than 40,000 blog posts in recent months, the Rogue Scholar infrastructure, particularly the Opensearch search index, needs a hardware upgrade. I will take this opportunity to also change my deployment strategy and tooling, and will start to use Kamal to deploy the InvenioRDM software to a dedicated server (provided by Hetzner).

Improve Observability

Rogue Scholar uses metrics and logging provided by Prometheus/Grafana and error reporting provided by Sentry. More work is needed to improve this observability to better handle incidents such as this week's upgrade issues.

References

Fenner, M. (2025, May 27). Major update on Commonmeta Crossref DOI registration. Front Matter. https://doi.org/10.53731/69k7z-w7030