New Docs Part 2: The Gory Details

Continuous improvement is at the heart of Agile methodology: always striving to make changes that advance a project or team towards better results. This philosophy applies not just to code, but to documentation as well.

In a recent blog, we introduced our new rebranded documentation, with enhanced content and functionality. Complex projects, particularly open-source ones, benefit from effective easy-to-access documentation for improved user experience. They can also benefit from a “docs as code” approach, where, among other things, documentation builds are integrated with those of the software.

Technical implementation: how we got here

Since 2002, OpenNMS has treated docs like source code. Initially we used DocBook as the format for content creation, moving to AsciiDoc in 2015. AsciiDoc was extremely feature rich, and it was easy to integrate into our Java Maven pipeline, generated simple static single page HTML, and came with cascading style sheets (CSS). To generate the output, we ran a simple mvn install command in the docs directory.

This strategy grew challenging when we added Helm (our data visualization project) to the mix. Single-page HTML was cumbersome, especially in large docs. We split the docs into Developers, Admin, and Users sections, then gathered them under a single page. Ben Reed wrote a UI and integrated it into our CircleCI pipeline and release process.

As we added even more projects, we needed a solution that would allow for versioned docs, integrated into the release process, pulled from different GitHub repositories to a single location, all with the same format. Antora, made by the group behind AsciiDoc, addressed a lot of the issues we faced. Antora does not use a single page, handles docs from multiple individual versioned projects, uses AsciiDoc, and has a clear separation between the UI content assembly and content.

Search functionality

As part of our documentation upgrade, we explored several options to add search functionality to the documentation, eventually settling on Lunr, a JavaScript search index, because of its ease of implementation. We already had a big project simply migrating our existing documentation set to Antora (Agile=incremental improvements!), so went with a solution that quickly gave us basic search functionality.

In a single page HTML document you can easily use your browser search, but with Antora-built docs we don’t have this feature anymore. We started with the Antora Lunr integration. We create a search index at build time and ship it along with the page, and all the search is done in the browser.

Challenge 1: syntax updates

Unfortunately, the heading syntax required for Antora generation was slightly different from what we had been using. Because we also updated some of the content while we migrated to Antora, we couldn’t do a simple search and replace for the headings. Doing this manually (for the most part) on such a big documentation set was time-consuming.

Challenge 2: build integration

We had to refactor our CI/CD pipeline in CircleCI a bit. For all docs, we now build from a single docs repository. At this point, CircleCI had no webhooks to trigger a build and publish the docs to our website. We had to find a workaround with nightly scheduled build jobs. As the content grows, we can still build everything in under 10 minutes. However, we did notice that build jobs started to fail infrequently and ran out of memory. We gradually increased our build machines from a medium size (2CPU 4GB) to a larger machine (4CPU 8GB) over time to keep the jobs reliably running.

And of course, we needed to decide what to do with the old (pre-Antora) documentation set. We created an archive of the old documentation that users can access from the “Archived Docs” menu on the new documentation site.

Challenge 3: search functionality

Unfortunately, by design, the Lunr search creates one large index file during the build process. This takes up a lot of memory and may cause the build to time out. It also causes storage problems, since the project bundle is significantly larger than we would like. We are exploring other options to address this issue.

We might face some challenges here with the size of the index we need to ship. Going forward, we may investigate external search index providers like Algolia, which has a lot of adoption in the Antora community.

Challenge 4: search engine optimization (SEO)

Changing our docs website to docs.opennms.com would affect our SEO. To avoid delivering too many 404s, we set up redirects to the docs.opennms.com entry page, but in the end, you have to break eggs to make an omelet.

Next steps

Now that we have migrated to Antora, not only will we continue to update the docs to describe new features, we also plan to update existing documentation to correct errors, restructure/reorganize content to make it even easier for users to find the answers they need, and improve search functionality. What we are missing right now is also a “latest” or “current” floating tag. We have to reference docs in external links with a specific version number, which is addressed in the roadmap in Antora 3.0.

Thanks to the community

Kudos to the AsciiDoc and Antora community for developing these great tools, and for the excellent documentation, support, and forums. Our implementation was much easier because of this support.

How to contribute

Don’t forget that you can contribute to the OpenNMS documentation. For information on how to help, see Write and Build Documentation, join our Write the Docs channel on Mattermost, or join discussions on Discourse.

Thanks to Ronny Trommer for describing the technical side of our migration to Antora and our documentation creation process.