For anyone who has worked in a large software project, especially one which is separated into lots of microservices or packages, we’ve had that moment where we realise we can’t keep the map of all the moving parts in our heads. Which changes are in testing and not yet in stable? How do they relate to eachother? This is all pretty easy with git branches when you’re on a single repository, but what if they’re split into 50 or even 100 repositories? It turns out that just a small bit of work in improving visibility can go a long way to improving the situation, and makes it much, much easier to pull the all important deployment trigger on new features.
Over at Ninja Blocks, we were in exactly this situation recently. The Ninja Sphere software platform is extremely modular, and the client software is split up into what are essentially microservices, each performing just one primary role. Each service has its own repository on GitHub, and our CI service builds it into a corresponding Debian package ready for installation on the client OS.
These packages, once built, are moved through a release pipeline (different apt repositories for unstable, testing, staging and stable), so Git isn’t involved anymore - all you know is that the binary was generated off master at the time it was created.
While the number of packages were small, everything was fine. Developers could keep track of their changes, and know roughly what would be moving between branches. But that didn’t last long! At the time of writing, there were over 50 packages making up the client software - drivers, apps, services, system tweaks, and lots more.
One day we realised this had to change, and made a tool to scrape our CI service, BuildKite, our repositories on GitHub and our apt repositories for a single purpose: to give visibility to exactly what version was where, and what had changed between each release channel.
The scraper correlates tags/versions from build outputs against git commits and apt channels, and generates this summary page. It’s not the prettiest thing in the world, but suddenly the team can see everything at a glance. To get a diff between versions, we even link to the diff on GitHub! The versions listed are versions from the apt repository correlated to the master commit it was generated from.
The interesting part is that with just a few hours of development time to write the scraper, visibility and efficiency increased and saved significantly more time than it took to write the tool itself. Most notably, it increased confidence in the action we were about to perform when pushing changes closer to end users, meaning we weren’t treading on eggshells every time we released.
Adding some visibility also encouraged further visibility, and within a week we added some extra hooks to keep git branches synced up to the commit that is in each release channel at any point in time, giving visibility on the Git side about which commits were in which channel. This tool is now used in standups every day to choose what’s ready to deploy.
The take home idea for me was that it’s sometimes worth taking a step back, and rather than just enduring the frustration of lack of visibility, tackle it with a bit of automation and development time and see what comes out - sometimes it might just become a core tool that’s used daily!