🍄Willow

I've been frustrated by the scattered state of software updates for a very long time. I manage NixNet. One piece of software can be found on GitHub, another piece on GitLab, one on Bitbucket, a fourth on SourceHut, and a fifth on the person's self-hosted Forgejo instance. Updates about that piece of software are usually on the git forge through RSS.

But not always. For example, GitLab just doesn't support RSS feeds for anything. They have their own API. Forgejo is in the same situation, though they do have plans for RSS feeds. Some updates are on the developers' personal blog. Sometimes there are CVEs for specific software and they get published on Not GitHub before they're fixed in a release.

I want to know all of this but I don't want to constantly be checking that software's release page every few days. That's just ridiculous.

I want to bring some organisation to this mess.

The idea is to bring all of those sources into a single UI that allows developers and admins to see what's going on at a glance. We'll start with the feeds themselves.

Example workflow

The user installs a piece of software on their server that hasn't been packaged by their distribution: a library or server application, for example. They paste the URL to the project into … we'll call it Willow for now, indicate which forge it's hosted on (Forgejo, SourceHut, GitHub, GitLab, or Other), hit Next, Willow fetches the repo's tag/release info and asks the user which version they're running. The user indicates which by selecting a radio option. From then on, Willow will keep track of the software's latest release vs what the user is currently running. When outdated, the UI will indicate this by making it stand out or bringing it to the top of the list or something and the user will be able to configure email/XMPP/IRC/Gotify/??? alerts. That will probably be offloaded to AlertManager

Organising

As the user tracks more software, it becomes more and more difficult to navigate the web UI in an efficient manner so we need some additional functionality. Enter projects, tags, and organisations.

Projects & organisations

Each piece of software can be associated with a larger project and each project with an organisation. For example, a user is developing an application for work and decides to add a dependency. They create a project in Willow and categorise the dependency underneath it. Each project can be associated with a particular organisation and a particular user. For example, Company develops a couple applications that User is responsible for. User also does some side work on various projects for Client. Client and Company both have their own organisations with sub-projects they assign to User. User will receive updates and notifications about those projects specifically. Nothing else Client or Company does with Willow generates noise for the developer because they're only responsible for a couple of projects.

Tagging

This facilitates more fine-grained organisation. The way I personally plan to use tags is to indicate language(s) and which server(s) are running the software. When the software has an update, I can click the project to see which servers I need to update it on. I can also click the server itself and see all of *its* updates in a single UI.

Configurable aggregate feeds

I'd like to allow users to select projects and feeds for aggregation into a single RSS feed they can add to their own reader. They might not need all updates from *all* projects, so they pick which ones they want to see.

Random thoughts

Noise level

Notifications and RSS feed are all opt-in, granular, and configurable. By default, things will just show up in the web UI. Maybe allow the user to select whether they want everything surfaced on the dashboard or just outdated software.

Licensing/monetisation

Will at least be source-available but very likely licensed under BSD-3-Clause with something like InvoiceNinja's $30/yr white-label license.

All Pro and Enterprise features from the hosted app are included in the open-code. We offer a $30 per year white-label license to remove the Invoice Ninja branding from client facing parts of the app.

— https://github.com/invoiceninja/invoiceninja

Potentially evaluate the Big Time License for dual-licensing as open source for non-profit use and small businesses with paid licences for larger companies.

Paid hosting will be provided by my company, potentially with multiple tiers. We'll see what happens when it's further along; I might decide monetisation is too much to deal with ¯\_(ツ)_/¯

2023-05-12 Notes

  • Git implementation in go

  • Sort with semver for now

    • As become aware of other strategies, build those in as options

    • go x lib impl

  • Initial work needs to be on cloning/fetching repos and checking version. Don't worry about the rest of it until that works. When we need to present information to the user, display it in a simple HTML table or something. Solve the simple cases and build on that.

  • Schedule work with time.Ticker

  • For channel stuff, consult the relevant gobyexample pages

Closing thoughts:

  • You can run GOPRIVATE=* go get -x <somerepo> to see the git commands that go runs when fetching a repo. That's likely the most efficient way to check for releases, so it's probably what you'll want to try to replicate

  • You can get away with a JSON configuration file instead of a database right up until you want to support multiple users. I think that's probably worth doing (staying with JSON) so that you can focus on getting other things right (your desired front-end UX, notifications, etc…). I agree that you'll want a DB in the end, but it can be a large pill to swallow, so working without.

  • You can use os/signal to receive signals (like SIGHUP or SIGINT) over a channel. This can let you do things like toggle on/off a runtime trace based on a signal. You can also be really clever and write an HTTP endpoint that collects a runtime trace in the response body, and then you can fetch traces with curl whenever you want. For added flexibility, make the duration to trace for a query parameter :D

Subhyphae