Ahoy!

Well it's just been way too long since we've done this, but we've done it! Here's our short sysadmin minutes from today's meeting.

# Roll call: who's there and emergencies

onionoo-backend running out of disk space ([tpo/tpa/team#41343][])

[tpo/tpa/team#41343]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41343

# Dashboard cleanup

Normal per-user check-in:

* https://gitlab.torproject.org/groups/tpo/-/boards?scope=all&utf8=%E2%9C%... * https://gitlab.torproject.org/groups/tpo/-/boards?scope=all&utf8=%E2%9C%... * https://gitlab.torproject.org/groups/tpo/-/boards?scope=all&utf8=%E2%9C%...

General dashboards:

* https://gitlab.torproject.org/tpo/tpa/team/-/boards/117 * https://gitlab.torproject.org/groups/tpo/web/-/boards * https://gitlab.torproject.org/groups/tpo/tpa/-/boards

Nextcloud roadmap / spreadsheet.

Overall, it seems we are as you would expect when returning from a rather chaotic vacation. Backlog is large, but things seem to be under control.

We added SVN back on the roadmap after one too many tickets asking for setup.

# Metrics of the month

* hosts in Puppet: 89, LDAP: 89, Prometheus exporters: 166 * number of Apache servers monitored: 37, hits per second: 626 * number of self-hosted nameservers: 6, mail servers: 10 * pending upgrades: 1, reboots: 0 * average load: 0.69, memory available: 3.58 TiB/4.98 TiB, running processes: 424 * disk free/total: 53.19 TiB/126.72 TiB * bytes sent: 403.47 MB/s, received: 269.04 MB/s * planned bullseye upgrades completion date: 2024-08-02 * [GitLab tickets][]: 196 tickets including... * open: 0 * icebox: 163 * needs information: 5 * backlog: 13 * next: 9 * doing: 4 * needs review: 2 * (closed: 3301)

[Gitlab tickets]: https://gitlab.torproject.org/tpo/tpa/team/-/boards

Upgrade prediction graph lives at:

https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bookworm/

Now also available as the main Grafana dashboard. Head to https://grafana.torproject.org/, change the time period to 30 days, and wait a while for results to render.

# Number of the month: 42

34 machines were upgraded from bullseye to bookworm in the two first days of last week! We calculated this was an average of 20 minutes per host to upgrade.

The trick, of course, is that things often break *after* the upgrade, and that "fixing" time is not counted here. That said, last estimate for this was one hour per machine, and we're doing a whole fleet upgrade every 2-3 years, which means about ten hours of work saved per year.

But the number of the month is, of course, 42, as we now have an equal number of bookworm and bullseye machine, after the upgrade. And that number is, naturally, [42][].

See also https://xkcd.com/1205/ which, interestingly, we fall out of scope of.

[42]: https://en.wikipedia.org/wiki/42