Summary: I deployed a new GitLab CI runner backed by Podman instead of Docker, we hope it will improve the stability and our capacity at building images, but I need help testing it.

# Background

We've been having [stability issues][] with the Docker runners for a while now. We also started looking again at container image builds, which are currently failing without Kaniko.

[stability issues]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41295 [ships instructions on how to build containers inside Podman]: https://docs.gitlab.com/runner/executors/docker.html#use-podman-to-build-con...

# Proposal

## Testers needed

I need help testing the new runner. Right now it's marked as not running "untagged jobs", so it's unlikely to pick your CI jobs and run them. It would be great if people could test the new runner.

See the [GitLab tag documentation][] for how to add tags to your configuration. It's basically done by adding a `tags` field to the `.gitlab-ci.yml` file.

Note that in TPA's [ci-test gitlab-ci.yaml file][], we use a `TPA_TAG_VALUE` variable to be able to pass arbitrary tags down into the jobs without having to constantly change the .yaml file, which might be a useful addition to your workflow.

The tag to use is `podman`.

You can send any job you want to the `podman` runner, but we'd like to test a broad variety of things before we put it in production, but especially image buildings. Upstream even has a [set of instructions to build packages inside podman][].

[ci-test gitlab-ci.yaml file]: https://gitlab.torproject.org/tpo/tpa/ci-test/-/blob/main/.gitlab-ci.yml [GitLab tag documentation]: https://docs.gitlab.com/ee/ci/yaml/#tags [set of instructions to build packages inside podman]: https://docs.gitlab.com/runner/executors/docker.html#use-podman-to-build-con...

## Long term plan

If this goes well, we'd like to converge towards using `podman` for all workloads. It's better packaged in Debian, and better designed, than Docker. It also allows us to run containers as non-root.

That, however, is not part of this proposal. We're already running Podman for another service (MinIO) but we're not proposing to *convert* all existing services to `podman`. If things work well enough for a long enough period (say 30 days), we might turn off the older Docker running instead.

# Alternatives considered

To fix the stability issues in Docker, it might be possible to upgrade to the latest upstream package and abandon the packages from Debian.org. We're hoping that will not be necessary thanks to Podman.

To build images, we could create a "privileged" runner. For now, we're hoping Podman will make building container images easier. If we do create a privileged runner, it needs to take into account the long term [tiered runner approach](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41044).

# Deadline

The service is already available, and will be running untagged jobs in two weeks unless an objection is raised.

# Status

This proposal is currently in the `proposed` state.

# References

Feedback can be provided in the [discussion issue][].

[discussion issue]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41296