Hello all,
I have recently found something interesting on my relays. On all relays and clients actually.
As always I am using Debian and apt to get Tor from deb.torproject.org tor-nightly-main-bullseye main (for example). I also have the keyring installed, proper key and everything.
Currently I am on:
Tor version 0.4.9.0-alpha-dev. This build of Tor is covered by the GNU General Public License (https://www.gnu.org/licenses/gpl-3.0.en.html) Tor is running on Linux with Libevent 2.1.12-stable, OpenSSL 1.1.1w, Zlib 1.2.11, Liblzma 5.2.5, Libzstd 1.4.8 and Glibc 2.31 as libc. Tor compiled with GCC version 10.2.1
Since October 2023.
If I manually check out deb.torproject.org with my browser I see there is another package released in February 2024, except it notes as the same version 4.9.0-alpha-dev but it has a different timestamp.
$ apt update reports no errors, looks like is working fine, but it doesn't notify there's a newer version and does not apply any updates to Tor. This happens only to Tor package from nightly, rest of packages from debian.org deb are updated as usual.
So, anyone else experienced this? Do I need to configure apt in a different way than we used to do? I maintain this style of setup for Tor since Debian 7 :-), any changes needed?
Thank you! -s7r
On Donnerstag, 15. Februar 2024 13:54:51 CET s7r wrote:
I have recently found something interesting on my relays. On all relays and clients actually.
As always I am using Debian and apt to get Tor from deb.torproject.org tor-nightly-main-bullseye main (for example). I also have the keyring installed, proper key and everything.
Currently I am on:
Tor version 0.4.9.0-alpha-dev.
Mee too.
I let nightly's upgrade automatically, but not stable. Therefore I have the following config in /etc/apt/50unattended-upgrades
Unattended-Upgrade::Origins-Pattern { ... // Update TorProject's nightly dev packages: (Suite & Codename: tor-nightly-main-bookworm) // apt-cache policy | grep release "o=TorProject,a=tor-nightly-main-bookworm,n=tor-nightly-main-bookworm"; };
If I manually check out deb.torproject.org with my browser I see there is another package released in February 2024, except it notes as the same version 4.9.0-alpha-dev but it has a different timestamp.
$ apt update reports no errors, looks like is working fine, but it doesn't notify there's a newer version and does not apply any updates to Tor. This happens only to Tor package from nightly, rest of packages from debian.org deb are updated as usual.
So, anyone else experienced this?
Damn, you're right:
http://deb.torproject.org/torproject.org/dists/tor-nightly-main-bullseye/InR... Suite: tor-nightly-main-bullseye Codename: tor-nightly-main-bullseye Date: Fri, 09 Feb 2024 11:28:02 UTC Valid-Until: Wed, 20 Mar 2024 11:28:02 UTC
root@boldsuck:~# apt update root@boldsuck:~# apt-cache show tor Package: tor Version: 0.4.9.0-alpha-dev-20230909T020422Z-1~d12.bookworm+1
So I also have the last version from September 9th, 2023, although one from February 9th, 2024 is in the archive. :-( Tor stable update is OK.
Full log output: https://paste.debian.net/1307420/
So, anyone else experienced this?
Damn, you're right:
http://deb.torproject.org/torproject.org/dists/tor-nightly-main-bullseye/InR... Suite: tor-nightly-main-bullseye Codename: tor-nightly-main-bullseye Date: Fri, 09 Feb 2024 11:28:02 UTC Valid-Until: Wed, 20 Mar 2024 11:28:02 UTC
root@boldsuck:~# apt update root@boldsuck:~# apt-cache show tor Package: tor Version: 0.4.9.0-alpha-dev-20230909T020422Z-1~d12.bookworm+1
So I also have the last version from September 9th, 2023, although one from February 9th, 2024 is in the archive. :-( Tor stable update is OK.
Full log output: https://paste.debian.net/1307420/
Yes, and it's not an isolated case. Absolutely all relays and bridges are I have are in the same situation. Most probably a lot of users are in the same situation (that run on nightly) and they don't even know, given apt does not complain at all. Debian is our largest user base IIRC.
My guess is that something changed with more recent releases of Debian (Bullseye and Bookworm) because in previous Buster this was not happening, even it was the same version number in nightly, like 4.8.0-aplha-dev, if would update if same version was found but with newer timestamp.
Running on nightly has been important for me, to detect bugs and submit test reports to people that work on patches.
I will try to investigate more, again, but from the apt logs it looks like everything is nice and fine - except it ignores the newer package in deb.tpo with newer timestamp...
On Donnerstag, 15. Februar 2024 20:42:03 CET s7r wrote:
My guess is that something changed with more recent releases of Debian (Bullseye and Bookworm) because in previous Buster this was not happening, even it was the same version number in nightly, like 4.8.0-aplha-dev, if would update if same version was found but with newer timestamp.
Looks to me like it's the archive:
http://deb.torproject.org/torproject.org/dists/tor-nightly-main-bookworm/mai... Package: tor Version: 0.4.9.0-alpha-dev-20230909T020422Z-1~d12.bookworm+1 Architecture: amd64
boldsuck wrote:
On Donnerstag, 15. Februar 2024 20:42:03 CET s7r wrote:
My guess is that something changed with more recent releases of Debian (Bullseye and Bookworm) because in previous Buster this was not happening, even it was the same version number in nightly, like 4.8.0-aplha-dev, if would update if same version was found but with newer timestamp.
Looks to me like it's the archive:
http://deb.torproject.org/torproject.org/dists/tor-nightly-main-bookworm/mai... Package: tor Version: 0.4.9.0-alpha-dev-20230909T020422Z-1~d12.bookworm+1 Architecture: amd64
For bullseye the same:
https://deb.torproject.org/torproject.org/dists/tor-nightly-main-bullseye/ma...
Version: 0.4.9.0-alpha-dev-20230909T020422Z-1~d11.bullseye+1
Seams to much more or less the 4.9.0-alpha-dev I have installed and is not upgrading...
What can we do to fix this? it's clear that there have been more more nightly releases in ~ 5 months obviously, so is it a problem at deb.tpo that needs addressed or on our machines?
On Fri, 16 Feb 2024, s7r wrote:
What can we do to fix this? it's clear that there have been more more nightly releases in ~ 5 months obviously, so is it a problem at deb.tpo that needs addressed or on our machines?
our gitlab-ci has not managed to build a tor nightly in ages.
eg https://gitlab.torproject.org/tpo/core/debian/tor/-/jobs/479743 | ERROR: Preparation failed: Error response from daemon: container | e4875f0d7fb7633bb78e9d16664db98e92c86fcca7114e75e44dfaa6a3bc973d does | not exist in database: no such container (manager.go:81:0s) | Will be retried in 3s ... | Using Docker executor with image debian:stable-slim ... | ERROR: Preparation failed: Error response from daemon: network name | runner-1yuaft6r-project-1218-concurrent-0-job-479743-network already | used: network already exists (manager.go:67:0s) | Will be retried in 3s ... | Using Docker executor with image debian:stable-slim ...
or
https://gitlab.torproject.org/tpo/core/debian/tor/-/jobs/479407
| ERROR: Job failed (system failure): Cannot connect to the Docker | daemon at unix:///var/run/docker.sock. Is the docker daemon running? | (docker.go:644:120s)
unless our gitlab-ci actually manages to build a whole set, you won't see packages on deb.tpo.
cf.
https://gitlab.torproject.org/tpo/core/debian/tor/-/pipelines?scope=all&...
some of these are actual tor building issues, like https://gitlab.torproject.org/tpo/core/debian/tor/-/jobs/479068
| sandbox/opendir_dirname: [forking] | FAIL ../src/test/test_sandbox.c:266: opendir: Operation not permitted [1] | [opendir_dirname FAILED] | sandbox/chmod_filename: [forking] OK
but since almost all build failures are actually problems with gitlab and not problems with the packaging (neither is that one), it's just tiresome to even start investigating.
Peter Palfrader wrote:
our gitlab-ci has not managed to build a tor nightly in ages.
Thank you for stepping in! No better person to ask :)
The upgrade via apt from nightly used to work every time, back since Debian Wheezy. It stopped to work since ~ autumn 2023.
The thing is, if you go with firefox on deb.torproject.org and look at the packages release you see a recent-ish timestamp on the tor package within max. 2 weeks old, however the system does not upgrade to it.
unless our gitlab-ci actually manages to build a whole set, you won't see packages on deb.tpo.
cf.
https://gitlab.torproject.org/tpo/core/debian/tor/-/pipelines?scope=all&...
some of these are actual tor building issues, like https://gitlab.torproject.org/tpo/core/debian/tor/-/jobs/479068
| sandbox/opendir_dirname: [forking] | FAIL ../src/test/test_sandbox.c:266: opendir: Operation not permitted [1] | [opendir_dirname FAILED] | sandbox/chmod_filename: [forking] OK
but since almost all build failures are actually problems with gitlab and not problems with the packaging (neither is that one), it's just tiresome to even start investigating.
Here is how a complete /etc/apt/sources.list file looks (at least under my system) - only pasting the deb.tpo related entries, the rest are the normal defaults of -security and -updates + distro main:
deb https://deb.torproject.org/torproject.org tor-nightly-main-bullseye main deb-src https://deb.torproject.org/torproject.org tor-nightly-main-bullseye main deb https://deb.torproject.org/torproject.org bullseye main deb-src https://deb.torproject.org/torproject.org bullseye main
There are non tor-nightly-main-* entries in the sources.list because it's the only way to install deb.torproject.org-keyring via apt, otherwise it will not find it.
---
Here is how apt-cache policy looks like:
Package files: 100 /var/lib/dpkg/status release a=now 500 https://deb.torproject.org/torproject.org bullseye/main amd64 Packages release o=TorProject,a=oldstable,n=bullseye,c=main,b=amd64 origin deb.torproject.org 500 https://deb.torproject.org/torproject.org tor-nightly-main-bullseye/main amd64 Packages release o=TorProject,a=tor-nightly-main-bullseye,n=tor-nightly-main-bullseye,c=main,b=amd64 origin deb.torproject.org 500 http://deb.debian.org/debian bullseye-updates/main amd64 Packages release v=11-updates,o=Debian,a=oldstable-updates,n=bullseye-updates,l=Debian,c=main,b=amd64 origin deb.debian.org 500 http://security.debian.org/debian-security bullseye-security/main amd64 Packages release v=11,o=Debian,a=oldstable-security,n=bullseye-security,l=Debian-Security,c=main,b=amd64 origin security.debian.org 500 http://deb.debian.org/debian bullseye/main amd64 Packages release v=11.9,o=Debian,a=oldstable,n=bullseye,l=Debian,c=main,b=amd64 origin deb.debian.org Pinned packages: --
If there are problems from gitlab that are hard to fix, what is the best way for testers and bug hunters to install the latest git main tor? git clone and build locally? This needs a lot of manual systemd configuration work, that was easily handled by apt :(
On Montag, 19. Februar 2024 00:27:04 CET s7r wrote:
Peter Palfrader wrote:
our gitlab-ci has not managed to build a tor nightly in ages.
Thank you for stepping in! No better person to ask :)
The upgrade via apt from nightly used to work every time, back since Debian Wheezy. It stopped to work since ~ autumn 2023.
Thanks to David everything is working again. https://gitlab.torproject.org/tpo/core/tor/-/issues/40861 https://gitlab.torproject.org/tpo/core/tor/-/issues/40918 'Disable a sandbox unit test that is failing on Debian Sid'
I just upgraded some relays.
unless our gitlab-ci actually manages to build a whole set, you won't see packages on deb.tpo.
cf.
https://gitlab.torproject.org/tpo/core/debian/tor/-/pipelines?scope=all&... ge=1&ref=debian-main
some of these are actual tor building issues, like https://gitlab.torproject.org/tpo/core/debian/tor/-/jobs/479068
| sandbox/opendir_dirname: [forking] | | FAIL ../src/test/test_sandbox.c:266: opendir: Operation not permitted | [1] | [opendir_dirname FAILED] | | sandbox/chmod_filename: [forking] OK
On 2/15/24 17:16, lists@for-privacy.net wrote:
I let nightly's upgrade automatically, but not stable. Therefore I have the following config in /etc/apt/50unattended-upgrades
Unattended-Upgrade::Origins-Pattern { ... // Update TorProject's nightly dev packages: (Suite & Codename: tor-nightly-main-bookworm) // apt-cache policy | grep release "o=TorProject,a=tor-nightly-main-bookworm,n=tor-nightly-main-bookworm"; };
I do have (in [1])
Unattended-Upgrade::Origins-Pattern { "origin=*"; }; Unattended-Upgrade::Package-Blacklist { }; Unattended-Upgrade::Automatic-Reboot "true";
which seems to work fine - it upgrades every package from every repo/release if needed.
[1] https://github.com/toralf/tor-relays/blob/main/playbooks/roles/setup/files/5...
-- Toralf
tor-relays@lists.torproject.org
-
boldsuck -
lists＠for-privacy.net
-
Peter Palfrader -
s7r -
Toralf Förster