Hello all, I have a problem with a Tor Exit.
Tor will not start correctly, but there is nothing in the logs. Here is the info I got.
tor --version Tor version 0.4.6.8. Tor is running on Linux with Libevent 2.1.8-stable, OpenSSL 1.1.1d, Zlib 1.2.11, Liblzma 5.2.4, Libzstd 1.3.8 and Glibc 2.28 as libc. Tor compiled with GCC version 8.3.0
journalctl -u tor --since="14 hours ago" -- Logs begin at Tue 2021-12-14 04:28:56 CET, end at Tue 2021-12-14 10:19:59 CET. -- Dec 14 04:28:59 tor.piratenpartei-nrw.de systemd[1]: Starting Anonymizing overlay network for TCP (multi-instance-master)... Dec 14 04:28:59 tor.piratenpartei-nrw.de systemd[1]: Started Anonymizing overlay network for TCP (multi-instance-master). Dec 14 10:12:00 tor.piratenpartei-nrw.de systemd[1]: tor.service: Succeeded. Dec 14 10:12:00 tor.piratenpartei-nrw.de systemd[1]: Stopped Anonymizing overlay network for TCP (multi-instance-master). Dec 14 10:12:00 tor.piratenpartei-nrw.de systemd[1]: Stopping Anonymizing overlay network for TCP (multi-instance-master)... Dec 14 10:12:00 tor.piratenpartei-nrw.de systemd[1]: Starting Anonymizing overlay network for TCP (multi-instance-master)... Dec 14 10:12:00 tor.piratenpartei-nrw.de systemd[1]: Started Anonymizing overlay network for TCP (multi-instance-master).
In the tor log I have: Dec 14 00:22:25.000 [notice] General overload -> DNS timeouts (162) fraction 1.0484% is above threshold of 1.0000% Dec 14 00:32:25.000 [notice] General overload -> DNS timeouts (154) fraction 1.0664% is above threshold of 1.0000% Dec 14 00:42:25.000 [notice] General overload -> DNS timeouts (140) fraction 1.0721% is above threshold of 1.0000% Dec 14 01:32:57.000 [notice] Interrupt: we have stopped accepting new connections, and will shut down in 30 seconds. Interrupt again to exit now. Dec 14 01:32:58.000 [notice] Delaying directory fetches: We are hibernating or shutting down. Dec 14 01:33:27.000 [notice] Clean shutdown finished. Exiting. Dec 14 10:12:01.000 [notice] Tor 0.4.6.8 opening log file. Dec 14 10:12:01.103 [notice] We compiled with OpenSSL 1010104f: OpenSSL 1.1.1d 10 Sep 2019 and we are running with OpenSSL 1010104f: 1.1.1d. These two versions should be binary compatible. Dec 14 10:12:01.104 [notice] Tor 0.4.6.8 running on Linux with Libevent 2.1.8-stable, OpenSSL 1.1.1d, Zlib 1.2.11, Liblzma 5.2.4, Libzstd 1.3.8 and Glibc 2.28 as libc. Dec 14 10:12:01.104 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
You can see the start was at 04:28:59 but it only started when I did "systemctl restart tor" at 10:12:01.103.
I have this issue for some weeks, maybe since the late summer. I do not restart the Exit that often, so just 3 restarts due to some updates of Debian.
Any idea how to troubleshoot this and what to look for?
Thanks yl
On Tue, Dec 14, 2021 at 11:39:50AM +0100, yl wrote:
Hello all, I have a problem with a Tor Exit.
Tor will not start correctly, but there is nothing in the logs. Here is the info I got.
Is this the debian package? What OS?
You can see the start was at 04:28:59 but it only started when I did "systemctl restart tor" at 10:12:01.103.
My first thought is that somehow during boot Tor fails to start. For example, if at that point in the boot process there is no network, maybe Tor aborts even before it gets to the point of writing anything in its logs. We've definitely had bugs like that over the years.
Any idea how to troubleshoot this and what to look for?
Tor will write to stdout before it switches over to writing to the logs. So whatever init process is starting Tor can see that output. How exactly to get at it... sounds like an adventure diving into how systemd works. :)
--Roger
Hello Roger, thanks for your helpful tip.
On 12/14/21 12:14 PM, Roger Dingledine wrote:
My first thought is that somehow during boot Tor fails to start. For example, if at that point in the boot process there is no network, maybe Tor aborts even before it gets to the point of writing anything in its logs. We've definitely had bugs like that over the years.
this was the one really helpful tip.
Any idea how to troubleshoot this and what to look for?
Tor will write to stdout before it switches over to writing to the logs. So whatever init process is starting Tor can see that output. How exactly to get at it... sounds like an adventure diving into how systemd works.:)
And then the tip that it writes to stdout and might be some other output besides from unit "tor".
I see the problem now, and I think it is like you guessed the network is not ready at that time.
Thanks, will check further.
yl
Are you able to start Tor manually without systemd? Anytime I encounter Tor start issues, I attempt to manually start Tor without the --quiet option to verify whether it's a torrc issue or something else. You might consider increasing the Tor logging level, too. Your existing Tor log shows a DNS timeout error. As I do not operate an exit relay, I don't know whether the DNS timeout error would cause Tor not to start. Respectfully,
Gary— This Message Originated by the Sun. iBigBlue 63W Solar Array (~12 Hour Charge) + 2 x Charmast 26800mAh Power Banks = iPhone XS Max 512GB (~2 Weeks Charged)
On Tuesday, December 14, 2021, 6:06:12 AM PST, yl tor@yl.ms wrote:
Hello Roger, thanks for your helpful tip.
On 12/14/21 12:14 PM, Roger Dingledine wrote:
My first thought is that somehow during boot Tor fails to start. For example, if at that point in the boot process there is no network, maybe Tor aborts even before it gets to the point of writing anything in its logs. We've definitely had bugs like that over the years.
this was the one really helpful tip.
Any idea how to troubleshoot this and what to look for?
Tor will write to stdout before it switches over to writing to the logs. So whatever init process is starting Tor can see that output. How exactly to get at it... sounds like an adventure diving into how systemd works.:)
And then the tip that it writes to stdout and might be some other output besides from unit "tor".
I see the problem now, and I think it is like you guessed the network is not ready at that time.
Thanks, will check further.
yl _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Hello friends,
I see this problem for a long time, but only with some VPS providers from Eastern Europe, with others Tor starts without problems.
As there is a discussion here if maybe some system components are not ready in time, it could also depend on the virtualizer used. Additionally, a slight improvement showed up when I upgraded from Debian 10 to Debian 11 Testing.
That's just for information. It does not seem to be a problem of a single user.
Olaf (some exit nodes)
Am 14.12.21 um 15:52 schrieb Gary C. New via tor-relays:
Are you able to start Tor manually without systemd? Anytime I encounter Tor start issues, I attempt to manually start Tor without the --quiet option to verify whether it's a torrc issue or something else. You might consider increasing the Tor logging level, too. Your existing Tor log shows a DNS timeout error. As I do not operate an exit relay, I don't know whether the DNS timeout error would cause Tor not to start. Respectfully,
Gary— This Message Originated by the Sun. iBigBlue 63W Solar Array (~12 Hour Charge)
- 2 x Charmast 26800mAh Power Banks
= iPhone XS Max 512GB (~2 Weeks Charged)
On Tuesday, December 14, 2021, 6:06:12 AM PST, yl <tor@yl.ms> wrote:Hello Roger, thanks for your helpful tip.
On 12/14/21 12:14 PM, Roger Dingledine wrote:
My first thought is that somehow during boot Tor fails to start. For example, if at that point in the boot process there is no network, maybe Tor aborts even before it gets to the point of writing anything in its logs. We've definitely had bugs like that over the years.
this was the one really helpful tip.
Any idea how to troubleshoot this and what to look for?
Tor will write to stdout before it switches over to writing to the logs. So whatever init process is starting Tor can see that output. How exactly to get at it... sounds like an adventure diving into how systemd works.:)
And then the tip that it writes to stdout and might be some other output besides from unit "tor".
I see the problem now, and I think it is like you guessed the network is not ready at that time.
Thanks, will check further.
yl _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Hello, as there were some follow up emails let me explain the problem.
The standard Debian tor@default.service has "After=network.target nss-lookup.target" in it, so one would think network should be fully functional when the services starts, but it was not in my case. At the time when tor started the eno0 did not yet have a IPv6, so it complained that it could not bind to some port in the config, because the IP supplied with the port was not ready yet.
The after that 5 retries to start tor it gets stuck, systemd status tells it is running, but nyx can says it is not running.
After the 5 restarts I saw that ifup had some issue too, which was caused by some config in /etc/network/interfaces, which was working ok and the IPv6 was also reachable, but still the ifup@eno0.service was not working well. I have no idea about how and where to find that service.
However, I changed the "network" parameter in the interfaces config to be supplied with the IPv6, so like 2001:0DB8:0001::0002/64 instead of "network 64" in a separate line. This satisfied ifup it seems, but still it was too slow to be up and running before Tor was started.
Then I searched again and found a quick and dirty (aka lazy) solution for that, in "tor@default.service" I added this line:
ExecStartPre=/bin/sh -c 'until ping -c1 google.com; do sleep 1; done;'
in the "[Service]" section.
I also tried to add "network-online.target" and some "ifup@eno0.service" in the "After=" line above, but I guess it must be somehow different, maybe has to be some target instead.
So with this line a ping of google.com has to work at least once for the service to be started. In combination with the standard "TimeoutStartSec=300" that should work well.
I am sure there is nicer solutions, but why bother.
yours yl
On Tue, Dec 14, 2021 at 09:26:21PM +0100, yl wrote:
The standard Debian tor@default.service has "After=network.target nss-lookup.target" in it, so one would think network should be fully functional when the services starts, but it was not in my case. At the time when tor started the eno0 did not yet have a IPv6, so it complained that it could not bind to some port in the config, because the IP supplied with the port was not ready yet.
Great find.
If it's easy for you to do, can you install a Tor 0.4.5 deb and see if it has the same behavior? If yes, then this is a problem with your particular situation, or a problem with IPv6 that has been there all along. But if the 0.4.5 deb works and the 0.4.6 deb doesn't, then it is a regression, perhaps related to the ipv6 changes we did in 0.4.6, and we should try to track it down.
Thanks, --Roger
tor-relays@lists.torproject.org
-
bobby stickel -
Gary C. New -
Olaf Grimm -
Roger Dingledine -
yl