Node 6E02FDEA15122A492A799A58C4C11D02637B145A is marked as down, but logs display no issues
Hi all, our node sacco.osservatorionessuno.org[1] has been marked down for more than a couple of days. It is at the latest version from Tor's Debian repository and I tried to restart both the service and the server. The process is running, and the 9001 and 9030 ports are open and succesfully reachable from the outside. The notice log I temporarily enabled to look for the issue apparently does not show any warning signs.
Here is an anonymized output of the log:
[notice] Tor 0.4.8.10 opening new log file. [notice] We compiled with OpenSSL XXX and we are running with OpenSSL XXX. These two versions should be binary compatible. [notice] Tor 0.4.8.10 running on Linux with Libevent XXX, OpenSSL XXX, Zlib XXX, Liblzma XXX, Libzstd XXX and Glibc XXX as libc. [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://support.torproject.org/faq/staying-anonymous/ [notice] Read configuration file "/etc/tor/torrc". [notice] Based on detected system memory, MaxMemInQueues is set to 720 MB. You can override this by setting MaxMemInQueues by hand. [notice] Opening Socks listener on 127.0.0.1:9050 [notice] Opened Socks listener connection (ready) on 127.0.0.1:9050 [notice] Opening OR listener on XXX:9001 [notice] Opened OR listener connection (ready) on XXX:9001 [notice] Opening OR listener on [XXX]:9001 [notice] Opened OR listener connection (ready) on [XXX]:9001 [notice] Opening Directory listener on XXX:9030 [notice] Opened Directory listener connection (ready) on XXX:9030 [notice] Opening Directory listener on [XXX]:9030 [notice] Opened Directory listener connection (ready) on [XXX]:9030 [notice] Parsing GEOIP IPv4 file /usr/share/tor/geoip. [notice] Parsing GEOIP IPv6 file /usr/share/tor/geoip6. [notice] Configured to measure statistics. Look for the *-stats files that will first be written to the data directory in 24 hours from now. [notice] Your Tor server's identity key fingerprint is 'sacco XXX' [notice] Your Tor server's identity key ed25519 fingerprint is 'sacco XXX' [notice] Bootstrapped 0% (starting): Starting [notice] Starting with guard context "default" [notice] Bootstrapped 5% (conn): Connecting to a relay [notice] Unable to find IPv4 address for ORPort 9001. You might want to specify IPv6Only to it or set an explicit address or set Address. [notice] Bootstrapped 10% (conn_done): Connected to a relay [notice] Bootstrapped 14% (handshake): Handshaking with a relay [notice] Bootstrapped 15% (handshake_done): Handshake with a relay done [notice] Bootstrapped 75% (enough_dirinfo): Loaded enough directory info to build circuits [notice] Bootstrapped 90% (ap_handshake_done): Handshake finished with a relay to build circuits [notice] Bootstrapped 95% (circuit_create): Establishing a Tor circuit [notice] External address seen and suggested by a directory authority: XXX [notice] Bootstrapped 100% (done): Done [notice] Now checking whether IPv4 ORPort XXX:9001 is reachable... (this may take up to 20 minutes -- look for log messages indicating success) [notice] Now checking whether IPv6 ORPort [XXX]:9001 is reachable... (this may take up to 20 minutes -- look for log messages indicating success) [notice] Self-testing indicates your ORPort XXX:9001 is reachable from the outside. Excellent. [notice] Performing bandwidth self-test...done. [notice] Self-testing indicates your ORPort [XXX]:9001 is reachable from the outside. Excellent. Publishing server descriptor. [notice] Heartbeat: It seems like we are not in the cached consensus. [notice] Heartbeat: Tor's uptime is XX hours, with XX circuits open. I've sent XX and received XX. I've received XXX connections on IPv4 and XXX on IPv6. I've made XXX connections with IPv4 and XXX with IPv6. [notice] While bootstrapping, fetched this many bytes: XXX (microdescriptor fetch) [notice] While not bootstrapping, fetched this many bytes: XXX (server descriptor fetch); XXX (server descriptor upload); XXX (consensus network-status fetch); XXX (microdescriptor fetch) [notice] Circuit handshake stats since last time: X/X TAP, X/X NTor. [notice] Since startup we initiated XXX and received XXX v1 connections; initiated 0 and received 0 v2 connections; initiated XXX and received XXX v3 connections; initiated XXX and received XXX v4 connections; initiated XXX and received XXX v5 connections. [notice] Heartbeat: DoS mitigation since startup: 0 circuits killed with too many cells, 0 circuits rejected, 0 marked addresses, 0 marked addresses for max queue, 0 same address concurrent connections rejected, 0 connections rejected, 0 single hop clients refused, 0 INTRODUCE2 rejected. [notice] Your network connection speed appears to have changed. Resetting timeout to XXXms after XXX timeouts and XXX buildtimes.
We would appreciate any hint on how to proceed to debug the issue in order to keep the server useful.
Thank you Giuio
[1] - https://metrics.torproject.org/rs.html#details/6E02FDEA15122A492A799A58C4C11...
On Thu, 14 Mar 2024 10:47:02 +0100 0N ODV via tor-relays tor-relays@lists.torproject.org wrote:
Hi all, our node sacco.osservatorionessuno.org[1] has been marked down for more than a couple of days. It is at the latest version from Tor's Debian repository and I tried to restart both the service and the server. The process is running, and the 9001 and 9030 ports are open and succesfully reachable from the outside. The notice log I temporarily enabled to look for the issue apparently does not show any warning signs.
The reason is that your IPv6 2a05:541:112:31::1 is not accessible. Did you check port 9001 on IPv6 from the outside?
Host Loss% Snt Last Avg Best Wrst StDev 1. 2001:bc8:6010:215::1 0.0% 10 1.9 1.7 0.8 2.4 0.6 2. 2001:bc8:400:100::bc 50.0% 10 4.3 1.9 0.8 4.3 1.4 3. 2001:bc8:400:100::b9 0.0% 10 2.8 2.2 1.4 3.8 0.7 4. ??? 5. ??? 6. ??? 7. rostelecom-as-as12389.e0-3.core3.fra2.he.net 88.9% 10 12.9 12.9 12.9 12.9 0.0 8. rt-47-msk-ix.rostelecom.ru 30.0% 10 48.5 48.7 48.3 49.2 0.3 9. 2a01:620:1:220e::2 0.0% 10 50.6 51.0 49.9 56.8 2.0 10. 2001:1bb0:8000:1::30 0.0% 10 49.2 53.6 48.4 70.4 7.7 11. 2001:1bb0:e000:16::2 88.9% 10 57.6 57.6 57.6 57.6 0.0 12. ??? 13. 2001:1bb0:e000:16::2 77.8% 10 67.5 63.4 59.4 67.5 5.7 14. ???
# nc6 -vvv 31.129.22.65 9001 Connection to 31.129.22.65 9001 port [tcp/*] succeeded!
# nc6 -vvv 2a05:541:112:31::1 9001 nc6: connect to 2a05:541:112:31::1 port 9001 (tcp) failed: No route to host
Here is an anonymized output of the log:
[notice] Tor 0.4.8.10 opening new log file. [notice] We compiled with OpenSSL XXX and we are running with OpenSSL XXX. These two versions should be binary compatible. [notice] Tor 0.4.8.10 running on Linux with Libevent XXX, OpenSSL XXX, Zlib XXX, Liblzma XXX, Libzstd XXX and Glibc XXX as libc. [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://support.torproject.org/faq/staying-anonymous/ [notice] Read configuration file "/etc/tor/torrc". [notice] Based on detected system memory, MaxMemInQueues is set to 720 MB. You can override this by setting MaxMemInQueues by hand. [notice] Opening Socks listener on 127.0.0.1:9050 [notice] Opened Socks listener connection (ready) on 127.0.0.1:9050 [notice] Opening OR listener on XXX:9001 [notice] Opened OR listener connection (ready) on XXX:9001 [notice] Opening OR listener on [XXX]:9001 [notice] Opened OR listener connection (ready) on [XXX]:9001 [notice] Opening Directory listener on XXX:9030 [notice] Opened Directory listener connection (ready) on XXX:9030 [notice] Opening Directory listener on [XXX]:9030 [notice] Opened Directory listener connection (ready) on [XXX]:9030 [notice] Parsing GEOIP IPv4 file /usr/share/tor/geoip. [notice] Parsing GEOIP IPv6 file /usr/share/tor/geoip6. [notice] Configured to measure statistics. Look for the *-stats files that will first be written to the data directory in 24 hours from now. [notice] Your Tor server's identity key fingerprint is 'sacco XXX' [notice] Your Tor server's identity key ed25519 fingerprint is 'sacco XXX' [notice] Bootstrapped 0% (starting): Starting [notice] Starting with guard context "default" [notice] Bootstrapped 5% (conn): Connecting to a relay [notice] Unable to find IPv4 address for ORPort 9001. You might want to specify IPv6Only to it or set an explicit address or set Address. [notice] Bootstrapped 10% (conn_done): Connected to a relay [notice] Bootstrapped 14% (handshake): Handshaking with a relay [notice] Bootstrapped 15% (handshake_done): Handshake with a relay done [notice] Bootstrapped 75% (enough_dirinfo): Loaded enough directory info to build circuits [notice] Bootstrapped 90% (ap_handshake_done): Handshake finished with a relay to build circuits [notice] Bootstrapped 95% (circuit_create): Establishing a Tor circuit [notice] External address seen and suggested by a directory authority: XXX [notice] Bootstrapped 100% (done): Done [notice] Now checking whether IPv4 ORPort XXX:9001 is reachable... (this may take up to 20 minutes -- look for log messages indicating success) [notice] Now checking whether IPv6 ORPort [XXX]:9001 is reachable... (this may take up to 20 minutes -- look for log messages indicating success) [notice] Self-testing indicates your ORPort XXX:9001 is reachable from the outside. Excellent. [notice] Performing bandwidth self-test...done. [notice] Self-testing indicates your ORPort [XXX]:9001 is reachable from the outside. Excellent. Publishing server descriptor. [notice] Heartbeat: It seems like we are not in the cached consensus. [notice] Heartbeat: Tor's uptime is XX hours, with XX circuits open. I've sent XX and received XX. I've received XXX connections on IPv4 and XXX on IPv6. I've made XXX connections with IPv4 and XXX with IPv6. [notice] While bootstrapping, fetched this many bytes: XXX (microdescriptor fetch) [notice] While not bootstrapping, fetched this many bytes: XXX (server descriptor fetch); XXX (server descriptor upload); XXX (consensus network-status fetch); XXX (microdescriptor fetch) [notice] Circuit handshake stats since last time: X/X TAP, X/X NTor. [notice] Since startup we initiated XXX and received XXX v1 connections; initiated 0 and received 0 v2 connections; initiated XXX and received XXX v3 connections; initiated XXX and received XXX v4 connections; initiated XXX and received XXX v5 connections. [notice] Heartbeat: DoS mitigation since startup: 0 circuits killed with too many cells, 0 circuits rejected, 0 marked addresses, 0 marked addresses for max queue, 0 same address concurrent connections rejected, 0 connections rejected, 0 single hop clients refused, 0 INTRODUCE2 rejected. [notice] Your network connection speed appears to have changed. Resetting timeout to XXXms after XXX timeouts and XXX buildtimes.
We would appreciate any hint on how to proceed to debug the issue in order to keep the server useful.
Thank you Giuio
[1] - https://metrics.torproject.org/rs.html#details/6E02FDEA15122A492A799A58C4C11... _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Hi Roman,
On 14/03/2024 13:20, Roman Mamedov wrote:
The reason is that your IPv6 2a05:541:112:31::1 is not accessible. Did you check port 9001 on IPv6 from the outside?
Thank you, I did not check indeed. Something has probably changed on the provider side, since no configuration has been changed in the recent weeks, and weirdly enough the server can still exit in ipv6, but does not answer to pings from outside. I have temporarily disabled IPv6Exit and the respective listening ports while we investigate.
Cheers Giulio
Hi,
this is a suggestion for improving page :
https://community.torproject.org/relay/setup/bridge/post-install/
where the page states :
See the file |obfs4_bridgeline.txt|, which is found inside Tor Data Directory, for example, in Debian/Ubuntu |/var/lib/tor/pt_state/obfs4_bridgeline.txt| or FreeBSD |/var/db/tor/pt_state/obfs4_bridgeline.txt|.
I believe this is now time to keep a standard with the pattern of respect that the recipe has had for the diversity of OS-communities and push this to state in hard:
See the file |obfs4_bridgeline.txt|, which is found inside Tor Data Directory: in Debian / Ubuntu / Fedora |/var/lib/tor/pt_state/obfs4_bridgeline.txt|
in FreeBSD |/var/db/tor/pt_state/obfs4_bridgeline.txt|.
(My personal experience is that under DEBIAN BOOKWORM (12) at least, the directory |/var/lib/tor/pt_state/ |DOES NOT EXIST this is infuriating when having set up the entire Bridge in deep study of the torproject recipe, the fatal outcome is that the Bridge is running yet the Bridge line is uncomposable for publication: Debian 12 is a standard, and Debian 11 becomes a dangerous OS to rely on: the bridge-line pt_state folder-issue must be urgently resolved!).
also the page :
source url : https://bridges.torproject.org/info
does not give clean examples of the exact torrc statement with the present double-quote (eg. "Settings") and this is very confusing for those who capitalize the first letter when the standard on pages I visit are often all in small letters:
|BridgeDistribution moat
|
would illustrate the standard to adopt and avoid potentially wasting time at every new Bridge being attempted by operators.
Perhaps keeping this good habit of looking what else is, to secure a basic tor server (after all the actual recipe mentions Unbound, ufw, firewalld, ... ) the torproject could push a step further and remind a concise, minimal yet expected standard (for every OS) - in changing openssh ssh port 22 for any other port TODO,
- in setting up ed25519 only,
- in setting up fail2ban to jail any TOR EXIT IPs, ... .
Truely, with little experience of mine, INFLATION BOMBS and local infrastructure hacking attacks have repeatedly used Tor to (dDOS-) attack Tor Relays from EXIT nodes.
Carlos.
tor-relays@lists.torproject.org
-
0N ODV -
Carlos -
Roman Mamedov