Hi everybody
Can someone explain the following tor log entry:
Removed 528 bytes by killing 385780 circuits; 0 circuits remain alive. (it's the nicest one, see below)
Memory stays at 3 - 4 GB before and after. Only tor restart gets rid of the memory.
We love to remove 528 bytes.
Tor log becomes 370 MB within 18 hours.
---- Tor 0.3.2.6-alpha (git-87012d076ef58bb9) MaxMemInQueues 2 GB
---- We receive some of these:
Dec 14 00:11:00.000 [notice] Heartbeat: Tor's uptime is 8 days 3:00 hours, with 508694 circuits open. Dec 14 00:30:08.000 [notice] We're low on memory. Killing circuits with over-long queues. (This behavior is controlled by MaxMemInQueues.) Dec 14 00:30:09.000 [notice] Removed 13344 bytes by killing 594572 circuits; 0 circuits remain alive. Also killed 3 non-linked directory connections.
Dec 14 01:11:00.000 [notice] Heartbeat: Tor's uptime is 8 days 4:00 hours, with 67530 circuits open. Dec 14 02:07:16.000 [notice] We're low on memory. Killing circuits with over-long queues. (This behavior is controlled by MaxMemInQueues.) Dec 14 02:07:17.000 [notice] Removed 206448 bytes by killing 484352 circuits; 0 circuits remain alive. Also killed 2 non-linked directory connections.
Dec 14 03:11:00.000 [notice] Heartbeat: Tor's uptime is 8 days 6:00 hours, with 379182 circuits open. Dec 14 03:13:17.000 [notice] We're low on memory. Killing circuits with over-long queues. (This behavior is controlled by MaxMemInQueues.) Dec 14 03:13:17.000 [notice] Removed 528 bytes by killing 385780 circuits; 0 circuits remain alive. Also killed 0 non-linked directory connections.
Dec 14 05:11:00.000 [notice] Heartbeat: Tor's uptime is 8 days 8:00 hours, with 303958 circuits open. Dec 14 05:15:17.000 [notice] We're low on memory. Killing circuits with over-long queues. (This behavior is controlled by MaxMemInQueues.) Dec 14 05:15:18.000 [notice] Removed 528 bytes by killing 317854 circuits; 0 circuits remain alive. Also killed 0 non-linked directory connections.
and lots of:
Dec 14 00:30:22.000 [notice] We're low on memory. Killing circuits with over-long queues. (This behavior is controlled by MaxMemInQueues.) Dec 14 00:30:22.000 [notice] Removed 528 bytes by killing 221 circuits; 0 circuits remain alive. Also killed 0 non-linked directory connections. Dec 14 00:30:22.000 [notice] We're low on memory. Killing circuits with over-long queues. (This behavior is controlled by MaxMemInQueues.) Dec 14 00:30:22.000 [notice] Removed 528 bytes by killing 297 circuits; 0 circuits remain alive. Also killed 0 non-linked directory connections. Dec 14 00:30:22.000 [notice] We're low on memory. Killing circuits with over-long queues. (This behavior is controlled by MaxMemInQueues.) Dec 14 00:30:22.000 [notice] Removed 528 bytes by killing 395 circuits; 0 circuits remain alive. Also killed 0 non-linked directory connections. Dec 14 00:30:22.000 [notice] We're low on memory. Killing circuits with over-long queues. (This behavior is controlled by MaxMemInQueues.) Dec 14 00:30:22.000 [notice] Removed 528 bytes by killing 468 circuits; 0 circuits remain alive. Also killed 0 non-linked directory connections. Dec 14 00:30:22.000 [notice] We're low on memory. Killing circuits with over-long queues. (This behavior is controlled by MaxMemInQueues.) Dec 14 00:30:22.000 [notice] Removed 528 bytes by killing 509 circuits; 0 circuits remain alive. Also killed 0 non-linked directory connections.
On 14 Dec (20:41:56), Felix wrote:
Hi everybody
Can someone explain the following tor log entry:
Removed 528 bytes by killing 385780 circuits; 0 circuits remain alive. (it's the nicest one, see below)
Memory stays at 3 - 4 GB before and after. Only tor restart gets rid of the memory.
We love to remove 528 bytes.
Tor log becomes 370 MB within 18 hours.
Tor 0.3.2.6-alpha (git-87012d076ef58bb9) MaxMemInQueues 2 GB
Do you see a log line that relates to which type of "Scheduler" you are using such as:
[notice] Scheduler type KIST has been enabled.
It is seriously a huge amount of circuit and very little data. For isntance 13344 bytes that is 0.013 MB for 594572 circuits is just weird.
Is there a chance you are being DoS in some capacity? That is bunch of circuits being opened constantly but with no traffic? You would see that with many inbound connections and if they come from non relays also.
Another possibility is that Tor is failing to cleanup inactive circuits but with more information, we can eliminate options more easily.
Thanks! David
We receive some of these:
Dec 14 00:11:00.000 [notice] Heartbeat: Tor's uptime is 8 days 3:00 hours, with 508694 circuits open. Dec 14 00:30:08.000 [notice] We're low on memory. Killing circuits with over-long queues. (This behavior is controlled by MaxMemInQueues.) Dec 14 00:30:09.000 [notice] Removed 13344 bytes by killing 594572 circuits; 0 circuits remain alive. Also killed 3 non-linked directory connections.
Dec 14 01:11:00.000 [notice] Heartbeat: Tor's uptime is 8 days 4:00 hours, with 67530 circuits open. Dec 14 02:07:16.000 [notice] We're low on memory. Killing circuits with over-long queues. (This behavior is controlled by MaxMemInQueues.) Dec 14 02:07:17.000 [notice] Removed 206448 bytes by killing 484352 circuits; 0 circuits remain alive. Also killed 2 non-linked directory connections.
Dec 14 03:11:00.000 [notice] Heartbeat: Tor's uptime is 8 days 6:00 hours, with 379182 circuits open. Dec 14 03:13:17.000 [notice] We're low on memory. Killing circuits with over-long queues. (This behavior is controlled by MaxMemInQueues.) Dec 14 03:13:17.000 [notice] Removed 528 bytes by killing 385780 circuits; 0 circuits remain alive. Also killed 0 non-linked directory connections.
Dec 14 05:11:00.000 [notice] Heartbeat: Tor's uptime is 8 days 8:00 hours, with 303958 circuits open. Dec 14 05:15:17.000 [notice] We're low on memory. Killing circuits with over-long queues. (This behavior is controlled by MaxMemInQueues.) Dec 14 05:15:18.000 [notice] Removed 528 bytes by killing 317854 circuits; 0 circuits remain alive. Also killed 0 non-linked directory connections.
and lots of:
Dec 14 00:30:22.000 [notice] We're low on memory. Killing circuits with over-long queues. (This behavior is controlled by MaxMemInQueues.) Dec 14 00:30:22.000 [notice] Removed 528 bytes by killing 221 circuits; 0 circuits remain alive. Also killed 0 non-linked directory connections. Dec 14 00:30:22.000 [notice] We're low on memory. Killing circuits with over-long queues. (This behavior is controlled by MaxMemInQueues.) Dec 14 00:30:22.000 [notice] Removed 528 bytes by killing 297 circuits; 0 circuits remain alive. Also killed 0 non-linked directory connections. Dec 14 00:30:22.000 [notice] We're low on memory. Killing circuits with over-long queues. (This behavior is controlled by MaxMemInQueues.) Dec 14 00:30:22.000 [notice] Removed 528 bytes by killing 395 circuits; 0 circuits remain alive. Also killed 0 non-linked directory connections. Dec 14 00:30:22.000 [notice] We're low on memory. Killing circuits with over-long queues. (This behavior is controlled by MaxMemInQueues.) Dec 14 00:30:22.000 [notice] Removed 528 bytes by killing 468 circuits; 0 circuits remain alive. Also killed 0 non-linked directory connections. Dec 14 00:30:22.000 [notice] We're low on memory. Killing circuits with over-long queues. (This behavior is controlled by MaxMemInQueues.) Dec 14 00:30:22.000 [notice] Removed 528 bytes by killing 509 circuits; 0 circuits remain alive. Also killed 0 non-linked directory connections.
-- Thanks and cheers, Felix _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Hi
Do you see a log line that relates to which type of "Scheduler" you are using such as:
[notice] Scheduler type KIST has been enabled.
It's Freebsd: [notice] Scheduler type KISTLite has been enabled.
It is seriously a huge amount of circuit and very little data. For isntance 13344 bytes that is 0.013 MB for 594572 circuits is just weird.
Is there a chance you are being DoS in some capacity? That is bunch of circuits being opened constantly but with no traffic? You would see that with many inbound connections and if they come from non relays also.
This happens every some weeks to different relays. Inbound connections are on a normal level. I believe some client(?) tries a DoS type of circuit exhaustion, at least it seems. The number of circuits goes up to over a million. If it goes with much memory consumption the remove/killing comes along.
Another possibility is that Tor is failing to cleanup inactive circuits but with more information, we can eliminate options more easily.
What can I do ? The relay is still on with 3.4 GB. But circuits are pretty relaxed since five hours.
tor-relays@lists.torproject.org
-
David Goulet -
Felix