Over Thanksgiving and into early this week, we ran an experiment to test a feedback mechanism to attempt to allocate usage of the Tor network such that the measured stream capacities through all relays became equal: https://gitweb.torproject.org/torflow.git/blob/HEAD:/NetworkScanners/BwAutho...
This experiment failed in 3 ways:
1. It drove many relays down to 0 utilization. Scott Bennett noted this in his post to tor-relays, and at least one 10Mbit relay operator also commented on the traffic drop-off of their node in #tor.
2. It only created one PID 'setpoint' for the entire network, even though different types of nodes see different load characteristics, and despite it being impossible to shift load from an Exit node to a Middle node, for example.
3. It kept allocating bandwidth to some relays (especially Middle and non-default-policy Exits) until they hit INT32_MAX in the consensus, and everything finally exploded. We then shut off the feedback by removing the consensus parameters.
I've made five major changes to try to address these issues:
1. Don't perform multiple rounds of negative feedback for slow nodes.
2. We now group nodes by their flags into four categories (Guard, Middle, Exit, and Guard+Exit), and compute a different PID setpoint for each class.
2. Circuit failure now counts more. Circuit failure is our CPU overload signal, as nodes that hit CPU overload being dropping onionskins and failing extends. Instead of using the circuit success rate as a multiplier against the pid_error, we now actually compute a circ_error similar to the pid_error, and use it as the pid_error if it is more negative. We also now set FastFirstHopPK 0 to ensure that Guard nodes also get tested for circuit failure.
3. Raised the PID setpoint slightly, which should prevent us from piling quite so much weight onto fast relays.
5. Cap feedback via a consensus parameter.
All of these changes are governed by consensus parameters. See: https://gitweb.torproject.org/torflow.git/blob/HEAD:/NetworkScanners/BwAutho... for more details.
The parameters governing feedback are bwauthnsbw=1 and bwauthti. So long as one or both of these are present and non-zero in the consensus parameter list, the feedback experiment is active.
We'll probably be running this next experiment for about a week (or perhaps longer if it doesn't explode and seems to improve performance on https://metrics.torproject.org/performance.html) starting tonight or tomorrow.
Please keep an eye on your relays and tell us if anything unexpected happens over the next week or so.
Thanks!
Am Sat, 3 Dec 2011 18:57:22 -0800 schrieb Mike Perry mikeperry@torproject.org:
Hi,
I've made five major changes to try to address these issues: happens over the next week or so.
Well, somebody has to say it and it seems to be me. The second try is also a complete bust. Since your post the performance is getting worse and worse every day, though my relays behaviour is pretty normal.
Any conclusions from the second try Mike ?
Thus spake Sebastian Urbach (sebastian@urbach.org):
Am Sat, 3 Dec 2011 18:57:22 -0800 schrieb Mike Perry mikeperry@torproject.org:
Hi,
I've made five major changes to try to address these issues: happens over the next week or so.
Well, somebody has to say it and it seems to be me. The second try is also a complete bust. Since your post the performance is getting worse and worse every day, though my relays behaviour is pretty normal.
Yes, I have noticed the huge gain in the metrics.tp.o graph.
Any conclusions from the second try Mike ?
I do not fully understand the cause of it yet, but I did find a rather nasty bug in the treatment of Guard nodes, where we were not properly using the "bwauthmercy" consensus param for them, and were punishing slow guards through multiple rounds of negative feedback. This could skew the metrics data, and could alter the flow of traffic through the rest of the network, but there may be other issues at work, too :/.
I am going to wait until the Guard numbers recover from the bug, which should take 24-48hrs, and then dig deeper next week if the metrics numbers still persist to be bad.
Please ping back then if your relays still appear to be underloaded (or overloaded to the point of emitting warns about having too many circuit creation requests in the log).
Am Fri, 9 Dec 2011 19:56:23 -0800 schrieb Mike Perry mikeperry@torproject.org:
Hi Mike,
I do not fully understand the cause of it yet, but I did find a rather nasty bug in the treatment of Guard nodes, where we were not properly using the "bwauthmercy" consensus param for them, and were punishing slow guards through multiple rounds of negative feedback. This could skew the metrics data, and could alter the flow of traffic through the rest of the network, but there may be other issues at work, too :/.
Im anxious to here more about that issue because i running a guard ;-)
I am going to wait until the Guard numbers recover from the bug, which should take 24-48hrs, and then dig deeper next week if the metrics numbers still persist to be bad.
Seems to get better in the last hours ...
I want to suggest strongly a change for the metrics / performance site. The displayed default size is 50KB and should be changed to 1MB. 50 KB ist out of touch with reality for any service i can think of. Could you please encourage your colleagues to do that ?
Thx.
In adding further info on the topic, I have noticed and looked back over the past couple of weeks and have seen a drop of of usage of about 43% s of today. I am not in the higher bracket as others, but I have been in the 3 bars bracket only up till recently.
Don't know if this will help, but if it does, add it to the rest of the info you have already received.
PS: I am also running Win7 with everything up to date and just started running the latest alpha version for windows actually as far as I can tell, appears to be more stable at this time than the last stable version. ( I like it when the alpha doesn't appear to have any bugs ) Good work guys and gals!!
Jon
Thus spake Jon (torance.ca@gmail.com):
In adding further info on the topic, I have noticed and looked back over the past couple of weeks and have seen a drop of of usage of about 43% s of today. I am not in the higher bracket as others, but I have been in the 3 bars bracket only up till recently.
Don't know if this will help, but if it does, add it to the rest of the info you have already received.
What helps more is your node nick and/or idhex string. If you're not comfortable talking about that publicly, knowing if you are Guard, Exit, Guard+Exit, or just Stable (Middle) node helps.
I actually do expect that this system may cause some slower nodes (especially those with capacities close to or below the network stream average of ~70Kbytes/sec) to experience less traffic. This does not mean the nodes are useless. It just means that we should try to use them infrequently enough such that when they are used, they can provide enough capacity to not be a bottleneck in a circuit.
We are still waiting for the effects of the Guard bug to fully dissipate. I am cautiously optimistic that things are getting better, but we'll need to keep an eye on things for a bit longer to be sure, I suspect.
On Mon, Dec 12, 2011 at 6:52 PM, Mike Perry mikeperry@torproject.org wrote:
Thus spake Jon (torance.ca@gmail.com):
In adding further info on the topic, I have noticed and looked back over the past couple of weeks and have seen a drop of of usage of about 43% s of today. I am not in the higher bracket as others, but I have been in the 3 bars bracket only up till recently.
Don't know if this will help, but if it does, add it to the rest of the info you have already received.
What helps more is your node nick and/or idhex string. If you're not comfortable talking about that publicly, knowing if you are Guard, Exit, Guard+Exit, or just Stable (Middle) node helps.
I actually do expect that this system may cause some slower nodes (especially those with capacities close to or below the network stream average of ~70Kbytes/sec) to experience less traffic. This does not mean the nodes are useless. It just means that we should try to use them infrequently enough such that when they are used, they can provide enough capacity to not be a bottleneck in a circuit.
We are still waiting for the effects of the Guard bug to fully dissipate. I am cautiously optimistic that things are getting better, but we'll need to keep an eye on things for a bit longer to be sure, I suspect.
-- Mike Perry
Mike,
looking at the network status for my node nick today, it is showing that at last senses check, it was shown at 260 KBs with status as a guard
Thus spake Sebastian Urbach (sebastian@urbach.org):
Seems to get better in the last hours ...
I want to suggest strongly a change for the metrics / performance site. The displayed default size is 50KB and should be changed to 1MB. 50 KB ist out of touch with reality for any service i can think of. Could you please encourage your colleagues to do that ?
We want to optimize primarily for the low-latency web case, not bulk downloading.
The one exception to this is Youtube/user-generated video, which I agree does fall more in the 1MB case. However, the state of web browser tech for user-generated video still sucks in the absence of Flash, so it is still not yet our primary focus.
On Sat, Dec 03, 2011 at 06:57:22PM -0800, Mike Perry wrote:
We'll probably be running this next experiment for about a week (or perhaps longer if it doesn't explode and seems to improve performance on https://metrics.torproject.org/performance.html) starting tonight or tomorrow.
Please keep an eye on your relays and tell us if anything unexpected happens over the next week or so.
we're seeing a pretty significant rise in throughput on noisetor01, we peaked at 530 Mbps at around 2011-12-09 17:00 PST, versus previous peaks:
2011-12-08 21:00 480 Mbps 2011-12-07 21:00 470 Mbps 2011-12-06 22:00 410 Mbps
(there's a really horrific dropoff to zero around 13:00 on the 6th, and it clearly took a long time for throughput to recover, which impacted throughput that evening.)
Earlier peaks are in the 350 - 450 Mbps range.
-andy
On 10.12.2011 11:30, Andy Isaacson wrote:
Please keep an eye on your relays and tell us if anything unexpected happens over the next week or so.
we're seeing a pretty significant rise in throughput on noisetor01, we peaked at 530 Mbps at around 2011-12-09 17:00 PST, versus previous peaks:
I don't see any noticable change in bandwidth on our Gbit relays.
( http://nforce1.torservers.net/vnstat_d.png http://nforce2.torservers.net/vnstat_d.png http://axigy1.torservers.net/vnstat_d.png )
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 12/10/2011 6:56 AM, Moritz Bartl wrote:
On 10.12.2011 11:30, Andy Isaacson wrote:
Please keep an eye on your relays and tell us if anything unexpected happens over the next week or so.
we're seeing a pretty significant rise in throughput on noisetor01, we peaked at 530 Mbps at around 2011-12-09 17:00 PST, versus previous peaks:
I don't see any noticable change in bandwidth on our Gbit relays.
( http://nforce1.torservers.net/vnstat_d.png http://nforce2.torservers.net/vnstat_d.png http://axigy1.torservers.net/vnstat_d.png )
Moritz,
Any specific tips on how you're pushing 700Mbps from a single instance (I assume that's what that's indicating) beyond what's on your wiki? With AES-NI and most (maybe all) of the tweaks I've found on your pages (awesome resource, by the way, thanks for that!) the best I've been able to manage is ~250Mbps on a single Tor instance, I've had to use multiple instances on the same hardware to try to get any closer to filling Gbit.
Thanks, Tim
- -- Tim Wilde, Senior Software Engineer, Team Cymru, Inc. twilde@cymru.com | +1-847-378-3333 | http://www.team-cymru.org/
On 12.12.2011 20:18, Tim Wilde wrote:
Any specific tips on how you're pushing 700Mbps from a single instance (I assume that's what that's indicating) beyond what's on your wiki? With AES-NI and most (maybe all) of the tweaks I've found on your pages (awesome resource, by the way, thanks for that!) the best I've been able to manage is ~250Mbps on a single Tor instance, I've had to use multiple instances on the same hardware to try to get any closer to filling Gbit.
Thanks!
I try to keep everything I do documented on that wiki. All these servers run four instances of Tor each (one per core) and traffic is accounted for in total. Also, keep in mind that vnstat counts both incoming and outgoing traffic, so 700Mbps in vnstat are really only 375 per direction.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 12/12/2011 3:21 PM, Moritz Bartl wrote:
I try to keep everything I do documented on that wiki. All these servers run four instances of Tor each (one per core) and traffic is accounted for in total. Also, keep in mind that vnstat counts both incoming and outgoing traffic, so 700Mbps in vnstat are really only 375 per direction.
Ah, okay, thanks for the clarification, I was thinking those numbers were for single Tor instances. That makes me feel a lot better then, especially with the combination of directions. :) I'm pushing around 600Mb/sec total in+out on my piece of bit iron so I'm much closer to the same ballpark than I thought. Thanks, and thanks again for your documentation!
Regards, Tim
- -- Tim Wilde, Senior Software Engineer, Team Cymru, Inc. twilde@cymru.com | +1-847-378-3333 | http://www.team-cymru.org/
Thus spake Tim Wilde (twilde@cymru.com):
I try to keep everything I do documented on that wiki. All these servers run four instances of Tor each (one per core) and traffic is accounted for in total. Also, keep in mind that vnstat counts both incoming and outgoing traffic, so 700Mbps in vnstat are really only 375 per direction.
Ah, okay, thanks for the clarification, I was thinking those numbers were for single Tor instances. That makes me feel a lot better then, especially with the combination of directions. :) I'm pushing around 600Mb/sec total in+out on my piece of bit iron so I'm much closer to the same ballpark than I thought. Thanks, and thanks again for your documentation!
Moritz, Andy, Tim, and others with Gbit+ Guards and/or Exits:
Could you guys ensure you are not running into TCP socket exhaustion on any of your relays? It is a possibility, esp for Guard+Exits with gobs of CPU and gobs of throughput.
I am curious if we will need to do this or not: https://trac.torproject.org/projects/tor/ticket/4709
Thus spake Mike Perry (mikeperry@torproject.org):
Thus spake Tim Wilde (twilde@cymru.com):
I try to keep everything I do documented on that wiki. All these servers run four instances of Tor each (one per core) and traffic is accounted for in total. Also, keep in mind that vnstat counts both incoming and outgoing traffic, so 700Mbps in vnstat are really only 375 per direction.
Ah, okay, thanks for the clarification, I was thinking those numbers were for single Tor instances. That makes me feel a lot better then, especially with the combination of directions. :) I'm pushing around 600Mb/sec total in+out on my piece of bit iron so I'm much closer to the same ballpark than I thought. Thanks, and thanks again for your documentation!
Moritz, Andy, Tim, and others with Gbit+ Guards and/or Exits:
Could you guys ensure you are not running into TCP socket exhaustion on any of your relays? It is a possibility, esp for Guard+Exits with gobs of CPU and gobs of throughput.
I am curious if we will need to do this or not: https://trac.torproject.org/projects/tor/ticket/4709
It looks like Moritz is seeing some evidence of TCP sourceport exhaustion in his Tor logs: "[warn] Error binding network socket: Address already in use".
He's also monitoring TCP connection counts on each IP interface: netstat -ntap | grep $INTERFACE_IP | wc -l
It appears that right now, he's at only about ~10k connections per IP, and not experiencing any log lines at the moment. It is possible this is a transient condition caused by overly-agressive scrapers and/or torrenters who flock to the node for a short while and then move on?
Reports on the recent appearance or prevelance increase of that or other warns from others will be helpful.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 12/12/2011 9:14 PM, Mike Perry wrote:
It looks like Moritz is seeing some evidence of TCP sourceport exhaustion in his Tor logs: "[warn] Error binding network socket: Address already in use".
He's also monitoring TCP connection counts on each IP interface: netstat -ntap | grep $INTERFACE_IP | wc -l
It appears that right now, he's at only about ~10k connections per IP, and not experiencing any log lines at the moment. It is possible this is a transient condition caused by overly-agressive scrapers and/or torrenters who flock to the node for a short while and then move on?
Reports on the recent appearance or prevelance increase of that or other warns from others will be helpful.
Hey Mike,
We're not seeing source port exhaustion, but we are seeing two warns, one of which I haven't been able to nail down:
2011 Dec 13 20:22:07.000|[notice] We stalled too much while trying to write 8542 bytes to address "[scrubbed]". If this happens a lot, either something is wrong with your network connection, or something is wrong with theirs. (fd 409, type Directory, state 2, marked at main.c:990).
2011 Dec 13 22:26:45.000|[warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [18 similar message(s) suppressed in last 60 seconds]
The second warn I figure I should be tuning myself with MaxAdvertisedBandwidth, and it's happening on BigBoy, the relay on this box that's doing the majority of its bandwidth. So I'm not sure if it's anything that your feedback loop should be involved in or not.
The first warn, however, is happening on all four of the other relays on the box, somewhat sporadically (GreenDragon, GoldDragon, WhiteDragon, RedDragon). It seems, though I can't say for sure, that this (or something else I can't quite find) is constraining those four relays and preventing from them growing nearly as high in bandwidth consumption as BigBoy. They're all configured identically (non-exit middle / eventually entry if/when they get Guard). Those four also are different from BigBoy in that they kept names/configs but lost fingerprints/keys (ie they're all specifically Unnamed in the consensus) during a rebuild when trying to milk more bandwidth out of them. My understanding is that this /shouldn't/ make a big difference, but perhaps it is?
I should note that this all could be entirely unrelated to your PID feedback experiments; I coincidentally started experimenting with trying to bring these nodes up to their hardware potential at around the same time, so it could all be a red herring from my experimentation (but I must say, the fact that we're both experimenting together sure does make drawing valid conclusions difficult. :)).
One other data point, I have seen (also sporadically) some indications in my system logs of hardware hangs on the ethernet interface all of this is running through, so I'm slightly suspicious that it's to blame for the *Dragon problems. It doesn't really explain why BigBoy isn't affected though, and I haven't been able to definitively prove anything yet, so I'm just not sure.
Let me know if any more info would be of use (or if anyone has any ideas about that first message).
Thanks, Tim
- -- Tim Wilde, Senior Software Engineer, Team Cymru, Inc. twilde@cymru.com | +1-847-378-3333 | http://www.team-cymru.org/
Thus spake Tim Wilde (twilde@cymru.com):
We're not seeing source port exhaustion, but we are seeing two warns, one of which I haven't been able to nail down:
2011 Dec 13 20:22:07.000|[notice] We stalled too much while trying to write 8542 bytes to address "[scrubbed]". If this happens a lot, either something is wrong with your network connection, or something is wrong with theirs. (fd 409, type Directory, state 2, marked at main.c:990).
Hrm.. Haven't seen this one before...
2011 Dec 13 22:26:45.000|[warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [18 similar message(s) suppressed in last 60 seconds]
Ah, we should be handling this issue with the fix for #1984: https://trac.torproject.org/projects/tor/ticket/1984
The second warn I figure I should be tuning myself with MaxAdvertisedBandwidth, and it's happening on BigBoy, the relay on this box that's doing the majority of its bandwidth. So I'm not sure if it's anything that your feedback loop should be involved in or not.
It's a shame this log message makes such a crazy recommendation wrt MaxAdvertisedBandwidth. But I guess some tweak is better than no tweak. Hopefully we can make this go away without you needing to lower it, though. Can you ping me on IRC if you keep getting these warns after leaving MaxAdvertisedBandwidth alone?
One other data point, I have seen (also sporadically) some indications in my system logs of hardware hangs on the ethernet interface all of this is running through, so I'm slightly suspicious that it's to blame for the *Dragon problems. It doesn't really explain why BigBoy isn't affected though, and I haven't been able to definitively prove anything yet, so I'm just not sure.
This sounds incredibly familiar. What ethernet card + driver version do you have? Some combos of are pretty abysmal about IRQ load balancing and interrupt optimizations, or at least they were on old kernels (which may still apply if you are CentOS).
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 12/13/2011 10:34 PM, Mike Perry wrote:
Thus spake Tim Wilde (twilde@cymru.com):
We're not seeing source port exhaustion, but we are seeing two warns, one of which I haven't been able to nail down:
2011 Dec 13 20:22:07.000|[notice] We stalled too much while trying to write 8542 bytes to address "[scrubbed]". If this happens a lot, either something is wrong with your network connection, or something is wrong with theirs. (fd 409, type Directory, state 2, marked at main.c:990).
Hrm.. Haven't seen this one before...
I've seen LOTS of them. :) When I turned off SafeLogging briefly to see what the scrubbed addresses were it turns out they seem to be the dir auths, if that helps:
2011 Dec 14 16:53:17.000|[notice] We stalled too much while trying to write 273 bytes to address "193.23.244.244". If this happens a lot, either something is wrong with your network connection, or something is wrong with theirs. (fd 4108, type Directory, state 2, marked at main.c:990).
I haven't seen any examples of that error message that were not to a dir auth.
2011 Dec 13 22:26:45.000|[warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [18 similar message(s) suppressed in last 60 seconds]
Ah, we should be handling this issue with the fix for #1984: https://trac.torproject.org/projects/tor/ticket/1984
The second warn I figure I should be tuning myself with MaxAdvertisedBandwidth, and it's happening on BigBoy, the relay on this box that's doing the majority of its bandwidth. So I'm not sure if it's anything that your feedback loop should be involved in or not.
It's a shame this log message makes such a crazy recommendation wrt MaxAdvertisedBandwidth. But I guess some tweak is better than no tweak. Hopefully we can make this go away without you needing to lower it, though. Can you ping me on IRC if you keep getting these warns after leaving MaxAdvertisedBandwidth alone?
Will do.
This sounds incredibly familiar. What ethernet card + driver version do you have? Some combos of are pretty abysmal about IRQ load balancing and interrupt optimizations, or at least they were on old kernels (which may still apply if you are CentOS).
Yeah, some further investigation today indicates that may be the case. :| Running Intel PRO/1000s with the latest E1000E driver (1.6.3-NAPI), I do in fact see what looks like potential interrupt load issues. I've split the relays across to another NIC to see if that helps at all in the relatively short term, long term it looks like a migration away from CentOS for this is called for (it's good for some things, but not this :)). I also rediscovered that the receive packet/flow steering in 2.6.35+ kernels is one of the torservers.net optimizations I haven't done (and can't do on CentOS without a manual kernel installation due to the 2.6.32 kernel it ships with even in 6.0). So it looks like Debian is in this box's future (though I'll try to remember to keep my keys this time :)).
Thanks for the help and suggestions, if I can provide any more info with regards to the stalled writes to the dir auths, let me know.
Thanks, Tim
- -- Tim Wilde, Senior Software Engineer, Team Cymru, Inc. twilde@cymru.com | +1-847-378-3333 | http://www.team-cymru.org/
Hi Mike,
Around the 24 of November the performance graph reported a value below 4 sec. per 50kb, and right now including the last few days we can barely reach 6 sec. per 50kb.
Or in other words, we lost way more than 50 % peformance since that day. Unless you guys are about to really step up the game i strongly suggest to shred the chances made in the last weeks.
The original plan was to speed up things with the modified bandwidth scanners. Its probably safe to say that this experiment is a complete bust.
Thus spake Sebastian Urbach (sebastian@urbach.org):
Around the 24 of November the performance graph reported a value below 4 sec. per 50kb, and right now including the last few days we can barely reach 6 sec. per 50kb.
Or in other words, we lost way more than 50 % peformance since that day. Unless you guys are about to really step up the game i strongly suggest to shred the chances made in the last weeks.
The original plan was to speed up things with the modified bandwidth scanners. Its probably safe to say that this experiment is a complete bust.
Yeah, I agree. I think that it's now clear that both the variance and the mean of the torperf graphs are way above norm, and we don't have much other explaination for it other than the feedback experiment not working. The question is why?
Intuitively, feedback makes a lot of sense. If there is a ton of spare capacity on fast nodes, why shouldn't we try to use it? Similarly, if slow nodes can't keep up with the network throughput average, clearly they are a bottleneck and should have reduced traffic directed at them until they have more spare capacity.
What is causing it to break so badly on the torperf graphs, then? Is it bugs? Is it bad choice of setpoints? Is it that we are using the wrong metric?
Until we have answers for at least some of these questions, I am inclined to keep playing with it.
I've made the following changes this afternoon:
1. On the assumption that we're seeing the huge increase in variance on torperf because faster nodes are only *sometimes* at max capacity, but most of the time have enough spare capacity to get good measurements, I have stopped "filtering" measurements for them (ie I have stopped selecting only the fast measurements). I am still applying filtering to overly punished nodes, so that they don't get punished too much by being paired with slow peers.
2. I have changed the circ failure target from the network average to 0.0% failure.
3. I have changed the Guard feedback interval from 2 weeks to 1 week.
My plan is to let these changes run for another couple days, and if they don't seem to change anything, I plan to try 1 week on, 1 week off cycles of the experiment, to see if we can detect any patterns in exactly when and why torperf starts to go south.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 12/15/2011 3:43 PM, Mike Perry wrote:
I've made the following changes this afternoon:
- On the assumption that we're seeing the huge increase in
variance on torperf because faster nodes are only *sometimes* at max capacity, but most of the time have enough spare capacity to get good measurements, I have stopped "filtering" measurements for them (ie I have stopped selecting only the fast measurements). I am still applying filtering to overly punished nodes, so that they don't get punished too much by being paired with slow peers.
- I have changed the circ failure target from the network average
to 0.0% failure.
- I have changed the Guard feedback interval from 2 weeks to 1
week.
Mike,
Some more data points for you...
* We relieved the interrupt pressure on our big node by splitting across two interfaces, and as a result BigBoy has stabilized pretty well at Bandwidth=1900000 in the consensus, which seems to be right around the point at which it is consuming CPU in such a way that I would expect circuit creation errors if it went much higher (so that seems good).
* However, {Red,Green,Gold,White}Dragon all still seem to be stalled as far as their actual bandwidth usage, though I do see that their bandwidth measurements in the consensus have been changing somewhat (though some have gone down as others have gone up). I don't see any indications of failures in their logs (other than the write failures to the directory authorities periodically).
* Finally, I fired up a new relay on this box yesterday, Ramsgate, which has not gone above Bandwidth=82 yet. Pre-experiment (and even earlier in the experiment) this node would have been able to attract traffic by now (>12 hours after it was started).
Thanks, Tim
- -- Tim Wilde, Senior Software Engineer, Team Cymru, Inc. twilde@cymru.com | +1-847-378-3333 | http://www.team-cymru.org/
Am Thu, 15 Dec 2011 12:43:25 -0800 schrieb Mike Perry mikeperry@torproject.org:
Hi Mike,
Yeah, I agree. I think that it's now clear that both the variance and the mean of the torperf graphs are way above norm, and we don't have much other explaination for it other than the feedback experiment not working. The question is why?
I don't want to be the guy who posts always the bad news ...
I can happily report that the load, since your last mail, increased rapidly to max values and also the performance from the metrics page seems to be increasing very well right now.
If you want to take a look for yourself:
https://metrics.torproject.org/routerdetail.html?fingerprint=0aff5440ae93f2e...
Coincidink ;-) ?
I dont think so ...
I hope that this will relieve a little bit pressure from you, Mike.
Thus spake Sebastian Urbach (sebastian@urbach.org):
Yeah, I agree. I think that it's now clear that both the variance and the mean of the torperf graphs are way above norm, and we don't have much other explaination for it other than the feedback experiment not working. The question is why?
I don't want to be the guy who posts always the bad news ...
I can happily report that the load, since your last mail, increased rapidly to max values and also the performance from the metrics page seems to be increasing very well right now.
If you want to take a look for yourself:
https://metrics.torproject.org/routerdetail.html?fingerprint=0aff5440ae93f2e...
Coincidink ;-) ?
I dont think so ...
I hope that this will relieve a little bit pressure from you, Mike.
Just to be sure, I decided to shut off the experiment on Sunday at 3pm US Pacific time anyways. It looks like the performance has in fact gotten worse since then.
Does this mean the feedback was definitely working? Who the hell knows. But it does seem clear that the transition between the two states is definitely a rocky one that requires patience and repeated trials to sniff out.
I am going to leave it disabled for another week, just to see how it recovers, then turn it back on again, possibly with the addition of https://trac.torproject.org/projects/tor/ticket/4730 if I get a hot minute, but possibly without it, just to watch the transition again.
Robert Ransom is quite insistent on the need for this as well: https://trac.torproject.org/projects/tor/ticket/4708, but that system doesn't really need me to build it, as it can be done independent of the bwauths. I also think #4730 is more likely to capture the problem we were seeing with fast nodes being fast only sometimes, and it will be way less work.
Thanks for bearing with me, but we still might have a ways to go before we really get to the bottom of this one.
I will keep you posted.
Am Mon, 19 Dec 2011 16:32:19 -0800 schrieb Mike Perry mikeperry@torproject.org:
Hi,
Just to be sure, I decided to shut off the experiment on Sunday at 3pm US Pacific time anyways. It looks like the performance has in fact gotten worse since then.
Does this mean the feedback was definitely working? Who the hell knows. But it does seem clear that the transition between the two states is definitely a rocky one that requires patience and repeated trials to sniff out.
Thats exactly what i meant. My impression is that the feedback seems to be working after all. Why do you want to investigate the transition any further ? If the feedback is finally working we probably don't need the transition in the future anymore :-)
I am going to leave it disabled for another week, just to see how it recovers, then turn it back on again, possibly with the addition of https://trac.torproject.org/projects/tor/ticket/4730 if I get a hot minute, but possibly without it, just to watch the transition again.
Robert Ransom is quite insistent on the need for this as well: https://trac.torproject.org/projects/tor/ticket/4708, but that system doesn't really need me to build it, as it can be done independent of the bwauths. I also think #4730 is more likely to capture the problem we were seeing with fast nodes being fast only sometimes, and it will be way less work.
Thanks for bearing with me, but we still might have a ways to go before we really get to the bottom of this one.
Its way less frustrating if you have at least a hunch. But if you are stuck in the middle of nowhere ...
Hi Mike,
Call me crazy but i think that there is a hidden clue somewhere in the actual metrics graph from the last 2 days, regarding the experiment and the performance. <lol>
I've been seeing these messages ("hammer" is routine disk maintenance, which makes other things run slowly):
Dec 17 05:09:27 darner Tor[1138]: Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth conf ig option or choosing a more restricted exit policy. [159 similar message(s) sup pressed in last 60 seconds] Dec 17 05:09:27 darner Tor[1138]: Failed to hand off onionskin. Closing. [159 si milar message(s) suppressed in last 21600 seconds] Dec 17 05:10:22 darner Tor[1138]: Closing stream for '[scrubbed].onion': hidden service is unavailable (try again later). Dec 17 05:16:39 darner kernel: hammer: System has insufficient buffers to rebala nce the tree. nbuf < 3969 Dec 17 05:17:07 darner last message repeated 5 times
I also saw this:
Dec 16 22:11:27 darner Tor[1138]: Tor has not observed any network activity for the past 65 seconds. Disabling circuit build timeout recording. Dec 16 22:11:42 darner Tor[1138]: Tor now sees network activity. Restoring circu it build timeout recording. Network was down for 80 seconds during 3 circuit att empts.
but checking auth.log on my other box shows that this is when I was either sending a huge email or rsyncing some files on an ssh connection from overseas.
My two guard nodes are having a really hard time recovering from the experiments. Especially https://metrics.torproject.org/routerdetail.html?fingerprint=0ed899d7f81d85e... seems to be badly beaten and unable to recover.
My other node https://metrics.torproject.org/routerdetail.html?fingerprint=336477f7a86d22a... have recovered a bit in the last 24 hrs but is still far from previous traffic.
Both nodes have identical configuration and bandwidth, running on the same hardware.
When can we expect the performance to be back to the normal pre-experiment state? I believe it has never been worse than the last couple of weeks in all the years I've been running my relays.
Kind regards,
Tommy Igi
tor-relays@lists.torproject.org
-
Andy Isaacson -
cmeclax-sazri -
Jon -
Mike Perry -
Moritz Bartl -
Sebastian Urbach -
Tim Wilde -
Tommy Igi