Hello tor-dev,
This email is to detail the work being done on a framework to do performance measurements for hidden service. I'm looking to get some feedback on the whole idea, improvement and help on possible analysis. If this work is not thrown out the window, I'll definitely do a Wiki page for a detail documentation but for now this is still a bit too early I think.
*** PLEASE NOTE THAT THIS FRAMEWORK IS _NOT_ INTENDED TO BE USED ON THE REAL TOR NETWORK, ONLY TEST NETWORK! *** (I'll even add a check to make sure TestingTorNetwork is set to 1 else tor should not run.)
The whole idea of this effort is to have a way to do very precise measurements of hidden service so we can come up with a performance baseline and have a reliable way to measure our progress on improving HS. We want to run this on large private network such as Shadow in order to get a better understanding of this whole HS subsystem. Finally, this can be extended to all sorts of component in tor in the future to help us analyze the beast that tor has become. :)
This work uses tracing[0] for measurements. For that, I've instrumented the Tor code base with "tracepoints" that allows us to hook a tracer to those so we can collect data and do analysis afterward.
The following branch contains a bunch of tracepoints mostly for HS and uses the LTTng[1] user space tracer. It's based on tor 0.2.5 version because we wanted to measure regression or improvement between 0.2.5 and upcoming 0.2.6 (especially with Andrea's new scheduler work).
https://gitweb.torproject.org/user/dgoulet/tor.git [branch: hs-lttng-025]
To compile this branch you need to install lttng-ust (tracer). It's pretty easy to install, you can find out how here[2]. (You will need version 2.4.0 or above).
./configure --enable-lttng-tracing
In "src/or/trace/events.h", you can find all tracepoints that have been added with the data they collect. It is a bit too tidious to explain them all to you in this email but every "tor_trace_*" line is a tracepoint that you can grep in the code to see their location thus have a clearer picture of what they record.
First, if you see any tracepoints that record something but are not at the right place in the code, you should definitely let me know. Second, telling me where tracepoint should be added also is very useful! No need to understand the full tracing system, just "that function should have a tracepoint to record this", I'll do the rest.
Now to use tracing with Tor and analyze the data, you can find useful information and examples in this repository. (Pretty sure it's missing some documentation so please point it out!)
https://gitorious.org/tor-hs-stats/tor-hs-stats
Here are examples of graph that we can do with the collected data from the traces.
(Introcuction Points Ready Time) https://people.torproject.org/~dgoulet/hs/ip-025.png or (Rendezvous Point Ready Time) https://people.torproject.org/~dgoulet/hs/rp-025.png
Now to make analysis, I've build what is called in the tracing world a "state system". See that as a way to represent different objects in an application that are populated by the data that has been collected with tracing (trace). For this, please take a look at "lttng-analyze.py" in the gitorious repository. That script parses the trace event by event (event is what a tracepoint generated), populates the state system which is analyzed after the trace is fully parsed. It's pretty straight forward and here is the power of the state system. Once build from a trace, we can run *ALL* sorts of analysis on it to measure timings, find bad behaviors, etc..
If that interests you, I encourage you to play with all this and see for yourself what can be achieved thus opening your eyes on the possibilities. :)
Ok, so there is a lot to explain to make things clear. I'm pretty sure I'm forgetting some here. Understanding the benifits of tracing is not that trivial at first glance so I'll be really happy to explain it to anyone asking.
Any feedback is positive feedback! I'm happy to answer any questions. I'll most probably follow up this with a post about some results at some point.
Cheers! David
[0] https://en.wikipedia.org/wiki/Tracing_%28software%29 [1] https://lttng.org [2] https://lttng.org/download/
FYI, these are the stats we have now:
https://trac.torproject.org/projects/tor/ticket/13792#comment:4
Cheers! David
On 17 Dec (20:20:50), David Goulet wrote:
Hello tor-dev,
This email is to detail the work being done on a framework to do performance measurements for hidden service. I'm looking to get some feedback on the whole idea, improvement and help on possible analysis. If this work is not thrown out the window, I'll definitely do a Wiki page for a detail documentation but for now this is still a bit too early I think.
*** PLEASE NOTE THAT THIS FRAMEWORK IS _NOT_ INTENDED TO BE USED ON THE REAL TOR NETWORK, ONLY TEST NETWORK! *** (I'll even add a check to make sure TestingTorNetwork is set to 1 else tor should not run.)
The whole idea of this effort is to have a way to do very precise measurements of hidden service so we can come up with a performance baseline and have a reliable way to measure our progress on improving HS. We want to run this on large private network such as Shadow in order to get a better understanding of this whole HS subsystem. Finally, this can be extended to all sorts of component in tor in the future to help us analyze the beast that tor has become. :)
This work uses tracing[0] for measurements. For that, I've instrumented the Tor code base with "tracepoints" that allows us to hook a tracer to those so we can collect data and do analysis afterward.
The following branch contains a bunch of tracepoints mostly for HS and uses the LTTng[1] user space tracer. It's based on tor 0.2.5 version because we wanted to measure regression or improvement between 0.2.5 and upcoming 0.2.6 (especially with Andrea's new scheduler work).
https://gitweb.torproject.org/user/dgoulet/tor.git [branch: hs-lttng-025]
To compile this branch you need to install lttng-ust (tracer). It's pretty easy to install, you can find out how here[2]. (You will need version 2.4.0 or above).
./configure --enable-lttng-tracing
In "src/or/trace/events.h", you can find all tracepoints that have been added with the data they collect. It is a bit too tidious to explain them all to you in this email but every "tor_trace_*" line is a tracepoint that you can grep in the code to see their location thus have a clearer picture of what they record.
First, if you see any tracepoints that record something but are not at the right place in the code, you should definitely let me know. Second, telling me where tracepoint should be added also is very useful! No need to understand the full tracing system, just "that function should have a tracepoint to record this", I'll do the rest.
Now to use tracing with Tor and analyze the data, you can find useful information and examples in this repository. (Pretty sure it's missing some documentation so please point it out!)
https://gitorious.org/tor-hs-stats/tor-hs-stats
Here are examples of graph that we can do with the collected data from the traces.
(Introcuction Points Ready Time) https://people.torproject.org/~dgoulet/hs/ip-025.png or (Rendezvous Point Ready Time) https://people.torproject.org/~dgoulet/hs/rp-025.png
Now to make analysis, I've build what is called in the tracing world a "state system". See that as a way to represent different objects in an application that are populated by the data that has been collected with tracing (trace). For this, please take a look at "lttng-analyze.py" in the gitorious repository. That script parses the trace event by event (event is what a tracepoint generated), populates the state system which is analyzed after the trace is fully parsed. It's pretty straight forward and here is the power of the state system. Once build from a trace, we can run *ALL* sorts of analysis on it to measure timings, find bad behaviors, etc..
If that interests you, I encourage you to play with all this and see for yourself what can be achieved thus opening your eyes on the possibilities. :)
Ok, so there is a lot to explain to make things clear. I'm pretty sure I'm forgetting some here. Understanding the benifits of tracing is not that trivial at first glance so I'll be really happy to explain it to anyone asking.
Any feedback is positive feedback! I'm happy to answer any questions. I'll most probably follow up this with a post about some results at some point.
Cheers! David
[0] https://en.wikipedia.org/wiki/Tracing_%28software%29 [1] https://lttng.org [2] https://lttng.org/download/
tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
-
David Goulet