Hello!
My non-exit Tor relay "4pyro2eu3org0" in the United Kingdom is served by a consumer FTTC broadband line. It had a rare, unexplained outage on 2016-01-19 between 00:00 and 02:00 UTC. Two other lines with the same provider, terminating in the same local telephone exchange, were unaffected.
When it came back, latency was different (lower) than before, to the first hop (the LNS in Birmingham) suggesting ATM traffic maybe taking a different (faster) route through BT's network *or* just my modem having resynced with different modulation. (Smokeping graph attached).
But what concerns me, are intermittent bursts of packet loss (for about 3 seconds, happening every 10-60 seconds) which have been happening for more than 24 hours since.
My ISP has checked with BT and assures me no maintenance work was scheduled that night; they see in their logs that my PPP session dropped at that time but believe no other customers at my local telephone exchange were affected.
I consider that - whatever may be causing this - this kind of interruption to Tor relay traffic, could make timing attacks easier for an observer. So I'm shutting down my relay until this can be explained and fixed.
At the very least, I should replace the BT Openreach-supplied VDSL modem (Huawei HG521) or its firmware with something I have more control over (and third parties have less control over...)
Here's how the issue looks in mtr (at 1-second intervals); it affects all IP traffic. Maybe this is something benign, but I encourage other relay operators to be vigilant nonetheless.
1. 217.155.40.118 (Tor relay) ................................................................................... 2. 62.3.80.17 (LNS) .........................???......................???............???............... 3. x.x.x.x (test machine on same ISP) .........................???......................???............???...............
and in reverse:
1. x.x.x.x (test machine on same ISP) ................................................................................... 2. 62.3.80.17 (LNS) ................................................................................... 3. 217.155.40.118 (Tor relay) .........................???......................???............???...............
Regards,
Apparently some change to hw/fw/sw/topology was made. If the isp's say nothing changed then you'd want to test and graph every layer2/3 / tunnel hop on the way out to the internet, including to the node of some neighbor on the same fiber or headend as you, as well as the same set of tests from an unaffected neighbor. That's a large % drop in latency, though neither that or pkt loss necessarily means anything nefarious.
So, I've concluded that these little bursts of packet loss are really just some failed equipment of the backhaul carrier, and that it isn't fixed yet is most simply explained by incompetence. I just discovered it affects other connections locally.
So I'm bringing my Tor relay back online, and throttling it up so that it now eats twice as much of their bandwidth \o/
(I thought it prudent to keep the relay offline until I understood what was really going on. Short bursts of packet loss like this, if someone was doing that deliberately with a set pattern, would have been an ideal way to watermark streams going in and out of the Tor network, to do some timing correlations. But I'm satisfied that's not what happened here.)
Stay vigilant!
Regards,
On Wed, Feb 15, 2017 at 06:32:56PM +0000, Steven Chamberlain wrote:
So I'm bringing my Tor relay back online
Great!
Short bursts of packet loss like this, if someone was doing that deliberately with a set pattern, would have been an ideal way to watermark streams going in and out of the Tor network, to do some timing correlations. But I'm satisfied that's not what happened here.
Actually, there's a really interesting research subfield on *undetectable* watermarks -- that is, injecting a signal that is essentially noise unless you know the key that generated it, and then detecting that signal elsewhere in the network.
For papers in this area, check out
https://www.freehaven.net/anonbib/#ndss09-rainbow https://www.freehaven.net/anonbib/#ndss11-swirl https://www.freehaven.net/anonbib/#pets13-flow-fingerprints
--Roger
Steven wrote:
So, I've concluded that these little bursts of packet loss are really just some failed equipment of the backhaul carrier, and that it isn't fixed yet is most simply explained by incompetence.
At first all I read in your graph was the latency drop. But yes now I see the underlying loss shift from green to a darker base. It can be crap hardware / connection or bandwidth overload. In those cases you can open a ticket with whatever data you can find and see what comes back. If you hit gold, ask them for a job if you want one :)
Latency drops are usually cutting legacy routers and needless layers out of the path, or more direct physical routes. Fewer hops are also sometimes seen with mpls or enabled by longer range optics layers in place of routers.
If there are any "stupid" adversaries left, they might show up as increase in latency / loss / hop count, rarely a decrease, unless their TTL editing is broken.
It's also will never be seen as a bad move to shut down a relay if adversary action is suspected until explained otherwise. Reasonable caution and consideration and input sought, whether public or private, is a good thing in this game.
Short bursts of packet loss like this, if someone was doing that deliberately with a set pattern, would have been an ideal way to watermark streams going in and out of the Tor network, to do some timing correlations.
Actually, there's a really interesting research subfield on *undetectable* watermarks -- that is, injecting a signal that is essentially noise unless you know the key that generated it, and then detecting that signal elsewhere in the network.
For papers in this area, check out https://www.freehaven.net/anonbib/#ndss09-rainbow https://www.freehaven.net/anonbib/#ndss11-swirl https://www.freehaven.net/anonbib/#pets13-flow-fingerprints
Roger points out some good papers, and has listed others previously that you can look for in the archives.
I often submit [reasonably or not] that, and as suggested in at least one of those papers... 1) if an adversary can inject / mod undetectables into, or otherwise unrevertably data tag, your encrypted datastream, you need to rethink that stream. 2) you can defeat network traffic analysis / timing / correlation by GPA's, including active fill / loss and modulation attacks upon the wire itself, by establishing fulltime dynamic fill traffic in place of otherwise voids in user demand load, and by enforcing strict expected and negotiated channel parameters with peers upon penalty of rejection, and by reclocking traffic that you pass. (This is meant p2p parameters, not a solution to rogue nodes otherwise privy to or generating underlying encryption layers over user cleartext content.)
I think the research and development fields on this topic regarding application to potential [overlay] networks (not just necessarily tor) are still very much wide open for those who would like to take them up :)
[cc: tor-talk, for the non relay ops aspect]
tor-relays@lists.torproject.org
-
grarpamp -
Roger Dingledine -
Steven Chamberlain