On Sun, Jun 04, 2017 at 07:14:06PM -0500, Scott Bennett wrote:

 Which versions are the Running votes coming from versus the non-Running?

You can see the votes at https://www.seul.org/~arma/moria1-v3-status-votes

 I have a few commands in a crontab entry that extract relay IP addresses

from the most recently received consensus, sort them, and load them into a table in pf. They run every 15 minutes. Anything coming from the addresses in the table is immediately passed. Anything not passed gets checked against a much larger table of probers, attempted intruders, etc. and is blocked if it matches a table entry.

Well heck. Sounds like we have a winner here.

Turn off your weird censorship infrastructure, confirm that things start working after a while, and now you have something to debug. :)

--Roger

Roger Dingledine arma@mit.edu wrote:

Or at least partially solved, at any rate. The problem is solved, but the mystery remains.

On Sun, Jun 04, 2017 at 07:14:06PM -0500, Scott Bennett wrote:

...

 Which versions are the Running votes coming from versus the non-Running?

You can see the votes at https://www.seul.org/~arma/moria1-v3-status-votes

...

 I have a few commands in a crontab entry that extract relay IP addresses

from the most recently received consensus, sort them, and load them into a table in pf. They run every 15 minutes. Anything coming from the addresses in the table is immediately passed. Anything not passed gets checked against a much larger table of probers, attempted intruders, etc. and is blocked if it matches a table entry.

Well heck. Sounds like we have a winner here.

Turn off your weird censorship infrastructure, confirm that things start working after a while, and now you have something to debug. :)

The misuse of the word "censorship" aside, I tried disabling pf and restarting tor. To my surprise, the authorities connected to my relay successfully and distributed its information in subsequent consensuses! I do not know why, which is the remaining mystery. However, having to turn off one's firewall in order to run tor is not a solution to the problem. But, while going back through the logs, I noticed the following among tor's startup messages.

Jun 05 23:25:46.849 [warn] OpenSSL version from headers does not match the version we're running with. If you get weird crashes, that might be why. (Compiled with 100020bf: OpenSSL 1.0.2k 26 Jan 2017; running with 100020cf: OpenSSL 1.0.2l 25 May 2017). Jun 05 23:25:46.895 [notice] Tor 0.3.0.7 (git-cfd9c1bdc0582656) running on FreeBSD with Libevent 2.1.8-stable, OpenSSL 1.0.2l and Zlib 1.2.8.

The date of 25 May 2017 means that my last run of 0.3.0.6 was using an older OpenSSL version, but 0.3.0.7 was using the newer one, although it had been compiled with the earlier version shown in the first message. I'm not sure how that might interact badly with pf at all. However, I rebuilt 0.3.0.7, stopped the running copy, enabled pf, and restarted tor to run from the rebuilt version. Enough authorities must be happy with it because they are now distributing its information in the consensus and also its most recent descriptor. In summary, the problem is resolved, but the mystery remains unsolved. What I think I know is that the mismatched openssl stub and library versions somehow interact with pf and pre-0.3.0.7 tor authorities in some mysterious manner that causes connections from those authorities to my relay to fail, but connections from 0.3.0.7 authorities to work just fine. Having the stub version and the library version be identical apparently avoids the erroneous interaction(s).

Scott Bennett, Comm. ASMELG, CFIAG ********************************************************************** * Internet: bennett at sdf.org *xor* bennett at freeshell.org * *--------------------------------------------------------------------* * "A well regulated and disciplined militia, is at all times a good * * objection to the introduction of that bane of all free governments * * -- a standing army." * * -- Gov. John Hancock, New York Journal, 28 January 1790 * **********************************************************************

teor teor2345@gmail.com wrote:

...

On 8 Jun 2017, at 18:44, Scott Bennett bennett@sdf.org wrote:

Roger Dingledine arma@mit.edu wrote:

Or at least partially solved, at any rate.  The problem is solved,

but the mystery remains.

...

On Sun, Jun 04, 2017 at 07:14:06PM -0500, Scott Bennett wrote:

...

Which versions are the Running votes coming from versus the non-Running?

You can see the votes at https://www.seul.org/~arma/moria1-v3-status-votes

...

I have a few commands in a crontab entry that extract relay IP addresses

from the most recently received consensus, sort them, and load them into a table in pf. They run every 15 minutes. Anything coming from the addresses in the table is immediately passed. Anything not passed gets checked against a much larger table of probers, attempted intruders, etc. and is blocked if it matches a table entry.

Well heck. Sounds like we have a winner here.

Turn off your weird censorship infrastructure, confirm that things start working after a while, and now you have something to debug. :)

The misuse of the word "censorship" aside, I tried disabling pf and

restarting tor. To my surprise, the authorities connected to my relay successfully and distributed its information in subsequent consensuses! I do not know why, which is the remaining mystery. However, having to turn off one's firewall in order to run tor is not a solution to the problem.

If your firewall is blocking connections that tor needs to make, turning off that rule is precisely the solution to the problem.

As noted more than once previously, the pf rules *pass* all traffic from relay addresses *first*, so that traffic has already gone on to tor before the block list is applied. As also noted previously, the rules had not (and still have not) been changed in a long time. They were working just fine for years, and then suddenly they were not. Now they are working just fine again since tor got rebuilt, so they are not the problem.

...

What I think I know is that the mismatched openssl stub and library versions somehow interact with pf and pre-0.3.0.7 tor authorities in some mysterious manner that causes connections from those authorities to my relay to fail, but connections from 0.3.0.7 authorities to work just fine. Having the stub version and the library version be identical apparently avoids the erroneous interaction(s).

It's possible. But to confirm, you'd need to provide the sections of the firewall log that show what happens when a directory authority tries to check the reachability of your relay.

That stuff isn't logged because it would soon exhaust the available disk space with garbage.

(As an aside, I'd encourage you not to keep logs at this level on an ongoing basis, they would contain client IP addresses and connection times.)

That's one reason; the other is the space and kernel overhead requirements.

Or perhaps your firewall collected enough state to want to block tor while it was running, and lost that state when you turned it off and on again.

I don't understand how an application could directly interfere with the packet filter's operation. I've previously posted the rules. As I described, it appears to be a problem of openssl misbehaving, apparently due to mismatched stub and library versions, though I don't understand how openssl trouble could occur with pf enabled and not occur with pf disabled. In any case, rebuilding tor so that the stub and library versions match has eliminated the problem. tor now plays nicely with the authorities, regardless of whether pf is enabled. Now, having said that, I should mention that tor does mess around with /dev/bpf, at least during startup, which is one reason why tor has to have root privileges during startup. I've never looked into exactly what it does with /dev/bpf, however, so I have no idea whether that could be involved and caused to malfunction somehow by openssl problems.

You have a complex enough system that there are probably more possibilities neither of us have thought of.

Complex? Compared to what? You've already seen the handful of rules involved, which includes some near duplication of ORPort rules to provide the same protection from miscreants to the DirPort that is given to the ORPort. In a nutshell, the rules look at incoming SYN packets to see if they come from tor relay addresses and immediately lets pass those that do. (By doing this step first, relay traffic is protected from the next step, even if the relay's address has gotten onto the block list, e.g., an exit that has been abused by some cracker or wannabe-cracker.) If they do not, then they are checked against a very large block list and, if found in the block list, are dropped with no RST. (Yes, I know dropping without RST violates the TCP standards, but crackers deserve no courtesies. Slowing them down is good. They can just wait until they time out.) Any SYN packets destined to the ORPort or DirPort that survive at this point are either from clients or from relays that have just started up and therefore do not yet appear in the relays list used in the first step. If a relay is new or has been down long enough to have been omitted from at least one consensus *and* is in the block list, then its SYNs will be dropped until up to 15 minutes after the next consensus that lists it has been downloaded. That is the only, and very temporary, situation in which the rules block traffic from legitimate source addresses. Authorities will rarely, and only briefly (i.e., not much over a couple of hours at most, certainly not *days*), ever be in this situation. There should *never* be a time when a majority of authorities is in this situation simultaneously. It's worth pointing out that, without pf's very efficient protection (or that of some other packet filter), tor itself is stuck with the load of handling useless inbound connection attempts. pf takes that load off of tor when it is caused by IPv4 addresses that are known sources of trouble.

I'd keep an eye on things over the next few weeks.

I think I will be keeping an eye on openssl updates to make sure that tor gets rebuilt each time the openssl library gets replaced, now that I know that openssl updates can subvert one of the advantages of dynamically linked libraries. :-(

Scott Bennett, Comm. ASMELG, CFIAG ********************************************************************** * Internet: bennett at sdf.org *xor* bennett at freeshell.org * *--------------------------------------------------------------------* * "A well regulated and disciplined militia, is at all times a good * * objection to the introduction of that bane of all free governments * * -- a standing army." * * -- Gov. John Hancock, New York Journal, 28 January 1790 * **********************************************************************

Roman Mamedov rm@romanrm.net wrote:

Hi Roman!

On Thu, 08 Jun 2017 09:43:00 -0500 Scott Bennett bennett@sdf.org wrote:

...

 As noted more than once previously, the pf rules *pass* all traffic

from relay addresses *first*, so that traffic has already gone on to tor before the block list is applied.

There are most likely some relays which use a different IP for outgoing connections than what is listed in the consensus, due to multiple IPs or provider multihoming. Your scheme does not seem to account for that, so those connections may fail. In effect you will be leaving the Tor network permanently semi-broken by running a relay while employing such filtering.

You're right. I recall spending a lot of time trying to find a way to get those addresses, so that I could include them in the TorRelays table or some other arrangement to let their traffic through. Unfortunately, tor doesn't publish them anywhere that I could find nor did anything else back at the time I set these rules and tables up. However, I figured most relays probably have only a single WAN interface, so the rules would probably cause few failures. Also, clients don't give up after something like that, but rather continue to try more circuits, so the end user may experience a short delay, but won't actually go unserved in such cases. Consider another case. Users have often complained that running a tor relay results in their IP addresses being blocked by all manner of services around the Internet. The providers of those services say they have suffered attacks originating from tor relays. The project's response was to create an automatically, frequently updated list of IP addresses of exit relays and make that list available for download by anyone wishing to block traffic from tor exits, while allowing traffic from all other relays. That list of addresses suffers the same problem of not including alternative IP addresses for those relays. Even worse, troublesome connections from those alternative addresses *can* be traced back, in some cases, to the exit relay. Once those services have identified the offending traffic as coming from a machine they had been promised by the tor project would be in the downloadable list of exit relay addresses, they may decide that they had been deceived by the tor project, which could lead to many bad things in the future.

In any case I don't think there is any reasonable threat scenario against which you must protect by not just allowing all connections from anywhere to ORPort/DirPort of a Tor relay.

That is something that each system administrator has to determine for the system(s) under his care. My determination for my system apparently differs from your evaluation of your situation. What originally prompted me to look into using a packet filter to block traffic from identifiable sources of trouble was the frequency of attacks on my system. It was so intense at times that even the kernel was complaining in console messages that it was limiting RSTs to 200/s. These messages were happening so fast that syslogd was reducing them in most cases with messages like "last line repeated 1353 times" (not an exact quotation, but I'm sure you've seen syslogd's message flood response before). At first, setting

net.inet.tcp.blackhole=2 net.inet.udp.blackhole=1

seemed to take care of the problem. However, after a couple of years, the messages resumed. China, Malaysia, Nigeria, and a few other countries were being especially persistent and aggravating in the frequency of their attacks. (Just in case, I've since added "net.inet.sctp.blackhole=2" to the above list.) Although providing a tor relay and directory is currently the only thing my system offers to benefit other people, I decided that, if I ever chose to offer other services, I would want to deny those services to the offending parties, too, which is why those block rules apply to all ports on my system and not only to tor's ports. The pass rules apply to traffic destined to tor's ports.

Scott Bennett, Comm. ASMELG, CFIAG ********************************************************************** * Internet: bennett at sdf.org *xor* bennett at freeshell.org * *--------------------------------------------------------------------* * "A well regulated and disciplined militia, is at all times a good * * objection to the introduction of that bane of all free governments * * -- a standing army." * * -- Gov. John Hancock, New York Journal, 28 January 1790 * **********************************************************************

Roger Dingledine arma@mit.edu wrote:

On Thu, Jun 08, 2017 at 05:30:37PM -0500, Scott Bennett wrote:

...

 Consider another case.  Users have often complained that running a tor

relay results in their IP addresses being blocked by all manner of services around the Internet. The providers of those services say they have suffered attacks originating from tor relays. The project's response was to create an automatically, frequently updated list of IP addresses of exit relays and make that list available for download by anyone wishing to block traffic from tor exits, while allowing traffic from all other relays. That list of addresses suffers the same problem of not including alternative IP addresses for those relays. Even worse, troublesome connections from those alternative addresses *can* be traced back, in some cases, to the exit relay. Once those services have identified the offending traffic as coming from a machine they had been promised by the tor project would be in the downloadable list of exit relay addresses, they may decide that they had been deceived by the tor project, which could lead to many bad things in the future.

I think we might have to agree to disagree about a lot of these topics, but I wanted to correct this one.

The bulk exit list: https://check.torproject.org/cgi-bin/TorBulkExitList.py along with TorDNSEL is designed to handle exactly this situation, and it does it pretty well.

Okay, I'll take your word for it, but I'm curious to know the source of the addresses collected if tor doesn't report them.

Scott Bennett, Comm. ASMELG, CFIAG ********************************************************************** * Internet: bennett at sdf.org *xor* bennett at freeshell.org * *--------------------------------------------------------------------* * "A well regulated and disciplined militia, is at all times a good * * objection to the introduction of that bane of all free governments * * -- a standing army." * * -- Gov. John Hancock, New York Journal, 28 January 1790 * **********************************************************************

teor teor2345@gmail.com wrote:

...

On 9 Jun 2017, at 08:30, Scott Bennett bennett@sdf.org wrote:

Roman Mamedov rm@romanrm.net wrote:

...

On Thu, 08 Jun 2017 09:43:00 -0500 Scott Bennett bennett@sdf.org wrote:

...

As noted more than once previously, the pf rules *pass* all traffic

from relay addresses *first*, so that traffic has already gone on to tor before the block list is applied.

There are most likely some relays which use a different IP for outgoing connections than what is listed in the consensus, due to multiple IPs or provider multihoming. Your scheme does not seem to account for that, so those connections may fail. In effect you will be leaving the Tor network permanently semi-broken by running a relay while employing such filtering.

This also excludes any direct client connections to your relay. For an Exit, clients should only connect if UseEntryGuards is 0 (the default is 1, except for Tor2Web and Single Onion Service clients).

It only excludes client connections from IP addresses with a history of offending behavior against my system. That exclusion is an express aim of the packet filter rules I posted. By excluding their access to anything on my system, I keep them from causing me further trouble. Unfortunately, if such troublemaking IP addresses also happen to host relays, then I have to let them have access to the ORPort and DirPort, but I can still exclude them from everything else. Bad behavior should be expected to have consequences. Clients at non-offending IP addresses are allowed to connect without interference.

It also excludes connections from relays that come up and down frequently.

Well, laying aside the fact that we don't want them to be up and down frequently, my TorRelays table is regenerated every 15 minutes from the most recently retrieved consensus. Consensuses are rarely fetched more than once during the same hour, but even if they are and regardless of the time during the clock hour the consensus is retrieved, only a few minutes will pass before the table is produced fresh and loaded into pf. That is much faster than a new consensus is propagated to all relays or clients anyway.

...

You're right.  I recall spending a lot of time trying to find a way to

get those addresses, so that I could include them in the TorRelays table or some other arrangement to let their traffic through. Unfortunately, tor doesn't publish them anywhere that I could find nor did anything else back at the time I set these rules and tables up. However, I figured most relays probably have only a single WAN interface, so the rules would probably cause few failures.

My relays have multiple IP addresses on a single WAN interface. They all use OutboundBindAddress to separate OR and Dir traffic from outbound traffic. And they make up about 1% of Tor guard/middle bandwidth.

So, although the traffic is not separated on that machine, you use the source addresses to allow a router to separate the traffic as it leaves your network? Or is it indeed separated on that machine via routing table entries?

I think you will find this is not an uncommon configuration among high-bandwidth relays.

I will check further into the procedure for which Roger posted a URL to see whether it will indeed give me a list of such addresses that I can use. If so, I will certainly begin running it, scraping the addresses, and merging them with the addresses already going into table creation.

Some directory authorities also have multiple IP addresses.

See above.

...

Also, clients don't give up after something like that, but rather continue to try more circuits, so the end user may experience a short delay, but won't actually go unserved in such cases.

What actually happens depends on a number of factors:

• whether the other relay has successfully connected to your relay,
• whether both relays think the connection is canonical,
• whether either relay has a large exponential backoff on retries.

So in some cases, clients will be unable to connect to your exit via some

Although I allow exits to a small list of places, my relay does not allow general exiting in a manner that would qualify it for an Exit flag on its entry in the consensus, so in that sense, there is no exit here.

middle relays. This reduces your exit traffic, and also reduces the number of different circuit paths available to clients. (Using a wide variety of paths is one of the building blocks of Tor's anonymity.)

My daily exit traffic is ordinarily zero. Non-zero days are so rare as not to be worth calculating their frequency. I no longer remember the last time I saw one, but many versions of tor have come and gone since then.

My point is that there are a lot of moving parts here. And there could be multiple contributing factors, not just OpenSSL.

For the record, we generally suggest the following firewall configuration:

• allow incoming connections to your ORPort and DirPort from any IP
• allow all outgoing connections.

But we might have to agree to disagree here.

Exactly. I block outbound connections to known (to me) malware sites (currently just three entries), but otherwise everything outbound is basically just passed on through. Inbound connections are subject to the rules previously outlined. If I can get my hands on the extra relay addresses, that will be great. I'd love to have them included in the TorRelays table. Like I wrote before, I did try to find them long ago in order to do just that, so it will be good to have them at last.

Scott Bennett, Comm. ASMELG, CFIAG ********************************************************************** * Internet: bennett at sdf.org *xor* bennett at freeshell.org * *--------------------------------------------------------------------* * "A well regulated and disciplined militia, is at all times a good * * objection to the introduction of that bane of all free governments * * -- a standing army." * * -- Gov. John Hancock, New York Journal, 28 January 1790 * **********************************************************************

Roger Dingledine arma@mit.edu wrote:

On Fri, Jun 09, 2017 at 01:05:43AM -0500, Scott Bennett wrote:

...

...

I think you will find this is not an uncommon configuration among high-bandwidth relays.

 I will check further into the procedure for which Roger posted a URL

to see whether it will indeed give me a list of such addresses that I can use. If so, I will certainly begin running it, scraping the addresses, and merging them with the addresses already going into table creation.

It will not do what you want. It is only for exit addresses.

Yes, I saw that after I took a look. However, that shouldn't really be a problem because the Exit addresses are the ones that could have gotten into the block list simply by doing their job. A non-Exit on the block list got their by its operator's own bad behavior. If someone writes to me and promises never to do it again, I'll remove his address from the block list, but on the condition the promise is kept. Likewise if someone writes to me and says his relay used to run as an Exit, but now just runs as entry/middle, I'll also remove his address, again conditionally, because his address may have gotten into the block list when it was an Exit doing its job. Again, the published address would still get a pass anytime it appears in a consensus.

Scott Bennett, Comm. ASMELG, CFIAG ********************************************************************** * Internet: bennett at sdf.org *xor* bennett at freeshell.org * *--------------------------------------------------------------------* * "A well regulated and disciplined militia, is at all times a good * * objection to the introduction of that bane of all free governments * * -- a standing army." * * -- Gov. John Hancock, New York Journal, 28 January 1790 * **********************************************************************

I tried disabling pf and restarting tor. To my surprise, the authorities connected to my relay successfully and distributed its information in subsequent consensuses!

Haha. This is the least surprising thing I've read all week.