Hi, all. While skimming /r/tor on Reddit I saw a couple of posts saying they hadn't been able to get any usable bridges for several days, and thought I'd check into it. I downloaded & installed a fresh copy of the Windows TBB 8.5.1 and had it ask for bridges via Moat, and received 2 (not 3?) obfs4 bridge lines. Here's the Tor log from the "copy to clipboard" button that popped up on the TBB "connecting" dialog when it stalled at about 3/4th of the progress bar:
6/19/19, 10:46:16.735 [NOTICE] DisableNetwork is set. Tor will not make or accept non-control network connections. Shutting down all existing connections. 6/19/19, 10:46:16.735 [NOTICE] Switching to guard context "bridges" (was using "default") 6/19/19, 10:46:16.735 [NOTICE] DisableNetwork is set. Tor will not make or accept non-control network connections. Shutting down all existing connections. 6/19/19, 10:46:16.736 [NOTICE] DisableNetwork is set. Tor will not make or accept non-control network connections. Shutting down all existing connections. 6/19/19, 10:46:16.736 [NOTICE] Opening Socks listener on 127.0.0.1:9150 6/19/19, 10:46:16.736 [NOTICE] Opened Socks listener on 127.0.0.1:9150 6/19/19, 10:46:16.736 [NOTICE] Renaming old configuration file to "C:\Downloads\Tor Browser 8.5.1 - Copy\Browser\TorBrowser\Data\Tor\torrc.orig.1" 6/19/19, 10:46:23.608 [NOTICE] Bootstrapped 5%: Connecting to directory server 6/19/19, 10:46:23.611 [NOTICE] Bootstrapped 10%: Finishing handshake with directory server 6/19/19, 10:46:24.558 [NOTICE] Bootstrapped 15%: Establishing an encrypted directory connection 6/19/19, 10:46:24.748 [NOTICE] Bootstrapped 20%: Asking for networkstatus consensus 6/19/19, 10:46:24.954 [NOTICE] new bridge descriptor 'eldritchworld' (fresh): $<BRIDGE 1 FINGERPRINT REDACTED>~eldritchworld at <BRIDGE 1 IP REDACTED> 6/19/19, 10:46:25.589 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:26.781 [NOTICE] Bootstrapped 25%: Loading networkstatus consensus 6/19/19, 10:46:28.291 [NOTICE] I learned some more directory information, but not enough to build a circuit: We have no usable consensus. 6/19/19, 10:46:28.482 [NOTICE] Bootstrapped 40%: Loading authority key certs 6/19/19, 10:46:28.997 [NOTICE] The current consensus has no exit nodes. Tor can only build internal paths, such as paths to onion services. 6/19/19, 10:46:28.997 [NOTICE] Bootstrapped 45%: Asking for relay descriptors for internal paths 6/19/19, 10:46:28.997 [NOTICE] I learned some more directory information, but not enough to build a circuit: We need more microdescriptors: we have 0/6440, and can only build 0% of likely paths. (We have 100% of guards bw, 0% of midpoint bw, and 0% of end bw (no exits in consensus, using mid) = 0% of path bw.) 6/19/19, 10:46:28.997 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:28.997 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:28.997 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:28.997 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:28.997 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:28.997 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:29.275 [NOTICE] Bootstrapped 50%: Loading relay descriptors for internal paths 6/19/19, 10:46:29.463 [NOTICE] The current consensus contains exit nodes. Tor can build exit and internal paths. 6/19/19, 10:46:29.463 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:29.463 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:29.463 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:29.514 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:29.514 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:29.672 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:29.672 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:29.672 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:30.283 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:30.283 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:30.283 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:30.636 [NOTICE] Bootstrapped 57%: Loading relay descriptors 6/19/19, 10:46:30.636 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:30.636 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:30.636 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:30.636 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:30.636 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:30.684 [NOTICE] Bootstrapped 64%: Loading relay descriptors 6/19/19, 10:46:30.684 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:30.733 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:30.733 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:30.733 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:30.873 [NOTICE] Bootstrapped 73%: Loading relay descriptors 6/19/19, 10:46:30.873 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:30.873 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:30.881 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:30.881 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:30.881 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:30.937 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:30.969 [NOTICE] Ignoring directory request, since no bridge nodes are available yet. 6/19/19, 10:46:44.618 [WARN] Proxy Client: unable to connect to <BRIDGE 2 IP REDACTED> ("general SOCKS server failure")
It looks like bridge 1 (eldritchworld) gave me bad/incomplete relay info which prevented making circuits, and then Tor failed to connect to the fallback bridge 2 at all. Checking Relay Search, it shows that eldritchworld is a normal healthy-looking bridge that's been up for months with plenty of connections and data being transferred, so I don't know what's wrong there.
So, since the Moat bridges didn't work, I went to the BridgeDB request web page to try that. Clicking the "Just give me bridges" button and doing the captcha returned "Uh oh, spaghettios! There currently aren't any bridges available... Perhaps you should try going back and choosing a different bridge type!". What, no vanilla bridges available at all? Seriously? (As a side note, cutesy errors are pretty annoying when things are failing. Just sayin'.) And as usual the Metrics page still shows about 1000 bridges, so something's badly broken. I know the new bridgedb release doesn't give out ORports if there's a secure PT offered, but last I checked the majority of bridges were just vanilla, so surely there must be enough to put some into whichever bridge pool ring my request was pulled from.
I then tried asking for obfs4 bridges instead, and it did give me one (again, not 3?). With another fresh install of the TBB and entering that bridge line, the log shows:
6/19/19, 10:59:43.912 [NOTICE] DisableNetwork is set. Tor will not make or accept non-control network connections. Shutting down all existing connections. 6/19/19, 10:59:43.912 [NOTICE] Switching to guard context "bridges" (was using "default") 6/19/19, 10:59:43.912 [NOTICE] DisableNetwork is set. Tor will not make or accept non-control network connections. Shutting down all existing connections. 6/19/19, 10:59:43.912 [NOTICE] DisableNetwork is set. Tor will not make or accept non-control network connections. Shutting down all existing connections. 6/19/19, 10:59:43.912 [NOTICE] Opening Socks listener on 127.0.0.1:9150 6/19/19, 10:59:43.912 [NOTICE] Opened Socks listener on 127.0.0.1:9150 6/19/19, 10:59:43.912 [NOTICE] Renaming old configuration file to "C:\Downloads\Tor Browser 8.5.1 - Copy\Browser\TorBrowser\Data\Tor\torrc.orig.1" 6/19/19, 10:59:50.761 [NOTICE] Bootstrapped 5%: Connecting to directory server 6/19/19, 10:59:50.762 [NOTICE] Bootstrapped 10%: Finishing handshake with directory server 6/19/19, 11:00:11.767 [WARN] Proxy Client: unable to connect to <BRIDGE 3 IP> ("general SOCKS server failure")
So this 3rd bridge also failed to connect at all. With the Moat and website requests both returning bad/dead bridges (if any), the last thing to try was the email request channel. I sent "request bridges" to bridges@torproject.org and it replied back "Here are your bridges: None, None, None".
Well, okay then. Maybe if I try again later, which should give me different bridges to use? According to the bridgedb config file checked into the git repo, the rotation periods for the Moat and website bridge pool query positions are both 3 hours. So I waited 3.5 hours and tried those tests again on another fresh TBB install copy, and both Moat and the website returned back to me the exact same bridges as before, which then failed in the same ways as before. Maybe the running server isn't using the checked-in config settings? Waiting another 8 hours (11.5 total since the first try), Moat gave me the same first bridge but finally a different second bridge (still not 3), but that one also failed to connect at all with the same "general SOCKS server failure". The website still claims there are no vanilla bridges available at all, and gives me back the same single obfs4 choice as before, which still won't connect at all.
I don't know if it's the link between the bridge authority and the bridgdb server, or the new bridgedb release that just came out or what, but something's badly broken and the bridgedb appears to be pretty much useless right now. My test environment is very standard, just Windows 10 Pro using only IPV4 from a normal US ISP (Frontier FIOS near Portland, OR) with no proxy, running a fresh copy of the current TBB build. The TBB works fine if I don't use a bridge or if I manually enter the bridge info for a private bridge relay I temporarily set up, and I have no other internet issues.
Could someone else check if they see similar problems? Other people should supposedly get different bridgedb results pulled from the other bridge pool rings, but even if it's somehow only affecting one pool ring, that's like 20-25% of the bridgedb's users left with no viable results. Looking at the Metrics site I don't see any drop in current bridge user counts (actually it's in an uptrend, along with all tor user counts lately) but I'd assume most bridge users already have working bridge lines cached, so this would only affect new users or those whose old choices have gone down, but both of those categories will only grow with time.
Thanks for reporting this issue, Rick.
On Wed, Jun 19, 2019 at 23:30:40 UTC, Rick Huebner wrote:
It looks like bridge 1 (eldritchworld) gave me bad/incomplete relay info which prevented making circuits [...]
For what it's worth, I managed to bootstrap an obfs4 connection over this bridge.
I know the new bridgedb release doesn't give out ORports if there's a secure PT offered, but last I checked the majority of bridges were just vanilla, so surely there must be enough to put some into whichever bridge pool ring my request was pulled from.
Right now, 1,018 bridges have the 'Running' flag. 585 (57.5%) of these 1,018 bridges support obfs4.
Could someone else check if they see similar problems?
We're aware of this problem. See the following ticket for (slightly) more information: https://bugs.torproject.org/30441
As far as I can tell, there are at least three problems:
1. We don't have enough bridges and after fixing #28655, we are left with even fewer vanilla bridges.
2. Some bridges have their obfs4 port firewalled. After reaching out, some operators fixed this issue. Other obfs4 bridges don't have contact info, so we will have to blacklist them.
3. There's some churn among bridges and once a bridge goes offline, it takes ~2-3 hours until this event propagates to BridgeDB. In the meanwhile, BridgeDB is still handing out the bridge line.
I wouldn't be surprised if there's another issue related to how BridgeDB allocates bridges into rings. I'll look into it.
On a general note, BridgeDB development has slowed down a bit because a long-time maintainer is not around anymore and its current maintainers (which includes me) are just getting up to speed.
Cheers, Philipp
Hi, Phillip. Thanks for the response, and for taking on these issues. I also sent a more detailed direct report to frontdesk@torproject.org. It apparently crossed with your reply to this thread in the mail, I wasn't intending to nag. I assume that'll end up forwarded to you as well, hopefully the extra detail I was able to include there will be helpful in debugging the problems.
I think the hash ring allocation question may be the most severe and fundamental issue overall. The BridgeDB request page still just now insisted to me yet again that there are no vanilla bridges available at all. Even if plain vanilla ORport bridges are no longer the majority, there must be at least a few dozen (if not a few hundred) remaining, more than enough to ensure at least a few end up in every hash ring. I'd never expect to be told that there are no vanilla bridges available, period, but that's the only result I've gotten over many tries for a few days now. Only getting back 1 or 2 obfs4 choices, which often show up again well past the selection rotation interval, also seems like it points to not filling the hash rings properly. Or maybe something broke in the result filtering stage where it removes supposedly blocked or same-subnet choices? Even if a lot of the obfs4 bridges are unreachable, they should still be showing up in the response lists, and for some reason they aren't.
Anyway, thanks again, and if there's anything I can do to help test things feel free to email me directly.
On Fri, Jun 21, 2019 at 05:56:21AM -0700, Rick Huebner wrote:
Hi, Phillip. Thanks for the response, and for taking on these issues. I also sent a more detailed direct report to frontdesk@torproject.org. It apparently crossed with your reply to this thread in the mail, I wasn't intending to nag. I assume that'll end up forwarded to you as well, hopefully the extra detail I was able to include there will be helpful in debugging the problems.
Thanks for your help, Rick. I did see your frontdesk email.
Here's a quick update on the issue:
1. I blacklisted 53 bridges whose obfs4 port was unreachable: https://gitweb.torproject.org/project/bridges/bridgedb-admin.git/commit/?id=5a8fe00304aee3df49ebbcbeb16ae3854bda0556
After deploying https://bugs.torproject.org/28655, these bridges were effectively useless. Hopefully, this will reduce the odds of getting a non-functional obfs4 bridge.
2. I added a log message to BridgeDB that tells us how many bridge requests resulted in 0, 1, 2, and 3 bridge lines. Here are the results for a few hours worth of logs:
# of bridges | # of requests -------------+-------------- 0 | 188 (14%) 1 | 381 (29%) 2 | 395 (38%) 3 | 235 (18%)
(Interestingly, all requests that resulted in 0 bridges were HTTPS requests for obfs2, coming from Tor exit relays. BridgeDB no longer supports obfs2, which is why it responds with 0 bridges.)
Assuming that these numbers are correct, BridgeDB should be returning at least one bridge for every request it has seen over the last few hours. That clearly wasn't the case a few days ago but I wonder if it's the case now. Are you still unable to get bridges from BridgeDB?
Cheers, Philipp
It's working much better now. Things started improving just a few hours after I posted my previous message, around last Friday afternoon PDT. The web page and email channels started giving out vanilla bridges again (usually 2), and I consistently got 2 obfs4 bridges via Moat and the web page. I just checked again and got 3 obfs4 bridges via Moat, 2 vanilla bridges via both the web page and email, and 2 obfs4 bridges via the web page.
I'm not sure how obfs4 bridges being unreachable would have prevented bridgedb from just giving them out anyway (pretty sure it can't tell on its own), or how any obfs4 issues would prevent it from giving out vanillas, but whatever you did starting last Friday seems to have unclogged the pipes. I'm still seeing weird connection issues with the bridges that I do get; sometimes they work, others they fail with similar errors to the logs I reported before, even though I'm just using the stock TBB 8.5.3 release, and connecting it normally without a bridge or selecting one of the built-in default bridges always works fine so there don't seem to be any local system issues. I've no idea what's going on there, but at least bridges are being distributed again. Thanks for your efforts.
On Thu, Jun 27, 2019 at 07:00:14AM -0700, Rick Huebner wrote:
I'm not sure how obfs4 bridges being unreachable would have prevented bridgedb from just giving them out anyway (pretty sure it can't tell on its own) [...]
Correct, these are two orthogonal problems. Both cause significant user frustration and you may have been affected by the "some bridges are unreachable" problem too, which is why I brought it up.
Cheers, Philipp
tor-relays@lists.torproject.org
-
Philipp Winter -
Rick Huebner