Hi Karsten,
I was surprised that ornetradar did not send a single email for yesterday's new relays.
After looking into it, it turned out it is an onionoo problem.
"relays_published":"2016-05-13 12:00:00"
https://onionoo.torproject.org/details?limit=4
regards, nusenu
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 14/05/16 12:00, nusenu wrote:
Hi Karsten,
Hi Nusenu,
I was surprised that ornetradar did not send a single email for yesterday's new relays.
After looking into it, it turned out it is an onionoo problem.
"relays_published":"2016-05-13 12:00:00"
It looks like the Onionoo host has some serious load problems (again). It's at "relays_published":"2016-05-16 19:00:00" now, so it didn't stop entirely, but it's still behind. I just kicked it, maybe it'll catch up in the next few hours.
As a short-term fix, feel free to switch to the Onionoo mirror:
https://onionoo.thecthulhu.com/summary?limit=0
As a medium-term fix, extend your client to fetch both headers (/summary?limit=0) and get the full details from the host with more recent "relays_published" line.
As a long-term fix we should fix most/all inconsistencies between Onionoo instances (different encodings and other minor problems) and add a hostname that round-robins requests to all available mirrors.
regards, nusenu
Thanks for letting me know!
All the best, Karsten
Hi Thomas,
I'm using your onionoo instance to fetch details documents.
Currently it is stuck at 2016-05-26 00:00, you might want to look into it?
(even though its date does not change since a few days its file contents do)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 31/05/16 11:24, nusenu wrote:
Hi Thomas,
Hi nusenu,
I'm using your onionoo instance to fetch details documents.
(Thomas is kindly providing the host, but I'm operating the Onionoo instance on that.)
Currently it is stuck at 2016-05-26 00:00, you might want to look into it?
Huh, you're right. Looks like the updater got stuck while downloading descriptors from CollecTor:
2016-05-26 01:19:15,175 INFO o.t.o.cron.Main:176 Downloading descriptors. 2016-05-26 01:19:15,176 INFO o.t.o.u.DescriptorSource:64 Loading: RELAY_CONSENSUSES 2016-05-26 01:25:05,420 INFO o.t.o.u.DescriptorSource:64 Loading: RELAY_SERVER_DESCRIPTORS
That instance was offline for more than 3 days which means that I'll have to let it re-import the May 2016 tarballs that are currently produced. I need to wait for that and then let it chew on a lot of data. I hope it's back tomorrow.
(even though its date does not change since a few days its file contents do)
Not sure what you mean by "its file contents do change". Can you explain?
All the best, Karsten
That instance was offline for more than 3 days which means that I'll have to let it re-import the May 2016 tarballs that are currently produced. I need to wait for that and then let it chew on a lot of data. I hope it's back tomorrow.
Thanks for fixing it.
(even though its date does not change since a few days its file contents do)
Not sure what you mean by "its file contents do change". Can you explain?
I was trying to say that the details records provided over the last few days changed even though relays_published timestamp didn't -- but it turned out that I was wrong so you can ignore it.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 31/05/16 14:21, Karsten Loesing wrote:
On 31/05/16 11:24, nusenu wrote:
Hi Thomas,
Hi nusenu,
I'm using your onionoo instance to fetch details documents.
(Thomas is kindly providing the host, but I'm operating the Onionoo instance on that.)
Currently it is stuck at 2016-05-26 00:00, you might want to look into it?
Huh, you're right. Looks like the updater got stuck while downloading descriptors from CollecTor:
2016-05-26 01:19:15,175 INFO o.t.o.cron.Main:176 Downloading descriptors. 2016-05-26 01:19:15,176 INFO o.t.o.u.DescriptorSource:64 Loading: RELAY_CONSENSUSES 2016-05-26 01:25:05,420 INFO o.t.o.u.DescriptorSource:64 Loading: RELAY_SERVER_DESCRIPTORS
That instance was offline for more than 3 days which means that I'll have to let it re-import the May 2016 tarballs that are currently produced. I need to wait for that and then let it chew on a lot of data. I hope it's back tomorrow.
Update on https://onionoo.thecthulhu.com/: I ran into an issue that I didn't resolve yet. It's probably something trivial, I just didn't find the time to look more closely:
https://trac.torproject.org/projects/tor/ticket/19249
Update on https://onionoo.torproject.org/: This one should be running much more smoothly as of a few hours ago. There was a problem with Apache's disk cache which caused all ~1M/h requests to go through the Java process. That's what you can see in the last few weeks in the following graph. We're now back to a few 10k/h:
https://people.torproject.org/~karsten/volatile/onionoo-total-requests-2016-...
All the best, Karsten
(even though its date does not change since a few days its file contents do)
Not sure what you mean by "its file contents do change". Can you explain?
All the best, Karsten
Hi Karsten,
could you look into onionoo.tpo, looks like it is stuck again.
thanks, nusenu
"relays_published":"2016-10-30 23:00:00",
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Hi nusenu,
should be resolved now. Turns out one of the CollecTor modules silently died. It's back now.
If you notice similar problems in the future, be sure to let us know! We do have a few checks in place, but this issue slipped through somehow. Thanks!
All the best, Karsten
On 01/11/16 00:59, nusenu wrote:
Hi Karsten,
could you look into onionoo.tpo, looks like it is stuck again.
thanks, nusenu
"relays_published":"2016-10-30 23:00:00",
Karsten Loesing:
If you notice similar problems in the future, be sure to let us know! We do have a few checks in place, but this issue slipped through somehow.
I assume you are already aware that onionoo is currently a bit behind (2017-01-27 13:00).
https://lists.torproject.org/pipermail/tor-relays/2017-January/011812.html
On 28/01/17 10:06, nusenu wrote:
Karsten Loesing:
If you notice similar problems in the future, be sure to let us know! We do have a few checks in place, but this issue slipped through somehow.
I assume you are already aware that onionoo is currently a bit behind (2017-01-27 13:00).
Yes, I'm upgrading to protocol version 3.2, which requires re-importing bridge descriptor archives since December 2014. Of course that didn't go without problems, first with memory running out, then disk space. I hope it'll be back later today.
Thanks for the report!
All the best, Karsten
https://lists.torproject.org/pipermail/tor-relays/2017-January/011812.html
On 28/01/17 10:23, Karsten Loesing wrote:
On 28/01/17 10:06, nusenu wrote:
Karsten Loesing:
If you notice similar problems in the future, be sure to let us know! We do have a few checks in place, but this issue slipped through somehow.
I assume you are already aware that onionoo is currently a bit behind (2017-01-27 13:00).
Yes, I'm upgrading to protocol version 3.2, which requires re-importing bridge descriptor archives since December 2014. Of course that didn't go without problems, first with memory running out, then disk space. I hope it'll be back later today.
It should be back since yesterday afternoon. Please shout if any issues remain.
All the best, Karsten
Thanks for the report!
All the best, Karsten
https://lists.torproject.org/pipermail/tor-relays/2017-January/011812.html
-
Karsten Loesing -
nusenu