Hi Karsten,
just wanted to let you know that the delta between relays_published and current time is unusually high.
https://onionoo.torproject.org/details?limit=0
{"version":"5.0", "build_revision":"0bce98a", "relays_published":"2018-01-21 22:00:00",
This is currently blocking ornetradar reports.
thanks for having a look, nusenu
On 2018-01-22 09:03, nusenu wrote:
Hi Karsten,
Hi nusenu,
just wanted to let you know that the delta between relays_published and current time is unusually high.
https://onionoo.torproject.org/details?limit=0
{"version":"5.0", "build_revision":"0bce98a", "relays_published":"2018-01-21 22:00:00",
This is currently blocking ornetradar reports.
Looks like the primary CollecTor instance had a problem between 22:00 and 08:00 UTC. It works again now, as does Onionoo.
We didn't lose any data, because the primary CollecTor instance obtained all descriptors it had missed earlier from the backup CollecTor instance.
thanks for having a look,
Thanks for the report!
nusenu
All the best, Karsten
Looks like the primary CollecTor instance had a problem between 22:00 and 08:00 UTC. It works again now, as does Onionoo.
Karsten, thanks for the fast reaction.
We didn't lose any data, because the primary CollecTor instance obtained all descriptors it had missed earlier from the backup CollecTor instance.
Since I'm archiving onionoo data I'm "loosing" data (causing blind spots) everytime a "relays_published" timestamp is skipped. In theory one could spin up an onionoo instance to generate data for skipped timestamps but in practice this is hard (requires lots of resources). (I know, you are probably talking about not loosing any raw CollecTor data, but wanted to mention that nonetheless.)
Do you monitor onionoo for such problems ("relays_published" timestamp remaining unchanged for >1-2 hours)? Would you find something like that useful?
Thanks for keeping it running besides all the other things you do.
I'm wondering if the admin team would be available to cover such cases to reduce the operations load for developers.
kind regards, nusenu
Hi nusenu,
On 2018-01-22 18:57, nusenu wrote:
Looks like the primary CollecTor instance had a problem between 22:00 and 08:00 UTC. It works again now, as does Onionoo.
Karsten, thanks for the fast reaction.
We didn't lose any data, because the primary CollecTor instance obtained all descriptors it had missed earlier from the backup CollecTor instance.
Since I'm archiving onionoo data I'm "loosing" data (causing blind spots) everytime a "relays_published" timestamp is skipped. In theory one could spin up an onionoo instance to generate data for skipped timestamps but in practice this is hard (requires lots of resources). (I know, you are probably talking about not loosing any raw CollecTor data, but wanted to mention that nonetheless.)
Right, I meant not losing any raw CollecTor data. Your use case of archiving Onionoo data is special. It's okay that you do this, but it's not what Onionoo was designed for. Most people will find Onionoo data that is 6 or 12 hours behind still useful. But if we had lost 6 or 12 hours of CollecTor data, that would have been pretty bad.
What we can do, though, is think about providing more history in Onionoo, so that you can give up on archiving Onionoo data. After all, Onionoo already provides quite some history, including graph data like in bandwidth documents and others, times when a relay last changed its IP address or port, the time it was first seen, and so on. If you have ideas what else would be valuable to have history for, please open a ticket.
Do you monitor onionoo for such problems ("relays_published" timestamp remaining unchanged for >1-2 hours)? Would you find something like that useful?
We do have such monitoring, yes. Here's the Nagios script we're using:
https://gitweb.torproject.org/admin/tor-nagios.git/tree/tor-nagios-checks/ch...
Thanks for keeping it running besides all the other things you do.
I'm wondering if the admin team would be available to cover such cases to reduce the operations load for developers.
The admin team already handles operational issues with the hosts, though the metrics team is still in charge for running the services. I think that's a fine separation, and it has worked quite well for the last couple of years.
kind regards, nusenu
All the best, Karsten
What we can do, though, is think about providing more history in Onionoo, so that you can give up on archiving Onionoo data.
It is nice of you to consider that but it is not necessary (at least for me) I can life with my current hacks and other probably don't need more history, and you have already enough stuff on your plate.
Do you monitor onionoo for such problems ("relays_published" timestamp remaining unchanged for >1-2 hours)? Would you find something like that useful?
We do have such monitoring, yes.
So my email was redundant to your nagios check?
Would it be possible to publish these alerts on a mailing list? :)
On 2018-01-26 11:34, nusenu wrote:
What we can do, though, is think about providing more history in Onionoo, so that you can give up on archiving Onionoo data.
It is nice of you to consider that but it is not necessary (at least for me) I can life with my current hacks and other probably don't need more history, and you have already enough stuff on your plate.
Okay.
Do you monitor onionoo for such problems ("relays_published" timestamp remaining unchanged for >1-2 hours)? Would you find something like that useful?
We do have such monitoring, yes.
So my email was redundant to your nagios check?
Would it be possible to publish these alerts on a mailing list? :)
Not a crazy idea! I opened a ticket for further discussing this:
https://trac.torproject.org/projects/tor/ticket/25035
All the best, Karsten
thanks for looking into it
Karsten Loesing:
On 2018-02-03 01:32, nusenu wrote:
thanks for looking into it
Looks like the CollecTor host is down, along with several other hosts. I sent mail to the admins.
Does that imply that we are actually loosing raw CollecTor data until it comes back?
On 2018-02-03 12:53, nusenu wrote:
Karsten Loesing:
On 2018-02-03 01:32, nusenu wrote:
thanks for looking into it
Looks like the CollecTor host is down, along with several other hosts. I sent mail to the admins.
Does that imply that we are actually loosing raw CollecTor data until it comes back?
No, we still have the backup CollecTor host that downloads Tor descriptors and that the primary CollecTor host will sync from once it comes back.
All the best, Karsten
Karsten Loesing:
On 2018-02-03 12:53, nusenu wrote:
Karsten Loesing:
On 2018-02-03 01:32, nusenu wrote:
thanks for looking into it
Looks like the CollecTor host is down, along with several other hosts. I sent mail to the admins.
Does that imply that we are actually loosing raw CollecTor data until it comes back?
No, we still have the backup CollecTor host that downloads Tor descriptors and that the primary CollecTor host will sync from once it comes back.
great!
thank you for bringing it back!
-
Karsten Loesing -
nusenu