On 09/06/14 01:26, Damian Johnson wrote:
Oh, and another quick thought - you once mentioned that a descriptor search service would make ExoneraTor obsolete, and in looking it over I agree. The search functionality ExoneraTor provides is trivial. The only reason it requires such a huge database is because it's storing a copy of every descriptor ever made.
I suspect the actual right solution isn't to rewrite ExoneraTor at all, but rather develop a new service that can be queried for this descriptor data. That would make for a *much* more worthwhile project.
ExoneraTor? Nice to have. Descriptor archive service? Damn useful. :)
I agree, that was the idea behind Kostas' GSoC project last year. And I still think it's a good idea. It's just not trivial to get right.
Regarding your comment about storing a copy of every descriptor ever made, I believe that users trust ExoneraTor's results more if they see the actual descriptors that lead to results. Of course, I'm saying that without knowing what ExoneraTor users actually want. But let's not drop descriptor copies from the database easily.
And, heh, when you say that the search functionality ExoneraTor provides is trivial, a little part of me is dying. It's the part that spent a few weeks on getting the search functionality fast enough for production. That was not at all trivial. The oraddress24, oraddress48, and exitaddress24 fields as well as the indexes are the result of me running lots and lots of sample queries and wondering about Postgres' EXPLAIN ANALYZE results. Just saying that it's not going to be trivial to generalize the search functionality towards other fields than IP addresses and dates.
If others want to follow, here's the SQL code I'm talking about:
https://gitweb.torproject.org/exonerator.git/blob/HEAD:/db/exonerator.sql
So, I'm happy to talk about writing a searchable descriptor archive. It could _start_ with ExoneraTor's functionality (minus the target address and port thing discussed in that other email), and then we could consider adding more searches.
Pretty sure that Kostas is reading this (in fact, I just cc'ed him), so let me make one remark about optimizing Postgres defaults: I wrote quite a few database queries in the past, and some of them perform horribly (relay search) whereas others perform really well (ExoneraTor). I believe that the majority of performance gains can be achieved by designing good tables, indexes, and queries. Only as a last resort we should consider optimizing the Postgres defaults.
You realize that a searchable descriptor archives focuses much more on database optimization than the ExoneraTor rewrite from Java to Python (which would leave the database untouched)?
All the best, Karsten