Thanks George! Yup, work on that branch is in progress:

https://gitweb.torproject.org/user/atagar/stem.git/log/?h=hsv3

On Thu, Oct 17, 2019 at 8:01 AM George Kadianakis desnacked@riseup.net wrote:

Damian Johnson atagar@torproject.org writes:

...

...

Can I use `_descriptor_content()` to do that? Or should I call `_descriptor_content()` to generate the whole thing _without_ the sig, and then do the signature computation on its result and concatenate it after?

Hi George. Yup, to create a signed descriptor we create the bulk of the content then append the signature. Server and extrainfo descriptors already do this so I suspect you can do something similar...

https://gitweb.torproject.org/stem.git/tree/stem/descriptor/server_descripto... https://gitweb.torproject.org/stem.git/tree/stem/descriptor/__init__.py#n138...

Will this do the trick?

PS. Sorry about the duplicate. Hit reply rather than reply-all forgetting that you included the list.

Thanks for the reply Damian! That was super useful!

The current state of affairs can be found here: https://trac.torproject.org/projects/tor/ticket/31823#comment:1 (just in case you didn't check IRC that day)

peace

George Kadianakis desnacked@riseup.net writes:

Damian Johnson atagar@torproject.org writes:

...

Thanks George! Yup, work on that branch is in progress:

https://gitweb.torproject.org/user/atagar/stem.git/log/?h=hsv3

Hello Damian,

thanks for the reply here! I'm now back and ready to start working again on onionbalance/stem.

What is your plan with the hsv3 branch? Should I start reviewing your changes already, or give you more time to do more?

Thanks a lot for all the work! :)

Hello again,

I took a super quick look (particularly at the easy parts of your changes). Thanks for all the changes!

My only feedback so far is that the python2 port commits have broken python3 for me (particularly the ed25519 blinding implementation). In general, the ed25519 blinding implementation is very hairy Python3 crypto code and it won't be easy to support both versions I think.

Would it be egregious to provide hsv3 support only for python3 users so that we can use python3 features as we wish?

I personally plan to use HSv3 support for onionbalance and that will be in python3, so I wouldn't mind that at all. Not sure who else is gonna use hsv3 support in the near future.

Cheers!

PS: From now on perhaps we can use #31823 for code related discussions (sorry for the medium mixing)

Hi George,

On 28 Nov 2019, at 02:04, George Kadianakis desnacked@riseup.net wrote:

Hello Damian (and list),

here is another question about an issue I have encountered while developing onionbalance v3.

In particular, I'm fetching HS descriptors using HSFETCH and then adding an add_event_listener() event to a function that does the descriptor parsing and handling as follows:

   controller.add_event_listener(handle_new_desc_content_event, EventType.HS_DESC_CONTENT)

The problem is that the handle_new_desc_content_event() callback has grown to a non-trivial size and complexity, since it needs to parse the descriptor, put it in the right places, and tick off the right checkboxes.

Since its size has increased, so has the number of bugs and errors that are appearing during development. The problem is that because the callback is running on a separate thread (?) any errors and exceptions that get raised in that thread never surface to the my console and hence I don't see them. This means that I need to do very tedious printf debugging to find the exact place and type of error everytime something happens.

What's the proper way to do debugging and development in callbacks like that? Is there a way to make the exceptions float back to the main thread or something? Or any other tips?

In general, there are two ways to avoid exceptions disappearing in python threads, a code change, and a design change.

Code Change

The code change catches all exceptions in a thread, you should be able to do it right now. (Or Damian could implement it in stem, every time a thread is launched.)

1. Wrap all code called in a thread in a try/catch block 2. Catch every exception 3. Log it 4. Stop the thread/process/...

The code looks a bit like this:

try: run_thread() except e: print("Error in thread") log_error() # Pick one of these actions # Terminate the process immediately os._exit(1) # Re-raise the error - does this terminate the thread? raise # Stop the thread return 1

You'll want a log_error() function like this one: https://github.com/privcount/privcount/blob/master/privcount/log.py#L19

Another alternative is to join() threads when they terminate, and re-raise the exception in the main thread. But the code is a lot more complex: https://stackoverflow.com/a/6874161

Design Change

The design change moves all the work to the main thread, and just stores data/ sets a flag from other threads. That's not always possible, but it's usually a good idea to limit work in threads, because it avoids races, delays, and deadlocks.

In this case, you could store the data in the fetch thread, and then do the processing the next time the main thread tries to access that data, or anything related to it.

T

Damian Johnson atagar@torproject.org writes:

Thanks George, this is a great question! I've expanded our tutorial to hopefully cover this a bit better...

https://stem.torproject.org/tutorials/tortoise_and_the_hare.html#advanced-li...

Thanks both for this information! It was very useful! I basically followed the tutorial and it now works just fine!