Hi George,
On 28 Nov 2019, at 02:04, George Kadianakis desnacked@riseup.net wrote:
Hello Damian (and list),
here is another question about an issue I have encountered while developing onionbalance v3.
In particular, I'm fetching HS descriptors using HSFETCH and then adding an add_event_listener() event to a function that does the descriptor parsing and handling as follows:
controller.add_event_listener(handle_new_desc_content_event, EventType.HS_DESC_CONTENT)The problem is that the handle_new_desc_content_event() callback has grown to a non-trivial size and complexity, since it needs to parse the descriptor, put it in the right places, and tick off the right checkboxes.
Since its size has increased, so has the number of bugs and errors that are appearing during development. The problem is that because the callback is running on a separate thread (?) any errors and exceptions that get raised in that thread never surface to the my console and hence I don't see them. This means that I need to do very tedious printf debugging to find the exact place and type of error everytime something happens.
What's the proper way to do debugging and development in callbacks like that? Is there a way to make the exceptions float back to the main thread or something? Or any other tips?
In general, there are two ways to avoid exceptions disappearing in python threads, a code change, and a design change.
Code Change
The code change catches all exceptions in a thread, you should be able to do it right now. (Or Damian could implement it in stem, every time a thread is launched.)
1. Wrap all code called in a thread in a try/catch block 2. Catch every exception 3. Log it 4. Stop the thread/process/...
The code looks a bit like this:
try: run_thread() except e: print("Error in thread") log_error() # Pick one of these actions # Terminate the process immediately os._exit(1) # Re-raise the error - does this terminate the thread? raise # Stop the thread return 1
You'll want a log_error() function like this one: https://github.com/privcount/privcount/blob/master/privcount/log.py#L19
Another alternative is to join() threads when they terminate, and re-raise the exception in the main thread. But the code is a lot more complex: https://stackoverflow.com/a/6874161
Design Change
The design change moves all the work to the main thread, and just stores data/ sets a flag from other threads. That's not always possible, but it's usually a good idea to limit work in threads, because it avoids races, delays, and deadlocks.
In this case, you could store the data in the fetch thread, and then do the processing the next time the main thread tries to access that data, or anything related to it.
T