FYI, I cloned into a Digital Ocean cloud server running Ubuntu 17.04 (kernel 4.10.0-33 generic). It compiled and ran fine.

On Mon, Oct 30, 2017 at 2:06 PM, Toralf Förster toralf.foerster@gmx.de wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256

On 10/30/2017 08:35 PM, Damian Johnson wrote:

...

Would relay operators mind giving Nyx a whirl? To give it a try simply ensure you have a control port available in your torrc...

ControlPort 9051 CookieAuthentication 1

... and run the following...

% git clone https://git.torproject.org/stem.git % git clone https://git.torproject.org/nyx.git % cd nyx/ % ln -s ../stem/stem stem % ./run_nyx

Thanks! -Damian

mr-fox ~ # git clone https://git.torproject.org/stem.git Cloning into 'stem'... remote: Counting objects: 21863, done. remote: Compressing objects: 100% (1927/1927), done. remote: Total 21863 (delta 1969), reused 783 (delta 629) Receiving objects: 100% (21863/21863), 5.61 MiB | 21.29 MiB/s, done. Resolving deltas: 100% (17332/17332), done. mr-fox ~ # git clone https://git.torproject.org/nyx.git Cloning into 'nyx'... remote: Counting objects: 12723, done. remote: Compressing objects: 100% (691/691), done. remote: Total 12723 (delta 554), reused 265 (delta 152) Receiving objects: 100% (12723/12723), 8.03 MiB | 32.90 MiB/s, done. Resolving deltas: 100% (9735/9735), done. mr-fox ~ # cd nyx/ mr-fox nyx # ln -s ../stem/stem stem mr-fox nyx # ./run_nyx Traceback (most recent call last): File "./run_nyx", line 7, in <module> import nyx File "/root/nyx/nyx/__init__.py", line 49, in <module> import sqlite3 ImportError: No module named sqlite3 mr-fox nyx #

:-/

---

Toralf PGP C4EACDDE 0076E94E -----BEGIN PGP SIGNATURE-----

iI0EAREIADUWIQQaN2+ZSp0CbxPiTc/E6s3eAHbpTgUCWfeGUhccdG9yYWxmLmZv ZXJzdGVyQGdteC5kZQAKCRDE6s3eAHbpTnnxAP9OZR1lJoHj/Ksv/CTs9aj1X2xs 8NfJgK2PXHVeyYjWVgEAiFiSRpdm59zyyMFLpAD6UnPJo3qWs1iN610DATN3IMA= =qCop -----END PGP SIGNATURE----- _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256

On 10/30/2017 09:27 PM, Damian Johnson wrote:

Thanks Toralf! What python version are you using? Sqlite3 should be built in nowadays...

Oh, this is Gentoo Linux, so sqlite3 isn't built per default, only if I explicitely wants it by using USE= flag here I do assume - will try it.

FWIW, Python is here in version 2.7.2 and 3.4.5

- -- Toralf PGP C4EACDDE 0076E94E

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256

On 10/30/2017 09:35 PM, Damian Johnson wrote:

Ahhh, gotcha. Please let me know when you find the magic ingredient for sqlite3 on Gentoo an I'll add better messaging for it.

Oh, the message should be simply something like "You must emerge dev-lang/python with USE=sqlite"

And - it works now - Thx for developing nyx. Damian !

- -- Toralf PGP C4EACDDE 0076E94E

i

Hi Damian,

Thanks for the quick fix for the os.putenv() FreeBSD issue. That resolved it.

I think I've found a bug with the Connections pane. nyx appears to munge all the connections into "outbound", like:

Connections (4852 outbound, 1 control):

Whereas arm on the same system displays them correctly like:

Connections (2196 inbound, 2651 outbound, 1 control):

Otherwise I'm finding nyx to be a very good experience so far! Thanks again!

Nyx attempts to connect to some default locations but if you're running on another port it indeed needs the '-i' argument. Unfortunately I'm not sure of a good self-discovery mechanism. Would alternative messaging help?

It didn't take me long to figure out, but additional messaging might be helpful. Right now nyx just displays "Unable to connect to tor. Are you sure it's running?" Perhaps you could add something like "If you have set a custom Control Port for Tor, you can specify it with the '-i' argument."?

Hi Damian,

I can confirm the connections now display correctly. Thanks!

Is 'tor' listed when you run "ps -ao ucomm="?

It is *not* listed. In case it helps, Tor 0.3.1.8 was installed via ports/security/tor (https://www.freshports.org/security/tor), and I haven't done anything to modify the startup process. I think this is a fairly common way to install Tor on FreeBSD. The process is running /usr/local/bin/tor as user _tor.

I found one other issue. On startup nyx is displaying "nyx: environment corrupt; missing value for" (there's nothing after the "for"). I see it on the console twice after exiting nyx. This may be related to the comments Andrew Deason made about the environment?

Is it shown in the ps output as '/usr/local/bin/tor'? If so then I'll adjust Stem to look for that too.

Yep. 'ps ax' output looks like:

PID TT STAT TIME COMMAND 83797 - R 1152:14.26 /usr/local/bin/tor -f /usr/local/etc/tor/torrc --PidFile /var/run/tor/tor.pid --RunAsDaemon 1 --DataDirectory /var/db/tor

1. Comment out the putenv calls (add a '#' at the start of lines 91 and 96)

Yep, that's it. I commented out lines 90..98, both of the try/except blocks with os.putenv(). The error goes away and nyx seems to run fine.

On 31. Oct 2017, at 00:19, Damian Johnson atagar@torproject.org wrote:

...

I think I've found a bug with the Connections pane. nyx appears to munge all the connections into "outbound", like:

Connections (4852 outbound, 1 control):

Whereas arm on the same system displays them correctly like:

Connections (2196 inbound, 2651 outbound, 1 control):

Oops, great catch! Turned out that we were generally failing to pick up non-outbound connections at all due to a bug in Stem. Fixed...

https://gitweb.torproject.org/stem.git/commit/?id=8772dbe

Hi,

I'm afraid that's still buggy for some ipv4 configurations and ipv6 complicates this logic massively. I haven't looked at the code in detail nor do I understand why it might be important to know about local ports, but for ipv4 all of 127/8 should be considered local.

For IPv6, you have :: for "bind everything" (like 0.0.0.0), you have ::1 for localhost, but you also have the (awesome!) concept of link-local addresses to be aware of (fe80::/10). Since I don't exactly know how the logic in Stem works here, you may need to check for that, too.

Let me know if anything above needs more explanation :)

Cheers Sebastian

I updated the ticket to say something like this :-)

Thanks Tim, thanks Sebastian. Pushed a fix to improve this...

https://gitweb.torproject.org/stem.git/commit/?id=66a9b77

Also, are you aware that inbound connections can come in on ORPorts that are configured like this, and therefore aren't bound to a local address at all?

ORPort 91.121.230.208:9001 ORPort [2001:41d0:d:22a6::1:0]:9001

Gotcha. Think at this point I'm willing to say 'good enough'. If this becomes an actual problem in practice we'll cross that bridge.

I'm not sure if it's clear, but this is FreeBSD complaining that the environment string is invalid (an entry is missing the '=' separating the name and value). It's probably worth looking into why that's happening if you are able; whether nyx/stem/python is somehow causing that, or if it's something wrong/weird with your machine.

The commit to address this will silence the error, but it still seems like something is wrong; all env-modifying calls should fail like this after this point, it seems like.

Agreed. I can't (since I don't run FreeBSD) but if somebody that does could suggest an alternate os.putenv command for that platform I'd love to include it in Nyx.

On Sun, 5 Nov 2017 14:42:43 -0500 grarpamp grarpamp@gmail.com wrote:

...

...

...

OSError: [Errno 14] Bad address

 14 EFAULT Bad address.  The system detected an invalid address in
         attempting to use an argument of a call.

That's the general definition of EFAULT, yes. A more helpful definition is from getenv(3):

ERRORS [...] [EFAULT] The functions setenv(), unsetenv() or putenv() failed to make a valid copy of the environment due to the environment being corrupt (i.e., a name without a value). A warning will be output to stderr with information about the issue.

...

I'm not sure if it's clear, but this is FreeBSD complaining that the environment string is invalid (an entry is missing the '=' separating the name and value).

No, os.putenv is a two argument python function and is correct as shown above.

I didn't say otherwise. FreeBSD will return an EFAULT error if the environment string for the process is malformed at the time of the putenv. You can see the logic for this here:

https://github.com/freebsd/freebsd/blob/master/lib/libc/stdlib/getenv.c

On Mon, 30 Oct 2017 17:40:05 -0500 Andrew Deason adeason@dson.org wrote:

It's probably worth looking into why that's happening if you are able; whether nyx/stem/python is somehow causing that, or if it's something wrong/weird with your machine.

Looks like the same bug (or a very similar one) has been found before: https://trac.torproject.org/projects/tor/ticket/9804

Damian Johnson atagar@torproject.org wrote:

...

Is FreeBSD supported? I've run into a few hurdles, and overcame several, but can't get over the last one.

Hi. Yup, FreeBSD is supported *but* I don't have a system to test on, so I rely on user reports like this to provide comparability.

...

I had to do:

sudo pkg install py27-sqlite3

Given that nyx is supposedly now written in python 3, perhaps a better choice would be one of py3[456]-sqlite3, according to which python 3 version is installed, instead of mixing python versions.

Perfect! I'll change Nyx so when sqlite3 is unavailable on FreeBSD it suggests for folks to run that.

See note above.

Scott Bennett, Comm. ASMELG, CFIAG ********************************************************************** * Internet: bennett at sdf.org *xor* bennett at freeshell.org * *--------------------------------------------------------------------* * "A well regulated and disciplined militia, is at all times a good * * objection to the introduction of that bane of all free governments * * -- a standing army." * * -- Gov. John Hancock, New York Journal, 28 January 1790 * **********************************************************************

Adjusted the message to specify installing for whatever interpreter version they're running... https://gitweb.torproject.org/nyx.git/commit/?id=93f20f9

Damian: there's now a missing ')' at the end of nyx/__init__.py line 78 causing a syntax error.

Scott: thanks for the tip about the different sqlite3 versions, although my system seems to actually still use Python 2.7.14 by default.

tor tor@anondroid.com wrote:

...

Adjusted the message to specify installing for whatever interpreter version they're running... https://gitweb.torproject.org/nyx.git/commit/?id=93f20f9

Damian: there's now a missing ')' at the end of nyx/__init__.py line 78 causing a syntax error.

Scott: thanks for the tip about the different sqlite3 versions, although my system seems to actually still use Python 2.7.14 by default.

Try "pkg install python36". If nyx is written for python 3, then running it with python 2 might not be such a good idea. It may work for now, but nothing guarantees that it will continue to do so.

Scott Bennett, Comm. ASMELG, CFIAG ********************************************************************** * Internet: bennett at sdf.org *xor* bennett at freeshell.org * *--------------------------------------------------------------------* * "A well regulated and disciplined militia, is at all times a good * * objection to the introduction of that bane of all free governments * * -- a standing army." * * -- Gov. John Hancock, New York Journal, 28 January 1790 * **********************************************************************

Thanks Andrew! Great points, changes pushed.

On Mon, Oct 30, 2017 at 3:32 PM, Andrew Deason adeason@dson.org wrote:

On Mon, 30 Oct 2017 14:00:44 -0700 Damian Johnson atagar@torproject.org wrote:

...

Hi all. Pushed a couple changes to address feedback thus far...

Sorry if this is not the right place for nitpicking/bikeshedding, but:

...

• Fixed the os.putenv() issue that came up for FreeBSD...

  https://gitweb.torproject.org/nyx.git/commit/?id=bcb0122

Bare 'except:' clauses are really not recommended for error handling, since it catches things like SystemExit and GeneratorExit (not to mention it also triggers if you misspell 'putenv' or something). You can use 'except Exception:' instead, but here I would suggest at least 'except OSError:'.

...

• When sqlite3 is unavailable encouraging folks to contact us so we can provide per-platform advice. For FreeBSD providing the 'pkg install' command mentioned earlier...

  https://gitweb.torproject.org/nyx.git/commit/?id=4bfe05d

s/Unfortunatley/Unfortunately/

And thanks for your work; don't mistake this for me being ungrateful :)

-- Andrew Deason adeason@dson.org

---

tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Thanks Toralf!

The # of digits after the comma might be reduced: "Outbound (3414, avg: 3275.896126452135412364):"

That's odd. Was this the title bar of the connections page?

And if I press "m" here under Gentoo, then sometoimes I do get :

mr-fox nyx # ./run_nyx -i 29051 Traceback (most recent call last): File "./run_nyx", line 14, in <module> nyx.main() File "/root/nyx/nyx/__init__.py", line 147, in main nyx.starter.main() File "/root/nyx/stem/util/conf.py", line 289, in wrapped return func(*args, config = config, **kwargs) File "/root/nyx/nyx/starter.py", line 94, in main nyx.curses.start(nyx.draw_loop, acs_support = config.get('acs_support', True), transparent_background = True, cursor = False) File "/root/nyx/nyx/curses.py", line 216, in start curses.wrapper(_wrapper) File "/usr/lib64/python3.4/curses/__init__.py", line 94, in wrapper return func(stdscr, *args, **kwds) File "/root/nyx/nyx/curses.py", line 214, in _wrapper function() File "/root/nyx/nyx/__init__.py", line 192, in draw_loop nyx.menu.show_menu() File "/root/nyx/nyx/menu.py", line 203, in show_menu menu = _make_menu() File "/root/nyx/nyx/menu.py", line 243, in _make_menu submenu = panel.submenu() File "/root/nyx/nyx/panel/log.py", line 273, in submenu [RadioMenuItem(opt, filter_group, opt) for opt in self._filter.latest_selections()], File "/root/nyx/nyx/log.py", line 432, in latest_selections return list(reversed(self._past_filters.keys())) TypeError: argument to reversed() must be a sequence

Pushed a change that might or might not fix it. Mind giving it a whirl?

https://gitweb.torproject.org/nyx.git/commit/?id=b4989bf

...

That's odd. Was this the title bar of the connections page?

Pls see the atatched screen shot

Thanks Toralf! Pushed a fix for that, as well as a change so nyx can quit more quickly. However, unfortunately it won't help with the quitting issue you mentioned off list. That stacktrace indicated we got stuck terminating the controller itself.

Per chance were you logging at the DEBUG runlevel at that time? On a busy relay the amount of messages that slams the control port with can make it slow to shut down. Unfortunately besides that I'm at a loss for why it gets stuck for you. Looking at your debug logs would be the next step. Happy to discuss it off list if that would be ok.

On 31.10.17 19:05, Damian Johnson wrote:

I assume your user's able to read /proc/net/tcp'?

I see the "unable to query connections..." notices on Gentoo and Debian Linux machines. On both platforms, the permissions are as follows:

$ ls -l /proc/net/tcp -r--r--r-- 1 root root 0 Oct 31 19:33 /proc/net/tcp

If so then mind running 'nyx --debug' and sending me the log (minus anything you consider private)?

I'd like to help, alas 'run_nyx --debug /path/to/log' writes staggering amounts of data. From what I see, both DEBUG and TRACE level entries are logged, is this deliberate? I can see auth challenges and whatnot, so I am not exactly sure what I need to remove before sending logs to you.

-Ralph

On 01.11.2017 19:45, Damian Johnson wrote:

...

'run_nyx --debug /path/to/log' writes staggering amounts of data. From what I see, both DEBUG and TRACE level entries are logged, is this deliberate?

Yup, it's intentional for the debug log to log low level messages. That's why it's useful. :P

Lord give me patience, but make it fast... ;-) Let me rephrase: Is it deliberate that nyx logs both DEBUG and TRACE level messages when I use the --debug option? I would only expect debug level entries.

I'd just need the first minute or so of logs to see what's up with the connection resolution attempts.

That generates many MB of output data through which I'd rather not wade aimlessly, trying to identify what to anonymise/delete. Can I for example get rid of all TRACE entries for starters, or would that render the output useless to you? What other data would I need to remove so you don't get unnecessary insight into my nodes' details? I mentioned auth challenges already.

-Ralph

P.S.: Perhaps we should discuss this further off-list?

I am running Tor 0.3.0.10 on a standard Debian install. I downloaded nyx about 10 am Nov.2, so I assume that has all the bugs fixed mentioned in this thread. I am running it without any nyx config.

1) I don't have any line showing Connection Count: Instead I have the same as arm showing Bandwidth (limit, burst, measured)

2)I don't have any Accounting (awake) line in nyx that shows my current totals vs max limit. This line just appeared in arm when I added accounting lines to the torrc, so I assume it should do the same in nyx?

TIA,

--torix

Sent with [ProtonMail](https://protonmail.com) Secure Email.

-------- Original Message -------- Subject: Re: [tor-relays] Testers needed for Nyx beta release Local Time: November 1, 2017 3:26 PM UTC Time: November 1, 2017 7:26 PM From: atagar@torproject.org To: tor-relays@lists.torproject.org

...

Lord give me patience, but make it fast... ;-) Let me rephrase: Is it deliberate that nyx logs both DEBUG and TRACE level messages when I use the --debug option? I would only expect debug level entries.

Yes, it is.

That generates many MB of output data through which I'd rather not wade aimlessly, trying to identify what to anonymise/delete. Can I for example get rid of all TRACE entries for starters, or would that render the output useless to you? What other data would I need to remove so you don't get unnecessary insight into my nodes' details? I mentioned auth challenges already.

Sure. If you'd care to 'grep -v' those out then feel free. For this issue the tor trace logs are not helpful.

P.S.: Perhaps we should discuss this further off-list?

Sure, sounds good to me.

---

tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Thanks about the S; I probably knew it when I first started, but don't use arm much; the box pretty much runs itself.

More specifically (now that I know about the S menu), the Accounting lines only appear with the Bandwidth selected. My torrc lines that are driving this are: AccountingStart day 00:01 AccoutingMax 15 GBytes RelayBandwidthRate 500 KBytes RelayBandwidthBurst 500 KBytes

TIA,

-- torix

Sent with [ProtonMail](https://protonmail.com) Secure Email.

-------- Original Message -------- Subject: Re: [tor-relays] Testers needed for Nyx beta release Local Time: November 2, 2017 6:22 PM UTC Time: November 2, 2017 10:22 PM From: atagar@torproject.org To: tor-relays@lists.torproject.org

Hi torix.

...

• I don't have any line showing Connection Count:

Instead I have the same as arm showing Bandwidth (limit, burst, measured)

Sorry, not sure I follow. I assume you mean the connection count graph? If so then like arm we only one graphed stat at a time. If you press 's' then select Connections you should get the connection graph.

2)I don't have any Accounting (awake) line in nyx that shows my current totals vs max limit. This line just appeared in arm when I added accounting lines to the torrc, so I assume it should do the same in nyx?

Interesting! Nyx certainly should have accounting information if you have that configured in your torrc. I'll look into that when I'm home from work.

---

tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Arm says 1568 outbound 151 exit. Since My exit policy blocks abused ports and has only one /8 range for port 80 I think that is true.

Nyx says 668 outbound 1173 exit. Also nyx does not list the exits by port number

Gerry

-----Original Message----- From: tor-relays [mailto:tor-relays-bounces@lists.torproject.org] On Behalf Of Damian Johnson Sent: 02 November 2017 22:50 To: tor-relays@lists.torproject.org Subject: Re: [tor-relays] Testers needed for Nyx beta release

More specifically (now that I know about the S menu), the Accounting lines only appear with the Bandwidth selected.

Ahhh! Gotcha. Yes, it's intended for accounting information to only be present when showing the bandwidth graph. _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

But to be clear, nyx still does not show the accounting lines that arm does.

Ah! Found it, accounting stats fixed. Thanks for pointing this out!

https://gitweb.torproject.org/nyx.git/commit/?id=6e9efcb

New issue - again I only have one machine running, so I don't know if it is just my problem - page 2: Inbound connections in nyx look like this: │ IP:51396 (??) --> my ip:8443 + 3.7m (INBOUND) Arm shows me the country name instead of (??). TIA,

-- torix

Sent with [ProtonMail](https://protonmail.com) Secure Email.

-------- Original Message -------- Subject: Re: [tor-relays] Testers needed for Nyx beta release Local Time: November 3, 2017 1:07 PM UTC Time: November 3, 2017 5:07 PM From: atagar@torproject.org To: tor-relays@lists.torproject.org

...

But to be clear, nyx still does not show the accounting lines that arm does.

Ah! Found it, accounting stats fixed. Thanks for pointing this out!

https://gitweb.torproject.org/nyx.git/commit/?id=6e9efcb

---

tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

New issue - again I only have one machine running, so I don't know if it is just my problem - page 2: Inbound connections in nyx look like this: │ IP:51396 (??) --> my ip:8443 + 3.7m (INBOUND) Arm shows me the country name instead of (??).

Thanks torix. Unfortunately I don't have tor's geoip db for either of the tor instances I'm testing on. Would you mind sending me your debug log off-list (scrubbed of anything you consider secret)? I'd like to see what it says about the ip-to-country queries.

Thanks for this great job!

Dunno if it is an expected behavior or not, but when I enter the menu, the gui is updated once after the first keypress, and then is not updated at all then until the menu is left. Then all missed frames are caught up when leaving the menu. This gives a weird feeling with graphs being updated extremely fast.

I am running debian stretch.

Thanks Guinness! Gotta swap over to my day job but I'll try to circle back to look into the menus in a day or two. I definitely noticed some annoying issues there.

"Menu" -> "Exit" works here under Gentoo now, but "q" -> "q" still not, after Ctrl-C I do get :

Hi Toralf, that's really puzzling. The menu and 'q' keys should do the same thing. Just took a peek at the code and I can't spot a sizable difference. Just tried again a few times to repro but no luck. As mentioned earlier...

Per chance were you logging at the DEBUG runlevel at that time? On a busy relay the amount of messages that slams the control port with can make it slow to shut down. Unfortunately besides that I'm at a loss for why it gets stuck for you. Looking at your debug logs would be the next step. Happy to discuss it off list if that would be ok.

That stacktrace indicates that Stem is getting stuck while waiting for the control socket to close. Nothing about it is Nyx specific so wonder if we'd be able to repro the issue with Stem's integ tests...

https://stem.torproject.org/faq.html#how-do-i-run-the-tests

Mind giving those a whirl to see if they provide a consistent repro?

Since there's only a single report of this think I'll move forward with a release but I'd love to sort this out if we're able to get a good repro case to troubleshoot.

Cheers! -Damian

After pulling in the latest nyx/stem changes, nyx seems to be taking much longer for connections to appear in the Connections pane. After switching to that view the first time, it's taking about 30 seconds to load the list. Once it's loaded I can switch panes and come back and the list is still populated. It's not really critical but it does seem to have changed with recent commits. This is on FreeBSD.

Hi, do you have 'DisableDebuggerAttachment 0' in your torrc? If so then a recent change I made could indeed account for this...

https://gitweb.torproject.org/nyx.git/commit/?id=b801d16

Unfortunately that makes the connection panel appear sooner for others. :/

Mind pulling nyx and trying again? https://gitweb.torproject.org/nyx.git/commit/?id=08dbde2

Yeah, that fixed it. :)

Before reading your last email, I tried removing "DisableDebuggerAttachment 0". That actually made things worse. The connections weren't showing at all, even after waiting 5 minutes. It may be the same underlying issue that you just fixed, but thought I'd mention it. (I'd test it again, but that particular config option requires a hard restart of the Tor service, so I'm going to leave it alone for now).

Thanks again.