I spent the past week in Sweden, attending the Stockholm Internet Forum, http://www.stockholminternetforum.se/, for part of it. I made a number of tails 0.10.2 usb sticks for people on request. I also asked a lot of people their impressions of Tor and Tails. I received a plethora of feedback. All 8 people are involved in the Internet Freedom policy, technology, or freedom of speech communities. They had very different levels of self-assessed technical skill. The 8 people represented 6 countries.
tldr; 8 people were tested, 8 people had trouble doing simple things with tor browser in Tails. Issues 4-6 are directly related to Tor.
And by asked, I mean, I stuck them in front of my laptop, put the usb stick in the computer, and asked them to browse to their favorite site. No one wanted to be video recorded, even if I offered to only record the screen and not audio.
Everyone managed to power on the laptop and wait for Tails to boot.
# First issue: Language selection
The first issue was on the language selection screen. 4 of 8 people were confused why it was called "Debian Live System" and not "Tails Live System". 8 of 8 knew what language selection meant, but weren't sure how this mapped to Tails.
# Second issue: wifi and tor browser
The tor browser starts up before the wireless is configured. The tor browser then reports a proxy error. With some prompting, all 8 figured out the wifi and then didn't know what to do. Tor does keep trying to load, and takes forever because it needs to download the entire directory. Users have no feedback as to what's going on behind the scenes because vidalia is hidden.
8 of 8 waited patiently for something to happen on the screen.
# Third issue: green onion
3 of 8 people saw the green onion appear in the menu bar up top. These three people hovered over it and saw the 'Connected to the Tor Network' message. No one knew to double-click on it to get a menu of other things to do. No one knew to right-click on it to get the drop-down menu. They were presented with the default check.torproject.org 'congratulations' page and then sat there.
# Fourth issue: check.tpo is not helpful
8 of 8 people saw the default check.torproject.org site telling them 'congratulations. Your browser is configured to use tor.' 7 of 8 people asked 'where is my browser?' The one who didn't ask this question was already a firefox user and recognized the interface. 0 of 8 understood what the IP address message meant. Comments ranged from 'is that different than my current IP address?' to 'what's an ip address?'
As an aside, when showing someone TBB on their own laptop, they saw the check.tpo site, and then went to Safari and started it up. When asked why they did this, the answer was 'safari is my browser. this says your browser is configured to use tor.'
No one used the language selections at the bottom of check.tpo, nor even understood why they were there.
# Fifth issue: exit relay congestion/failures
8 of 8 people tried to get to their own sites. 'I wonder what my site looks like when I'm anonymous' was the most common comment (5 of 8). For 6 of 8 people, their site didn't load at all, and tor browser reported their site was unreachable. All 6 then tried to go to google search in their own language; meaning google.es, google.se, etc. For 3 of those 6, this didn't work either. They gave up and assumed tor was broken or was censoring their destinations.
I intervened, opened the vidalia network map, closed the circuit in question, and asked them to repeat their browsing.
5 of 6 were able to get to their sites now. The one that was not able to had the same exit relay as last time, Amunet1, in a new circuit and just couldn't get anywhere through it. After yet another new circuit, they could get through to everything.
The user has no feedback as to why their site didn't work. And tor assumes everything is working fine.
When asked "please find a video you like", they all went to youtube. Most of the videos they wanted to see resulted in 'This video is currently unavailable.' 8 of 8 assumed it was because youtube was blocking tor, not because the video is flash-required. 2 of 8 started randomly clicking videos suggested by youtube to see if any of them worked. Eventually, 2 of 8 got videos to work with youtube and were amazed it worked at all.
# Sixth issue: no flash, no warning
2 of 8 people had flash apps on their website. 4 of 8 had ad banners that used flash. All were surprised at the red outline with a snake in it appearing instead of their flash apps. None understood what happened.
After an explanation, one person suggested changing the red outline with snake to an actual message written inside, along the lines of 'this app blocked for your protection. click here to unblock it.' I explained why that wouldn't work (because there is no flash, java, silverlight plugins installed) and their answer was 'then do not show it at all'. Inside noscript, I unchecked the 'show placeholder..' option and had them browse again. they were happy. It seems if the user cannot do anything about the blocked apps, not showing them may be preferred.
# Seventh issue: shutdown
I asked all 8 to shutdown tails and let me know when they thought their data was safely no longer on the system. 1 of 8 figured out how to shutdown tails by clicking the big red button in the upper right corner. The rest hit the power button on the laptop.
After rebooting, i showed them all they could just pull the usb drive to do it as well. As soon as tails started shutting down, they all assumed everything was safe and tried to power off the laptop.
On 2012-04-21, Andrew Lewman andrew@torproject.is wrote:
# Third issue: green onion
3 of 8 people saw the green onion appear in the menu bar up top. These three people hovered over it and saw the 'Connected to the Tor Network' message. No one knew to double-click on it to get a menu of other things to do. No one knew to right-click on it to get the drop-down menu.
What should they have wanted to do with Vidalia?
They were presented with the default check.torproject.org 'congratulations' page and then sat there.
# Fourth issue: check.tpo is not helpful
8 of 8 people saw the default check.torproject.org site telling them 'congratulations. Your browser is configured to use tor.' 7 of 8 people asked 'where is my browser?' The one who didn't ask this question was already a firefox user and recognized the interface. 0 of 8 understood what the IP address message meant. Comments ranged from 'is that different than my current IP address?' to 'what's an ip address?'
As an aside, when showing someone TBB on their own laptop, they saw the check.tpo site, and then went to Safari and started it up. When asked why they did this, the answer was 'safari is my browser. this says your browser is configured to use tor.'
That is exactly why I suggested the phrase “Congratulations. *This* browser is configured to use Tor.” (emphasis added) on https://bugs.torproject.org/2289 . But when I explained on IRC that there is a big difference between “this browser” and “your browser”, no one believed that users would interpret them differently.
Robert Ransom
On Sat, 21 Apr 2012 16:21:09 +0000 Robert Ransom rransom.8774@gmail.com wrote:
# Third issue: green onion
What should they have wanted to do with Vidalia?
Perhaps seeing the progress bar as tor bootstraps would have been helpful. Perhaps knowing they do have some control over this blackbox thing called tor browser would have been helpful.
Another thought is that using tor requires some education. Just downloading it and running is probably dangerous for a subset of users. One analogy is that tor is a race car. While many know how to drive a car on the street, driving a race car takes some education and practice to do safely.
That is exactly why I suggested the phrase “Congratulations. *This* browser is configured to use Tor.” (emphasis added) on https://bugs.torproject.org/2289 . But when I explained on IRC that there is a big difference between “this browser” and “your browser”, no one believed that users would interpret them differently.
I agree with you.
I spent the past week in Sweden, attending the Stockholm Internet Forum, http://www.stockholminternetforum.se/, for part of it. I made a number of tails 0.10.2 usb sticks for people on request. I also asked a lot of people their impressions of Tor and Tails. I received a plethora of feedback. All 8 people are involved in the Internet Freedom policy, technology, or freedom of speech communities. They had very different levels of self-assessed technical skill. The 8 people represented 6 countries.
Very interesting user study (of sorts)! If you were to guage each of the user's level of familiarity with Tor conceptually from 1 (never heard of it) to 5 (I use Tor often!), where would these 8 people fit?
I'm also curious how much explaining of Tails or Tor conceptually you did before the users started to interact with the system.
# Sixth issue: no flash, no warning
2 of 8 people had flash apps on their website. 4 of 8 had ad banners that used flash. All were surprised at the red outline with a snake in it appearing instead of their flash apps. None understood what happened.
After an explanation, one person suggested changing the red outline with snake to an actual message written inside, along the lines of 'this app blocked for your protection. click here to unblock it.' I explained why that wouldn't work (because there is no flash, java, silverlight plugins installed) and their answer was 'then do not show it at all'. Inside noscript, I unchecked the 'show placeholder..' option and had them browse again. they were happy. It seems if the user cannot do anything about the blocked apps, not showing them may be preferred.
This is a tricky usability minefield in my opinion, it's a really hard line to navigate between hiding something from a user they probably won't miss (ads) and hiding something from a user that would confuse the hell out of them (site flash intro .swf).
All that said, I might have to replicate your experiment and see what I can come up with :)
- warms0x --- xmpp: warms0x@riseup.net http: http://warms0x.github.com
I think this is useful info. There's also some previous user studies you've done lying around somewhere else too, yes?
I think it's unlikely for anything to come out of any of it without trac tickets, though. I've already forgotten where you even posted the previous user studies.
Thus spake Andrew Lewman (andrew@torproject.is):
I spent the past week in Sweden, attending the Stockholm Internet Forum, http://www.stockholminternetforum.se/, for part of it. I made a number of tails 0.10.2 usb sticks for people on request. I also asked a lot of people their impressions of Tor and Tails. I received a plethora of feedback. All 8 people are involved in the Internet Freedom policy, technology, or freedom of speech communities. They had very different levels of self-assessed technical skill. The 8 people represented 6 countries.
tldr; 8 people were tested, 8 people had trouble doing simple things with tor browser in Tails. Issues 4-6 are directly related to Tor.
And by asked, I mean, I stuck them in front of my laptop, put the usb stick in the computer, and asked them to browse to their favorite site. No one wanted to be video recorded, even if I offered to only record the screen and not audio.
Everyone managed to power on the laptop and wait for Tails to boot.
# First issue: Language selection
The first issue was on the language selection screen. 4 of 8 people were confused why it was called "Debian Live System" and not "Tails Live System". 8 of 8 knew what language selection meant, but weren't sure how this mapped to Tails.
# Second issue: wifi and tor browser
The tor browser starts up before the wireless is configured. The tor browser then reports a proxy error. With some prompting, all 8 figured out the wifi and then didn't know what to do. Tor does keep trying to load, and takes forever because it needs to download the entire directory. Users have no feedback as to what's going on behind the scenes because vidalia is hidden.
8 of 8 waited patiently for something to happen on the screen.
# Third issue: green onion
3 of 8 people saw the green onion appear in the menu bar up top. These three people hovered over it and saw the 'Connected to the Tor Network' message. No one knew to double-click on it to get a menu of other things to do. No one knew to right-click on it to get the drop-down menu. They were presented with the default check.torproject.org 'congratulations' page and then sat there.
# Fourth issue: check.tpo is not helpful
8 of 8 people saw the default check.torproject.org site telling them 'congratulations. Your browser is configured to use tor.' 7 of 8 people asked 'where is my browser?' The one who didn't ask this question was already a firefox user and recognized the interface. 0 of 8 understood what the IP address message meant. Comments ranged from 'is that different than my current IP address?' to 'what's an ip address?'
As an aside, when showing someone TBB on their own laptop, they saw the check.tpo site, and then went to Safari and started it up. When asked why they did this, the answer was 'safari is my browser. this says your browser is configured to use tor.'
No one used the language selections at the bottom of check.tpo, nor even understood why they were there.
# Fifth issue: exit relay congestion/failures
8 of 8 people tried to get to their own sites. 'I wonder what my site looks like when I'm anonymous' was the most common comment (5 of 8). For 6 of 8 people, their site didn't load at all, and tor browser reported their site was unreachable. All 6 then tried to go to google search in their own language; meaning google.es, google.se, etc. For 3 of those 6, this didn't work either. They gave up and assumed tor was broken or was censoring their destinations.
I intervened, opened the vidalia network map, closed the circuit in question, and asked them to repeat their browsing.
5 of 6 were able to get to their sites now. The one that was not able to had the same exit relay as last time, Amunet1, in a new circuit and just couldn't get anywhere through it. After yet another new circuit, they could get through to everything.
The user has no feedback as to why their site didn't work. And tor assumes everything is working fine.
When asked "please find a video you like", they all went to youtube. Most of the videos they wanted to see resulted in 'This video is currently unavailable.' 8 of 8 assumed it was because youtube was blocking tor, not because the video is flash-required. 2 of 8 started randomly clicking videos suggested by youtube to see if any of them worked. Eventually, 2 of 8 got videos to work with youtube and were amazed it worked at all.
# Sixth issue: no flash, no warning
2 of 8 people had flash apps on their website. 4 of 8 had ad banners that used flash. All were surprised at the red outline with a snake in it appearing instead of their flash apps. None understood what happened.
After an explanation, one person suggested changing the red outline with snake to an actual message written inside, along the lines of 'this app blocked for your protection. click here to unblock it.' I explained why that wouldn't work (because there is no flash, java, silverlight plugins installed) and their answer was 'then do not show it at all'. Inside noscript, I unchecked the 'show placeholder..' option and had them browse again. they were happy. It seems if the user cannot do anything about the blocked apps, not showing them may be preferred.
# Seventh issue: shutdown
I asked all 8 to shutdown tails and let me know when they thought their data was safely no longer on the system. 1 of 8 figured out how to shutdown tails by clicking the big red button in the upper right corner. The rest hit the power button on the laptop.
After rebooting, i showed them all they could just pull the usb drive to do it as well. As soon as tails started shutting down, they all assumed everything was safe and tried to power off the laptop.
-- Andrew http://tpo.is/contact pgp 0x6B4D6475 _______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Hi,
Andrew Lewman wrote (21 Apr 2012 14:42:54 GMT) :
I spent the past week in Sweden, attending the Stockholm Internet Forum, http://www.stockholminternetforum.se/, for part of it. I made a number of tails 0.10.2 usb sticks for people on request. I also asked a lot of people their impressions of Tor and Tails. I received a plethora of feedback. [...]
Thanks a *lot*, Andrew, for this testing and feedback! Much appreciated. Please go on. I'm sorry we did not reply to this earlier with words, but as you'll see bellow, we replied with actual changes to some of these issues.
# First issue: Language selection
The first issue was on the language selection screen. 4 of 8 people were confused why it was called "Debian Live System" and not "Tails Live System". 8 of 8 knew what language selection meant, but weren't sure how this mapped to Tails.
The language selection specific should be fixed by Tails >= 0.11 (no language menu anymore in bootloader), the "Debian Live" confusing message is fixed in Tails >= 0.12 (custom Tails syslinux menu).
# Second issue: wifi and tor browser
The tor browser starts up before the wireless is configured. The tor browser then reports a proxy error.
This is not the case anymore in Tails >= 0.11: iceweasel is started only after the network was successfully configured.
With some prompting, all 8 figured out the wifi and then didn't know what to do. Tor does keep trying to load, and takes forever because it needs to download the entire directory. Users have no feedback as to what's going on behind the scenes because vidalia is hidden.
I'm a bit surprised, since a notification about the time being sync'd, and this operation being necessary for Tor to work properly, should be displayed quite early since Tails 0.10.1.
8 of 8 waited patiently for something to happen on the screen.
OK.
# Third issue: green onion
3 of 8 people saw the green onion appear in the menu bar up top. These three people hovered over it and saw the 'Connected to the Tor Network' message. No one knew to double-click on it to get a menu of other things to do. No one knew to right-click on it to get the drop-down menu. They were presented with the default check.torproject.org 'congratulations' page and then sat there.
Hmm... they were asked "to browse to their favorite site", and did not do anything once they were presented a web browser? I'm at a loss... Any suggestions how we could make their experience better?
# Fourth issue: check.tpo is not helpful
8 of 8 people saw the default check.torproject.org site telling them 'congratulations. Your browser is configured to use tor.' 7 of 8 people asked 'where is my browser?' The one who didn't ask this question was already a firefox user and recognized the interface.
OK, this clarifies. So, this is a problem of our iceweasel not looking enough like "a" web browser, apparently. Oops. Probably check.tpo should be rephrased (s/your/this/, to start with) so that it's clearer That Thing is the web browser that's configured to use Tor.
0 of 8 understood what the IP address message meant. Comments ranged from 'is that different than my current IP address?' to 'what's an ip address?'
As an aside, when showing someone TBB on their own laptop, they saw the check.tpo site, and then went to Safari and started it up. When asked why they did this, the answer was 'safari is my browser. this says your browser is configured to use tor.'
This tends to confirm the need to s/your/this/ IMHO.
# Fifth issue: exit relay congestion/failures
8 of 8 people tried to get to their own sites. 'I wonder what my site looks like when I'm anonymous' was the most common comment (5 of 8). For 6 of 8 people, their site didn't load at all, and tor browser reported their site was unreachable. All 6 then tried to go to google search in their own language; meaning google.es, google.se, etc. For 3 of those 6, this didn't work either. They gave up and assumed tor was broken or was censoring their destinations. [...] The user has no feedback as to why their site didn't work. And tor assumes everything is working fine.
Wow, this is much worse than any experience I've had :/ I'm not sure how we can help from Tails' side.
When asked "please find a video you like", they all went to youtube. Most of the videos they wanted to see resulted in 'This video is currently unavailable.' 8 of 8 assumed it was because youtube was blocking tor, not because the video is flash-required. 2 of 8 started randomly clicking videos suggested by youtube to see if any of them worked. Eventually, 2 of 8 got videos to work with youtube and were amazed it worked at all.
OK, this is clearly suboptimal, to say the least, but it's so much better than it was before 0.10...
# Sixth issue: no flash, no warning
2 of 8 people had flash apps on their website. 4 of 8 had ad banners that used flash. All were surprised at the red outline with a snake in it appearing instead of their flash apps. None understood what happened.
After an explanation, one person suggested changing the red outline with snake to an actual message written inside, along the lines of 'this app blocked for your protection. click here to unblock it.' I explained why that wouldn't work (because there is no flash, java, silverlight plugins installed) and their answer was 'then do not show it at all'. Inside noscript, I unchecked the 'show placeholder..' option and had them browse again. they were happy. It seems if the user cannot do anything about the blocked apps, not showing them may be preferred.
I'm convinced. Adding as todo/NoScript:_hide_blocked_and_unsupported_elements to our TODO list.
# Seventh issue: shutdown
I asked all 8 to shutdown tails and let me know when they thought their data was safely no longer on the system. 1 of 8 figured out how to shutdown tails by clicking the big red button in the upper right corner. The rest hit the power button on the laptop.
This is unfortunate, but I'm not sure we can do much better than a big red button for first time users. Hopefully users who care (or should care) about RAM forensics eventually notice that button, or learn documentation...
After rebooting, i showed them all they could just pull the usb drive to do it as well. As soon as tails started shutting down, they all assumed everything was safe and tried to power off the laptop.
Any chance a graphical splash screen would work any better than the text-mode version of that we already have, which already explains what's going on (but probably a bit lost among other confusing text-mode message most users are not inclined to read)?
Thanks again, cheers! -- intrigeri | GnuPG key @ https://gaffer.ptitcanardnoir.org/intrigeri/intrigeri.asc | OTR fingerprint @ https://gaffer.ptitcanardnoir.org/intrigeri/otr.asc
-
Andrew Lewman -
intrigeri -
Mike Perry -
Robert Ransom -
warms0x