Hello
I'm Rohit from India, aspiring for gsoc-2015(TOR). This will be my 2nd consecutive year for gsoc participation. Previous mediawiki. Project:BUB tool http://tools.wmflabs.org/bub/
I stumbled across Panopticlick related project https://www.torproject.org/getinvolved/volunteer.html.en#Panopticlick in tor project ideas. I would like to propose a project relating to this. Panopticlick obtains browser fingerprints mainly via javascript objects(navigator, screen, window etc.) These objects are easy to fake in webkit browsers, without touching the underlying source code of browsers, eg. using *__defineGetter__() *after every* javascriptObjectCleared.* If we could compile a large dataset of possible values of js object for several popular browsers, we could use that to randomize the fingerprint for each network request. The dataset could also contain random http header values etc.
I am building a python library that does somewhat similar. https://github.com/rohit-dua/selkie (*in development*) It uses pyqt for headless browsing/scraping of webpages. It is a python library that mimics different browser fingerprints by faking(randomizing) the values of navigator, screen object, headers etc. I also intend to add biometric library that mimics humans mouse movements/ keypress statistics for clicking links and surfing pages.
I propose to build a similar headless bot that mimics several browsers fingerprints and could be used for anonymous scraping of data and/or adding a feature of random fingerprint in awesome tor tools. Also to improve anonymity location based datasets could be provided(*supported in the above library*) as extra/feature.(maybe downloaded from statcounter.com)
Thanks Rohit Dua IRC:rohit-dua github: rohit-dua https://github.com/rohit-dua/ (8ohit.dua@gmail.com)
I think I like this idea.. Well done.
On Wednesday, February 11, 2015, Rohit Dua 8ohit.dua@gmail.com wrote:
Hello
I'm Rohit from India, aspiring for gsoc-2015(TOR). This will be my 2nd consecutive year for gsoc participation. Previous mediawiki. Project:BUB tool http://tools.wmflabs.org/bub/
I stumbled across Panopticlick related project https://www.torproject.org/getinvolved/volunteer.html.en#Panopticlick in tor project ideas. I would like to propose a project relating to this. Panopticlick obtains browser fingerprints mainly via javascript objects(navigator, screen, window etc.) These objects are easy to fake in webkit browsers, without touching the underlying source code of browsers, eg. using *__defineGetter__() *after every* javascriptObjectCleared.* If we could compile a large dataset of possible values of js object for several popular browsers, we could use that to randomize the fingerprint for each network request. The dataset could also contain random http header values etc.
I am building a python library that does somewhat similar. https://github.com/rohit-dua/selkie (*in development*) It uses pyqt for headless browsing/scraping of webpages. It is a python library that mimics different browser fingerprints by faking(randomizing) the values of navigator, screen object, headers etc. I also intend to add biometric library that mimics humans mouse movements/ keypress statistics for clicking links and surfing pages.
I propose to build a similar headless bot that mimics several browsers fingerprints and could be used for anonymous scraping of data and/or adding a feature of random fingerprint in awesome tor tools. Also to improve anonymity location based datasets could be provided(*supported in the above library*) as extra/feature.(maybe downloaded from statcounter.com)
Thanks Rohit Dua IRC:rohit-dua github: rohit-dua https://github.com/rohit-dua/ (8ohit.dua@gmail.com javascript:_e(%7B%7D,'cvml','8ohit.dua@gmail.com'); )
Do you think this project idea(or related) can be turned into gsoc-2015 project?
On Thu, Feb 12, 2015 at 10:16 AM, Andrew F andrewfriedman101@gmail.com wrote:
I think I like this idea.. Well done.
On Wednesday, February 11, 2015, Rohit Dua 8ohit.dua@gmail.com wrote:
Hello
I'm Rohit from India, aspiring for gsoc-2015(TOR). This will be my 2nd consecutive year for gsoc participation. Previous mediawiki. Project:BUB tool http://tools.wmflabs.org/bub/
I stumbled across Panopticlick related project https://www.torproject.org/getinvolved/volunteer.html.en#Panopticlick in tor project ideas. I would like to propose a project relating to this. Panopticlick obtains browser fingerprints mainly via javascript objects(navigator, screen, window etc.) These objects are easy to fake in webkit browsers, without touching the underlying source code of browsers, eg. using *__defineGetter__() *after every* javascriptObjectCleared.* If we could compile a large dataset of possible values of js object for several popular browsers, we could use that to randomize the fingerprint for each network request. The dataset could also contain random http header values etc.
I am building a python library that does somewhat similar. https://github.com/rohit-dua/selkie (*in development*) It uses pyqt for headless browsing/scraping of webpages. It is a python library that mimics different browser fingerprints by faking(randomizing) the values of navigator, screen object, headers etc. I also intend to add biometric library that mimics humans mouse movements/ keypress statistics for clicking links and surfing pages.
I propose to build a similar headless bot that mimics several browsers fingerprints and could be used for anonymous scraping of data and/or adding a feature of random fingerprint in awesome tor tools. Also to improve anonymity location based datasets could be provided(*supported in the above library*) as extra/feature.(maybe downloaded from statcounter.com)
Thanks Rohit Dua IRC:rohit-dua github: rohit-dua https://github.com/rohit-dua/ (8ohit.dua@gmail.com)
tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Hi,
For anonymous scraping it could certainly be useful. This poses a problem as far as making Tor Project look as if it supports autonomous anonymous scraping of web data. Ultimately this impression could lead to even more blocking of Tor exits. Another problem with the idea of a randomized fingerprint is that it breaks useability. It might be great for scraping but web sites rely on knowing some of those parameters for proper display. Finally it's worth mentioning that the goal of TBB fingerprinting is to reduce entropy within TBB's user base. A random fingerprint violates this constraint.
I'm not commenting on gsoc eligibility--just that it's an edge case which will lead to blocking of Tor's exits. If more exit get blocked then you cannot scrape.
--leeroy
Hi Rohit,
Please check the ticket #11949 and the comment by Georg: https://trac.torproject.org/projects/tor/ticket/11949#comment:1
TL;DR research on the advantages of randomization over the current approach (making everyone look like same) may be useful before starting with the actual implementation.
Also, please check this thread on the limitations of JS hooks: https://lists.torproject.org/pipermail/tbb-dev/2014-June/000073.html
You can fool some fingerprinters by spoofing browser properties but more advanced scripts can easily uncover the real browser/device attributes by checking specific functionality [1] or using "side-channels" [2].
[1] see, "Evolution of functionality" subsection on https://seclab.cs.ucsb.edu/media/uploads/papers/sp2013_cookieless.pdf#page=1...
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=418986, see, esp. Camillo's test vectors.
Gunes
On 02/12/2015 06:12 PM, l.m wrote:
Hi,
For anonymous scraping it could certainly be useful. This poses a problem as far as making Tor Project look as if it supports autonomous anonymous scraping of web data. Ultimately this impression could lead to even more blocking of Tor exits. Another problem with the idea of a randomized fingerprint is that it breaks useability. It might be great for scraping but web sites rely on knowing some of those parameters for proper display. Finally it's worth mentioning that the goal of TBB fingerprinting is to reduce entropy within TBB's user base. A random fingerprint violates this constraint.
I'm not commenting on gsoc eligibility
+1 --just that it's an edge case
which will lead to blocking of Tor's exits. If more exit get blocked then you cannot scrape.
--leeroy
tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Thank you all for replying.
I can summarize some points after reading the links provided.
The random fingerprint headless browser should not be connected with Tor network as this can lead to more frequent blocking of exit nodes(due to more scraping bots using exit nodes)
A random fingerprint should be better than current approach(single fingerprint) because a company/government can easily block access to that one static/single while this will be unlikely for a random fingerprint from a very large dataset.(This will be very good for headless browsing)
Faking a fingerprint requires several properties. But most of them can be overwritten. Some of the ones with most problems: 1.) Faking fonts: webkit does not provide way to manually disable/enable selected fonts to be used. A way around this could be to disable all fonts and then load random fonts using @font-face just after javascriptObjectCleared, for each request. 2.) Timezone could easily be faked after overwriting Date object. 3.) Plugins/MimeTypes/navigator/colorDepth can be faked by overwriting their object. 4.) Window and Screen objects including innerHeight/screenWidth/ScreenHeight can also be overwritten but only for headless browsing as changing these will hinder the user view experience. 5.) Almost all browser unique features like screen.mozBrightness,navigator.mozSms,InstallTrigger[FIREFOX], navigator.webkitStartActivity,navigator.getStorageUpdates,window.chrome[CHROME] can be added explicitly to dom for corresponding browser specific fingerprint. 6.) The fingerprint properties should be location specific(eg. the accept-language headers should including the langauges of the location) so as to avoid statistics based attack(based on location data) 7.) To avoid timing attack, a local proxy can be used which will add an extra random delay time(based on location) in all TCP/IP packets sent(including syn/ack) 8.) Flash/Java has to disabled to avoid information leakage. While the plugins can still be faked as available in the Plugins Object. 9.) To avoid mouse cursor statistics based attack for headless browser, mouse movements can be faked with certain degree of randomness like a human.(several libraries are available for this.)
The headless bot can be built on a framework like qtwebkit.
Do you think headless browser with random fingerprint without Tor usage will be a useful project?
- Rohit Dua
On 2/12/15, Gunes Acar gunes.acar@esat.kuleuven.be wrote:
Hi Rohit,
Please check the ticket #11949 and the comment by Georg: https://trac.torproject.org/projects/tor/ticket/11949#comment:1
TL;DR research on the advantages of randomization over the current approach (making everyone look like same) may be useful before starting with the actual implementation.
Also, please check this thread on the limitations of JS hooks: https://lists.torproject.org/pipermail/tbb-dev/2014-June/000073.html
You can fool some fingerprinters by spoofing browser properties but more advanced scripts can easily uncover the real browser/device attributes by checking specific functionality [1] or using "side-channels" [2].
[1] see, "Evolution of functionality" subsection on https://seclab.cs.ucsb.edu/media/uploads/papers/sp2013_cookieless.pdf#page=1...
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=418986, see, esp. Camillo's test vectors.
Gunes
On 02/12/2015 06:12 PM, l.m wrote:
Hi,
For anonymous scraping it could certainly be useful. This poses a problem as far as making Tor Project look as if it supports autonomous anonymous scraping of web data. Ultimately this impression could lead to even more blocking of Tor exits. Another problem with the idea of a randomized fingerprint is that it breaks useability. It might be great for scraping but web sites rely on knowing some of those parameters for proper display. Finally it's worth mentioning that the goal of TBB fingerprinting is to reduce entropy within TBB's user base. A random fingerprint violates this constraint.
I'm not commenting on gsoc eligibility
+1 --just that it's an edge case
which will lead to blocking of Tor's exits. If more exit get blocked then you cannot scrape.
--leeroy
tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Hi Rohit, sorry about the delay. I've been trying to get ahold of the project mentor for panopticlick but no luck, so I'm dropping it from the volunteer page. You're certainly still welcome to take it on, but for the purposes of GSoC we seem to be missing a potential mentor for it.
Sorry about the confusion! -Damian
On Thu, Feb 12, 2015 at 1:53 PM, Rohit Dua 8ohit.dua@gmail.com wrote:
Thank you all for replying.
I can summarize some points after reading the links provided.
The random fingerprint headless browser should not be connected with Tor network as this can lead to more frequent blocking of exit nodes(due to more scraping bots using exit nodes)
A random fingerprint should be better than current approach(single fingerprint) because a company/government can easily block access to that one static/single while this will be unlikely for a random fingerprint from a very large dataset.(This will be very good for headless browsing)
Faking a fingerprint requires several properties. But most of them can be overwritten. Some of the ones with most problems: 1.) Faking fonts: webkit does not provide way to manually disable/enable selected fonts to be used. A way around this could be to disable all fonts and then load random fonts using @font-face just after javascriptObjectCleared, for each request. 2.) Timezone could easily be faked after overwriting Date object. 3.) Plugins/MimeTypes/navigator/colorDepth can be faked by overwriting their object. 4.) Window and Screen objects including innerHeight/screenWidth/ScreenHeight can also be overwritten but only for headless browsing as changing these will hinder the user view experience. 5.) Almost all browser unique features like screen.mozBrightness,navigator.mozSms,InstallTrigger[FIREFOX], navigator.webkitStartActivity,navigator.getStorageUpdates,window.chrome[CHROME] can be added explicitly to dom for corresponding browser specific fingerprint. 6.) The fingerprint properties should be location specific(eg. the accept-language headers should including the langauges of the location) so as to avoid statistics based attack(based on location data) 7.) To avoid timing attack, a local proxy can be used which will add an extra random delay time(based on location) in all TCP/IP packets sent(including syn/ack) 8.) Flash/Java has to disabled to avoid information leakage. While the plugins can still be faked as available in the Plugins Object. 9.) To avoid mouse cursor statistics based attack for headless browser, mouse movements can be faked with certain degree of randomness like a human.(several libraries are available for this.)
The headless bot can be built on a framework like qtwebkit.
Do you think headless browser with random fingerprint without Tor usage will be a useful project?
Rohit Dua
On 2/12/15, Gunes Acar gunes.acar@esat.kuleuven.be wrote:
Hi Rohit,
Please check the ticket #11949 and the comment by Georg: https://trac.torproject.org/projects/tor/ticket/11949#comment:1
TL;DR research on the advantages of randomization over the current approach (making everyone look like same) may be useful before starting with the actual implementation.
Also, please check this thread on the limitations of JS hooks: https://lists.torproject.org/pipermail/tbb-dev/2014-June/000073.html
You can fool some fingerprinters by spoofing browser properties but more advanced scripts can easily uncover the real browser/device attributes by checking specific functionality [1] or using "side-channels" [2].
[1] see, "Evolution of functionality" subsection on https://seclab.cs.ucsb.edu/media/uploads/papers/sp2013_cookieless.pdf#page=1...
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=418986, see, esp. Camillo's test vectors.
Gunes
On 02/12/2015 06:12 PM, l.m wrote:
Hi,
For anonymous scraping it could certainly be useful. This poses a problem as far as making Tor Project look as if it supports autonomous anonymous scraping of web data. Ultimately this impression could lead to even more blocking of Tor exits. Another problem with the idea of a randomized fingerprint is that it breaks useability. It might be great for scraping but web sites rely on knowing some of those parameters for proper display. Finally it's worth mentioning that the goal of TBB fingerprinting is to reduce entropy within TBB's user base. A random fingerprint violates this constraint.
I'm not commenting on gsoc eligibility
+1 --just that it's an edge case
which will lead to blocking of Tor's exits. If more exit get blocked then you cannot scrape.
--leeroy
tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
-
Andrew F -
Damian Johnson -
Gunes Acar -
l.m -
Rohit Dua