Hi!
Kevin Gallagher:
Hello tor-dev!
My name is Kevin and I'm a PhD student at NYU. Recently I've been working on creating a "Tor Friendliness Scanner" (TFS), or a scanner that will measure what features of a given website are broken (non-functional) when accessed on the Tor Browser (TB), along with actionable suggestions to improve it. In order to do this, we first must get an approximation of ground-truth data of how a given website should work. We then need to compare it to how the website works on the TB to determine any changes.
To generate a method of determining ground-truth, we decided to modify* the Firefox (FF) browser to log all of the steps of the creation of the Content Tree (also called the DOM tree), and to log the execution of all JavaScript functions (currently underway). We then will apply these changes to the TB as well, and run a scan of popular Web sites using the modified FF and the modified TB on all three of the TB security slider settings. We will then compare the resulting logs to determine where the tree creation processes differed* and why.
What are your criteria for saying "this is broken in Tor Browser" vs. "this is just rendered slightly different in Tor Browser"? For instance I suspect that you'd even get different ground-truths depending on the major Firefox version you use (like Firefox 65 vs. Firefox 60 ESR), yet you would hardly say "This is okay in Firefox 65 but broken in Firefox 60 ESR". Or maybe there *are* cases where you would say so? What I am saying is: mapping the creation of the DOM tree and logging JS execution might be a good means for you goal (I am not sure yet) but it does not seem to be sufficient to reach it.
Secondly, I am wondering how you plan to deal with the fact that websites show different content if the logic behind them assumes you come from a different country/region. How does that get incorporated into your ground-truth, for example?
Georg