Sorry in advance for the length. I just want to make sure that everything is included. If you have any questions/clarifications, just ask. It isn't every day that someone like me pops up on tor-dev, and I just want to make sure this is as productive and helpful as possible. This will probably be a very humbling experience, because unlike my fellow illegal onion operators both past and present, I will actually be outside of a jail cell and able to read the ruthless dissection of my set-up. On the bright side, you're all are getting way more info from me than the pigs will ever willingly cough up, which means if they have some sort of magic onion decloak trick, this mailing list discussion is a good chance at finding it. All of these files are in the hands of the cops anyway (And I have no plans of bringing doxbin back), so there are 0 real-time opsec concerns.
First, the files:
http://doxbin.strangled.net/ http://qhlkmirbijvet2dp.onion/
sha256 and sha512 checksums for all the files are at the bottom. NOTE: This isn't my box and I didn't set it up. WARNING: The .xz files will unpack to roughly 1 GB.
Some other info:
1. The box (An OpenVZ VPS) was hosted with Hetzner in Germany. People on Twitter keep asking if the box was in Bulgaria, but we didn't use Bulgaria for one simple reason: The very first doxbin box (Bought in 2011) was with a VPS company in Bulgaria. After the first month, they said "TOR IS ILLEGAL" (I shit you not), killed the box, and kept the $5 we had paid for the 2nd month. Can't say I blame them.
2. It wasn't transproxied. We did this once before, but it became a hassle to drop iptables rules just to upgrade tor, especially when tor was getting regular updates semi-recently. Console access with VPS companies generally requires java plugins or something just as gross, so there aren't a lot of sane options here. It gets annoying when there's a surge in the frequency of tor upgrades, so we stopped transproxying the box. inb4 OMG OPSEC MISTAKE DETECTED!!!11
3. This script (And copypasta that contained its important parts) was used to build nginx for the various boxes from roughly November 2011 to mid-October 2014: http://pastebin.com/dBC7E8Jd Early versions didn't use naxsi, and sed replaced the server banners in the source before compiling. Yes, I know it doesn't verify the source. See the next point:
4. Around 2 weeks ago, I started grabbing dotdeb.org's source versions of nginx and building .debs using hardening-wrapper with the following command at the end:
DEB_CFLAGS_SET="-O2 -fstack-protector-all -fforce-addr -ffast-math -fomit-frame-pointer -falign-functions=64 -falign-loops=32" dpkg-buildpackage -uc -us -j8
5. The tor build was from deb.torproject.org. I'm not able to check the box and never thought to back up /etc/apt/sources, but it used the experimental-wheezy repo. All updates were done within 24 hours of the new .debs going live.
6. Last but not least, the elephant in the room: The php is a headache-inducing nightmare. I inherited the code and it worked, so I just papered over its defects over the years (There was no anti-flood protection originally, for example) and built it up to resemble Kowloon Walled City expressed in php. I have a feeling that a lot of focus is going to be placed on the code, to the detriment of finding any possible tor bugs. In any case, everything about doxbin's setup is being disclosed in the spirit of making this an interesting learning experience for all parties involved.
BEGIN TINFOIL
Upon scrolling through the .xz files (I personally use xzless), you'll find a bunch of stuff like:
1 /%5C%22http://www.hackforums.net/code/fail/code/code/code/code/code/code/old/code/o...
All of the requests were around (If I recall correctly) 3KB in size. Oddly enough, it caused tor to hiccup pretty badly, although the web server itself was just fine and I didn't have any network bandwidth problems (i.e. No obscene traffic spikes). It's also worth pointing out that the tcp buffers weren't even close to maxed. The same box has handled a similar volume of legitimate requests before (Namely back in March, after The Hidden Wiki debacle; see https://twitter.com/loldoxbin/status/530765088366821377). The solution to getting tor availability back was to set ConstrainedSockets to 1 and play with ConstrainedSockSize (Originally set to 8192, then 4096). This made doxbin regularly available again, whereas before it was hit or miss. Once the requests stopped, I waited a couple of days before commenting those two config lines out and reloaded tor.
A month later, the same kinds of requests started coming in again. After the first few hours, I started 301 redirecting all requests containing /%5C%22 to the new Hidden Wiki's Hard Candy page. I also added a grep -v to my log report script in order to filter out the noise (Possibly a mistake, but we both tailed logs and watched for something like a different attack style that the ddos was being used to cover and never noticed anything). That was good enough to maintain availability, so I rolled with it and the requests eventually stopped. I have no hard data on that last point, just the fact that I tailed the access log and the requests went from 5 per second to 1 every 3-6 seconds before dying off completely.
I said on Twitter that we suspected it was a deanonymization attempt, but I didn't elaborate why because LOL 140 CHARACTERS. Intangir (The other admin, who took over for me from Halloween 2013 to some time around July of this year) and I talked about it back in August and decided that an attacker was probably involved spraying specially crafted packets at the box in order to mess up its circuits, and eventually get us on attacker-controlled nodes. Since we mitigated the availability loss, we deemed it as no big deal. In hindsight, that seems hilariously stupid/naive of us. The kid who started doxbin had a similar theory that I'm just going to paste verbatim:
<founder> ANYWAY <founder> i think <nachash> CONTINUE <founder> the attack <founder> was to DoS you <founder> until you created circuits <founder> entirely made of dickbleedable nodes <founder> and then dickbleeding them <nachash> but the server <nachash> got seized <founder> yeah, the IP was discovered by dickbleed though <founder> the entire circuit was leaking info <nachash> lol, did you just reproduce this? <founder> not yet, i'll be trying <nachash> Do you mind if I share this with tor devs? <founder> go ahead <founder> its just a theory at the moment
If either of these theories even remotely pan out, a possible mitigation for the next person like me (Which shouldn't require any tor dev work) might look something like this:
1. Make several public relays and configure the torrc of the hidden service to only make circuits which begin with those nodes. Private bridges would be a liability in this case because anyone who figures out the guard node by weaponizing one of attacks from a whitepaper (Which guard nodes were modded to mitigate) will be able to find the guard node and then quickly discover that it isn't part of the public tor network. Public bridges might be ok, but probably less so. 2. Cross your fingers and pray really, really hard that the money trail is correctly obscured. 3. ???? 4. PROFIT! (Or lulz, in my case)
(No, the similarity between the idea of drug market operators giving back to the network by donating nodes to the practice of drug cartels building schools and hospitals is not lost on me)
Another thing to consider is the recommendation against running a hidden service as a relay. Of course, the argument against doing so in the documentation is very sound. At the same time, the FBI has stated in the Silk Road criminal complaint that upon finding the IP, the investigator verified that it wasn't a tor node and knew he had won the onion lottery. Of course, that could still be psyops/parallel construction garbage to scare people into making their jobs a little easier, but it's something to thing about.
END TINFOIL
sha256sums of all the files:
04261115c2ca8d28c439ee16926e271479b8626cc9403d31d686317227f9f4b0 archive.php.txt 2811be83aa41666f245432a1a562a3ac6c7035a59a16beae09571a71723e734c captcha.php.txt c5b9bff30748b8d68666f6d4687757f955cde96e67bbb00edad703af45b776ae doxbin_2014_08_21.txt.xz cde066ef290ee22c8d78a34af17d33a393845105d21f9fd96db93e0a51290b77 doxbin_2014_08_22.txt.xz 43ec183ee7efc3e9d14d084e50fe054ac6ffb60919e596a8dc4824f116fa1d83 doxbin_2014_08_23.txt.xz 3e630b3843341926ab7667a41a023fa4d63a38d01a5a3549a66f8bc98507cd39 doxbin_2014_08_24.txt.xz 20af2efec930e477fe5595eee66418f87af047ad4316ffc399761002c2d5f45d doxbin_2014_08_25.txt.xz 1c24eecbc98456a4c2f489a6012a48ec2119927402e7dd1ac12cff7732d04192 doxbin_2014_08_27.txt.xz afcdafc5293b6a9dbe402da65522f01a231f3f9727a4e597e70cf44321f96c2f doxbin_php.txt 66d495a5d21df855eedc6e3493a159b2b522b1b8170610c7c7e013f31d01e0ba doxviewer.php.txt 34611133cb4da451fc45f09e2ca6a08a71267439755b68608cbbc9b2a151ccee error.php.txt f37e6be62b149e40fb604cef90313d321f2bf4d2156808ef997a82004307c22f faq.php.txt 10bc8b38757ad7b27576488a37cec4a2f9dd0ed2ef467cc605ac0da8e83f3e23 index.php.txt 9596d7339f7a89eca7f18dd36cc2f433cdaa23700497816a6a71889158f90bc7 legal.php.txt 912241933303556097d60b890bac18be4577cbefaa6794088b256a3bf1f3796e naxsi_core.rules.txt 36bfb91a072d43a07de816d78a7cae8b7350a7f8069798468c8f3556acbeab9b nginx.conf.txt 66df06cdf6e6aba5dab8655e72c30bc5020f2319db1329c2f93bc74e96a96ac4 post.php.txt b9c12279b289a56702c9a3dc9b873b453ca4642efbcacba597c8bd252a9092ab privacy.php.txt 728fb96cbecafd974a9afc3211e8d9eb7bbb49f73e3ccb64fe9f7390341b3d38 proscription.php.txt ae0b124f3ccd849653ef806b10dd41cf316ffd45ef5d8d5af8c2d117027835e4 resources.tar.xz 7ba7ab341f29fc6d5a449bcd2f18c8962b8475b34c8b5745676da4f4c8e51a90 search.php.txt 6aa7c5dbe7aaa25b729ab542e0396f7f7d322244935ea4c0ad0e468b01d19374 thw.php.txt
sha512sum:
7a0e5eaed12d6f7134c56cdd6e68e8255e49c00897fe05701abb84e91d215ed23f462d6b0485467da04a2d7c0a51d5e6a3fece943d9ce2137321a0913448afec archive.php.txt 804d679233f811aa66ca6fbc370e0734c7ba830959165496c66a0585637e4c5f966e6dfcbed618bea65ab11aa2f49351840376abc553e74c84001cf5e737da2b captcha.php.txt 734adca606f984fce830ebb5d5a05dee0c75d06ff05498483ff057cfd879f670f1a6bd42f5ad090bd63edd0f54d8eed86a0aa04cfe60669c11ac141f271ec366 doxbin_2014_08_21.txt.xz c9b55b8b9cf3d3f5820b8cde5b3de77cd609b17c1f6c234118b4b632978a8ef131227500fac66e2238756014437672178515ba028b70f8dc29b0b3b7ef158632 doxbin_2014_08_22.txt.xz c63ab075b0383be0703c97bf9fbe42e1dbb3f1ab01eddddbe1752af12bd38a71d6906c2c4c18695b0afa091049ca8583a86b747690cea17c45bd4edcaf269611 doxbin_2014_08_23.txt.xz 4486fbf61e5132a6167f46c7ec88f7fe25d879e72094d43b3189f786ad84a4e3fd75adc76fddf4ece26b7b670091d29e6a5e7631fd0660b4d549cc49d2e8bcfc doxbin_2014_08_24.txt.xz 1ac1e9359092d77e0a13a404ea428cc396b22a8f89dd7bb3241a80a4aa16a3d769aacf0289d1c83e3e367e24ca0a922ca9b3d950a14f962defd6f7c69c62cc63 doxbin_2014_08_25.txt.xz e6722d43221a13d6df3f521b040d11a17cd27a63017a73893bf25396468127bbb7f8d81b1b4c7c0aba52e82b09419e63c3df9df877c68fc7603fc9f19b8bda20 doxbin_2014_08_27.txt.xz c9d20c82b473260473824ed4909c165f10a3a1a93405cab1ac28d8ae17ee839754998f0d6e7e78c09fec4e21bf4efd6c65ece4734c87f437199cd256cd6756ab doxbin_php.txt b73db1ceefdc501f6ec0fd52c368a88f1e88d80049ed2824fb4a8c60d2c70a87079528f1a043bca6fd2d073ce186eb87531a9cf69b449f537a041786e4f634f3 doxviewer.php.txt 4f4702f4f4970f67f73b23ca357f764f62eddd63a6fa7878d1c0aeaa2f48ed3813e84261a278ee396aa3e0c6dbe4e5333d8d83b85d173e13be9bea337cf7a678 error.php.txt 6cfc57fd6f3aaf9a7829b83f8a43852e965dc11929be29f59c494bd8b5723b0fee9aa9ccdc734b8bb07127555d14a8619343061d6fe88508afe7f9ba569ac376 faq.php.txt d568388cd9abcad5c0db31a25feef410f7a1a1b26ca3d542b38d1e75c232dbc6d852040dffb3d754d74aac6ecf04e996844b673d4b74ac00c05bb920f63528d1 index.php.txt 35164a260e89a3ef296460ebfba7b3f5b902b4594fcbc5871e1b431f6c4b89c2e80ae372c50fd6a30d50edeaa94e7326c0cbefca92537e37a3deddbfa610e8f7 legal.php.txt 93fcbfd36e299239ec2e350fc9a768b9c819089a6922aa997c377d205e7895cd3f6beef00fbadf7de0e02f26efd3ce7a0e47c5d927ae5eb4c9c256c4be58855e naxsi_core.rules.txt 1c0ac0e10e715ef8049ea7d1ae3fc635b96a9d62811137e98a4c3794f8ba86f5b66ea883d5ebe939d7cb607cc575e04d8c47ee4ddaabf0887ef44e99c56c4a0a nginx.conf.txt 3f29d8d34bc91e1114f089a937bf736b7b74837f5aeb834ae0ace6e7612e2b401bc8f769fa74a432533d92c28b7165eb31dc40eb7ffbfaf82dd870815cc4e189 post.php.txt 8079dff1ceb2309d98198d3503db8bdaeb38bcdf9679548e8fe7ce1a8bbc87506f8d165713d90c7d2f6316031c1bd7eb7765441d325741c8bd5ca6bc2a6a2843 privacy.php.txt a1f6d218a2f65617203d53fbbf625da362d9f2e2676fbc1a28491a1717a6c21367b0a241bd1599c336be2d31d085717b16b3efaffeb383e0a11ce73839ce648f proscription.php.txt 94b8bd2fb8b25880e1c25d5fcf60ebe93d3c42857ef61cdf5261dfcaaf1d34cc5d085c358793d22dc7cfd66dcb7628f0eb9953560c3c646d2b7abc08a368c4d3 resources.tar.xz 0fee75a98e3f8ae4d0c578af399144c49382c7680626cc62e01aa250fad11d1aea522e87ec8e79e6136632bd54bffe681b2ecc283e1d56a7e75b7ab17f7e2319 search.php.txt e69bdba3aeb6c94a3b785b96385fafe0ec58e66688333f423004da1a4725bd54666f1e0b427d52068cdbefa430146162f1511c3c0930ea4379f6b022756a48bf thw.php.txt
Hope this helps,
- nachash