Hello,
Recently I've been fighting censorship[1] in the form of deep packet inspection.
Between the OpenVPN I an obfsproxy server outside the country and a client inside the country. But what I've found out is that Obfsproxy server needs to be running as the OpenVPN server and Obfsproxy client needs to be on the same machine as the OpenVPN client.
This makes stuff work but its not really optimal, since there are area of the country where internation traffic is really hard, having proper address translation would save the hassle of setting another proxy between obfsproxy and the clients would not need anything more than the OpenVPN client that they already have.
I opened issue #14217[2] and would like to provide a patch for it.
But I am not Python and Twisted expert, so I will be firing a few questions:
- How do I debug twisted ? Given the event-diven nature of the library, how do I debug incoming packets ? - How do you write server client tests? - What part of the code is reponsible from translating the incoming packeets on the server back to the client ?
Thanks in advance !
[1] https://forums.openvpn.net/topic17960.html [2] https://trac.torproject.org/projects/tor/ticket/14217
-- Regards, Alfredo Palhares
On Fri, 16 Jan 2015 12:05:41 +0100 Alfredo Palhares masterkorp@masterkorp.net wrote:
Between the OpenVPN I an obfsproxy server outside the country and a client inside the country. But what I've found out is that Obfsproxy server needs to be running as the OpenVPN server and Obfsproxy client needs to be on the same machine as the OpenVPN client.
So what you're saying is, you want to do something like:
* There is an obfsproxy client instance running on c.example.com.
* There is an obfsproxy server instance running on s.example.com, that feeds into an OpenVPN server instance running on v.example.com.
* Multiple clients use c.example.com as the SOCKS proxy for the OpenVPN client, connect to s.example.com to get to the OpenVPN server running on v.example.com.
My thoughts on the matter are:
1. This should work. If it can be shown to be broken via a trivial application/test case (Eg: netcat), then it should be fixed (The trival test case requirement is because I don't want to debug OpenVPN again).
2. Oh god, c.example.com is going to be running a public SOCKS proxy. Granted people trying to use it to get to most destinations will have a connection that fails, but bad people can use it as a DDoS amplification host (The SOCKS dialog is much much shorter than any of the client requests that would be sent).
3. I don't know enough about the OpenVPN protocol/implementation to know if there are application specific quirks unique to OpenVPN that would prevent this configuration from working. That would be an OpenVPN problem, unless obfsproxy is altering the data it's relaying (Extremely unlikely).
I'll hold off on closing the ticket for now, but unless the code is broken in the "1." sense, I'm inclined to do so.
Regards,
Hello Yawning,
- There is an obfsproxy client instance running on c.example.com.
There is an obfsproxy server instance running on s.example.com, that feeds into an OpenVPN server instance running on v.example.com.
Multiple clients use c.example.com as the SOCKS proxy for the OpenVPN client, connect to s.example.com to get to the OpenVPN server running on v.example.com.
Correct with the exception that the machines won't be public.
My thoughts on the matter are:
- This should work. If it can be shown to be broken via a trivial application/test case (Eg: netcat), then it should be fixed (The trival test case requirement is because I don't want to debug OpenVPN again).
OK, what would be the best way to present this data?
- Oh god, c.example.com is going to be running a public SOCKS proxy. Granted people trying to use it to get to most destinations will have a connection that fails, but bad people can use it as a DDoS amplification host (The SOCKS dialog is much much shorter than any of the client requests that would be sent).
The SOCKS proxy will not be public far from that. The advantage on that is that I setup SOCKS proxy on the best location that for international traffic. On this specific country international traffic is very hard from most locations, just because the whole censorship infrastructure can't deal with all the traffic. Having the possibility to set it remotely would save work on deploying obfsproxy trough the OpenVPN clients and to set up yet another HTTP proxy in between the proxies.
- I don't know enough about the OpenVPN protocol/implementation to know if there are application specific quirks unique to OpenVPN that would prevent this configuration from working. That would be an OpenVPN problem, unless obfsproxy is altering the data it's relaying (Extremely unlikely).
How can I get proof of this ?
I am willing to dig into the code and provide a patch. But I could use some help with obfsproxy internals.
Thank you for your reply.
-- Regards, Alfredo Palhares
On Fri, 16 Jan 2015 16:17:38 +0100 Alfredo Palhares masterkorp@masterkorp.net wrote:
My thoughts on the matter are:
- This should work. If it can be shown to be broken via a trivial application/test case (Eg: netcat), then it should be fixed (The trival test case requirement is because I don't want to
debug OpenVPN again).
OK, what would be the best way to present this data?
The easiest quick and dirty one off test I can think of for something like this would be (Using the same hostnames as the example e-mail, fill in the ports to taste):
c.example.com: $ obfsproxy --log-min-severity=debug \
--log-file=/tmp/obfsproxy-client.log \ obfs3 socks 0.0.0.0:CPORT
s.example.com: $ obfsproxy --log-min-severity=debug \
--log-file=/tmp/obfsproxy-server.log \ obfs3 --dest vpn.example.com:NPORT \ server 0.0.0.0:SPORT
vpn.example.com: $ nc -l NPORT
client: $ nc -x c.example.com:CPORT s.example.com SPORT
Typing into either netcat session should send bytes to the other side, on each newline (the newline requirement is due to the TTY code), and has nothing to do with obfsproxy. Naturally c.example.com:CPORT, s.example.com:SPORT and vpn.example.com:NPORT must allow incoming TCP/IP connections.
If this doesn't work, c.example.com:/tmp/obfsproxy-client.log and s.example.com:/tmp/obfsproxy-server.log should have enough information to investigate further. Make sure that there is no sensitive information in either logfile, and attach it to the ticket along with the exact commands used on each host.
If this does work, you are on your own from that point, because obfsproxy is doing exactly what it is supposed to do (relaying data).
- Oh god, c.example.com is going to be running a public SOCKS
proxy. Granted people trying to use it to get to most destinations will have a connection that fails, but bad people can use it as a DDoS amplification host (The SOCKS dialog is much much shorter than any of the client requests that would be sent).
The SOCKS proxy will not be public far from that. The advantage on that is that I setup SOCKS proxy on the best location that for international traffic. On this specific country international traffic is very hard from most locations, just because the whole censorship infrastructure can't deal with all the traffic. Having the possibility to set it remotely would save work on deploying obfsproxy trough the OpenVPN clients and to set up yet another HTTP proxy in between the proxies.
This sounds "public" to me, but ok, I'll take your word for it.
Regards,
Hello,
Sorry for the delay, I finnally set on the quest to understand obfsproxy. So with the help of pdb and a few questions here and there I got to some places, and with a lot of more questions to ask.
## Studying the client
The client launches a connection on obfsproxy/pyofsproxypy do_external_mode() the launch_transport.launch_transport_listener method is where the socks listeners is set. Right after that the Twisted library reactor module is initiated with reactor.run() on line 107 of the file obfsproxy/pyobfsproxy.py
After that the Twisted event loop is started and it will listen to events that are set on methods like listenTCP() the wiki page about servers[1]
I've set a pdb.set_trace() on the buildProtocol() method in the file obfsproxy/network/socks.py and the backtrace looks like.
(Pdb) bt /usr/lib/python2.7/runpy.py(162)_run_module_as_main() -> "__main__", fname, loader, pkg_name) /usr/lib/python2.7/runpy.py(72)_run_code() -> exec code in run_globals /usr/lib/python2.7/pdb.py(1338)<module>() -> pdb.main() /usr/lib/python2.7/pdb.py(1314)main() -> pdb._runscript(mainpyfile) /usr/lib/python2.7/pdb.py(1233)_runscript() -> self.run(statement) /usr/lib/python2.7/bdb.py(400)run() -> exec cmd in globals, locals <string>(1)<module>() /home/masterkorp/Public/obfsproxy/bin/obfsproxy(16)<module>() -> run() /home/masterkorp/Public/obfsproxy/obfsproxy/pyobfsproxy.py(207)run() -> pyobfsproxy() /home/masterkorp/Public/obfsproxy/obfsproxy/pyobfsproxy.py(189)pyobfsproxy() -> do_external_mode(args) /home/masterkorp/Public/obfsproxy/obfsproxy/pyobfsproxy.py(108)do_external_mode() -> reactor.run() /usr/lib/python2.7/site-packages/twisted/internet/base.py(1192)run() -> self.mainLoop() /usr/lib/python2.7/site-packages/twisted/internet/base.py(1204)mainLoop() -> self.doIteration(t) /usr/lib/python2.7/site-packages/twisted/internet/epollreactor.py(396)doPoll() -> log.callWithLogger(selectable, _drdw, selectable, fd, event) /usr/lib/python2.7/site-packages/twisted/python/log.py(88)callWithLogger() -> return callWithContext({"system": lp}, func, *args, **kw) /usr/lib/python2.7/site-packages/twisted/python/log.py(73)callWithContext() -> return context.call({ILogContext: newCtx}, func, *args, **kw) /usr/lib/python2.7/site-packages/twisted/python/context.py(118)callWithContext() -> return self.currentContext().callWithContext(ctx, func, *args, **kw) /usr/lib/python2.7/site-packages/twisted/python/context.py(81)callWithContext() -> return func(*args,**kw) /usr/lib/python2.7/site-packages/twisted/internet/posixbase.py(614)_doReadOrWrite() -> why = selectable.doRead() /usr/lib/python2.7/site-packages/twisted/internet/tcp.py(1062)doRead() -> protocol = self.factory.buildProtocol(self._buildAddr(addr)) /home/masterkorp/Public/obfsproxy/obfsproxy/network/socks.py(183)buildProtocol() -> circuit = network.Circuit(self.transport_class())
It goes directly to this method from Twistedm, which confirms that all the SOCKS is set. This creates a new Circuit. And the like the documentation says a Circuit is a pair of connections between and the obfsproxy client and openvpn client (in this case) or obfsproxy server and OpenVPN server.
Then I set a pdb.set_trace() dataReceived() on obfsproxy/network/network.py well because of the name. And a backtrace looks like:
(Pdb) bt /usr/lib/python2.7/runpy.py(162)_run_module_as_main() -> "__main__", fname, loader, pkg_name) /usr/lib/python2.7/runpy.py(72)_run_code() -> exec code in run_globals /usr/lib/python2.7/pdb.py(1338)<module>() -> pdb.main() /usr/lib/python2.7/pdb.py(1314)main() -> pdb._runscript(mainpyfile) /usr/lib/python2.7/pdb.py(1233)_runscript() -> self.run(statement) /usr/lib/python2.7/bdb.py(400)run() -> exec cmd in globals, locals <string>(1)<module>() /home/masterkorp/Public/obfsproxy/bin/obfsproxy(3)<module>() -> import sys, os /home/masterkorp/Public/obfsproxy/obfsproxy/pyobfsproxy.py(205)run() -> pyobfsproxy() /home/masterkorp/Public/obfsproxy/obfsproxy/pyobfsproxy.py(187)pyobfsproxy() -> do_external_mode(args) /home/masterkorp/Public/obfsproxy/obfsproxy/pyobfsproxy.py(106)do_external_mode() -> reactor.run() /usr/lib/python2.7/site-packages/twisted/internet/base.py(1192)run() -> self.mainLoop() /usr/lib/python2.7/site-packages/twisted/internet/base.py(1204)mainLoop() -> self.doIteration(t) /usr/lib/python2.7/site-packages/twisted/internet/epollreactor.py(396)doPoll() -> log.callWithLogger(selectable, _drdw, selectable,fd, event) /usr/lib/python2.7/site-packages/twisted/python/log.py(88)callWithLogger() -> return callWithContext({"system": lp}, func,*args, **kw) /usr/lib/python2.7/site-packages/twisted/python/log.py(73)callWithContext() -> return context.call({ILogContext: newCtx},func, *args, **kw) /usr/lib/python2.7/site-packages/twisted/python/context.py(118)callWithContext() -> return self.currentContext().callWithContext(ctx,func, *args, **kw) /usr/lib/python2.7/site-packages/twisted/python/context.py(81)callWithContext() -> return func(*args,**kw) /usr/lib/python2.7/site-packages/twisted/internet/posixbase.py(614)_doReadOrWrite() -> why = selectable.doRead() /usr/lib/python2.7/site-packages/twisted/internet/tcp.py(214)doRead() -> return self._dataReceived(data) /usr/lib/python2.7/site-packages/twisted/internet/tcp.py(220)_dataReceived() -> rval = self.protocol.dataReceived(data) /home/masterkorp/Public/obfsproxy/obfsproxy/network/socks5.py(163)dataReceived() -> self.processEstablishedData(data) /home/masterkorp/Public/obfsproxy/obfsproxy/network/socks.py(107)processEstablishedData() -> self.circuit.dataReceived(self.buffer, self) /home/masterkorp/Public/obfsproxy/obfsproxy/network/network.py(152)dataReceived() -> if self.closed:
Its not too far away for the previous backtrace
Inside the method I try to see what the heck is going on. (Pdb) print self <obfsproxy.network.network.Circuit instance at 0x7fbbec1c3758>
This is the circuit itself, passing self seems a into the methods seems a print self common thing to do. I verihave no idea why.
(Pdb) print conn <obfsproxy.network.socks.OBFSSOCKSv5Protocol object at 0x7fbbec1b75d0>
Conn is a socks connections. I wonder how can I extract the source IP from it.
(Pdb) print data <obfsproxy.network.buffer.Buffer object at 0x7fbbec1b7510>
data is butter object. Apparently is a First In First Out data container. Not sure where this object was initially created.
I've tried to use the peek method, whiting pdb. (Pdb) print data.peek(n=100) 8£öä¢ÂZ
Passing n=<whatever> always shows that same string. Maybe that is just it, kinda like an initial hanshake ?
Just a few steps foward here. there is a log.debug 2015-01-22 17:25:35,162 [DEBUG] circ_0x7fbbec1c3758: upstream: Received 16bytes.
Ok I think I am on the right path.
Then the receivedUpstream() on obfsproxy/network/transports/scramblesuit/scramblesuit.py
A few steps forward in: if self.protoState == const.ST_CONNECTED:
Where the const.ST_CONNECTED comes from ? printing shows an value of "2" which on this case its equal to self.protoState, that I think it means that obfsproxy client is connected to obfsproxy Server correct me if I am wrong
Then the data send to sendRemote. which obfsucates the data itself and reads from the Buffer object.
In here I could not get info about the SOCKS client Ip address. Wich is weird.
Can you shed some light ?
Thanks in advance?
[1] https://twistedmatrix.com/documents/current/core/howto/servers.html
-- Regards, Alfredo Palhares
-
Alfredo Palhares -
Yawning Angel