Hello, As Daniel suggested, I'm writing this email to keep you updated on my Google Summer of Code Project - APAF.
Following the hackathon of the 18-21 May, we reached a working build environment, and a primitive administration page. Below follows a brief update about what is the current status of the software:
Development: I've defined what a service really is in terms of implementation, and approached the administration service (the "panel") in such a way that any future application may hide or completely override it. This shall give not only a comfortable user experience but also elasticity to the software.
Testing: unittests cover most of the software, but even though APAF is still in an embrional state, I would be glad anyway to receive some feedbacks from you, about which OS the apaf was tested on and any bug you were able to find.
Documentation: currently is disorganized, and splitted between sphynx and github wiki. But, the data is there, and I promise that on the next days I will merge/rewrite the two :) Anyway, the code is highly documented, in my opinion the generated sphynx documentation is acceptable.
Deployment: I kept working on the build system, which at the current revision works on windows, osx and as vanilla python .egg., but there are still some issues - for example, the automation of .deb build and blobbing external tools on windows.
In case you need a quick link to the project, https://github.com/mmaker/apaf
Follows the second report.
Still working on packaging, lots of improvements done on the osx side, where thanks to mogui (m4rduk) we reached an apaf application bar. Packaging on debian works (see branch:debian), but I hope to discuss possible errors with weasel in a few days. I've also installed python and twisted on my android phone, but psutils, a dependency of txtorcon, is a wall for porting apaf into android. It would be awesome if the apaf project could run also on a mobile device: smartphones nowadays are really cheap, small and can suit perfectly the need of a small domestic web server.
Server-side: following hellais' suggestion, I've added cyclone as dependency and started defining a json api, leaving the interface, at least for now, unchanged. In fact, as discussed on irc, we cannot simply make a js interface, without support old, legacy browsers. Right now the api simply consists in: get/edit configuration and get services informations (name, port, hiddenservice, etc.).
Unit testing: none.
"=?UTF-8?Q?Michele_Orr=C3=B9?=" maker.py@gmail.com writes:
I've also installed python and twisted on my android phone, but psutils, a dependency of txtorcon, is a wall for porting apaf into android.
That used to be an optional dependency, so I will put that code back in (actually, probably just take out the dependency for psutils and report PIDs instead). It's really just "nice to have" anyway and users who want that could of course easily do it themselves.
thanks,
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
meejah meejah@meejah.ca writes:
That used to be an optional dependency, so I will put that code back in (actually, probably just take out the dependency for psutils and report PIDs instead). It's really just "nice to have" anyway and users who want that could of course easily do it themselves.
I've just pushed version 0.5 which removes psutil dependencies.
Thanks,
- -- mike
On 6/21/12 7:28 AM, meejah wrote:
meejah meejah@meejah.ca writes:
That used to be an optional dependency, so I will put that code back in (actually, probably just take out the dependency for psutils and report PIDs instead). It's really just "nice to have" anyway and users who want that could of course easily do it themselves.
I've just pushed version 0.5 which removes psutil dependencies.
Which maybe the relationship between APAF and Guardian Project's ORLib/OrBot?
I mean, we should not "reinvent the wheel", however APAF is going to provide Python developers an easy way to build desktop/server applications.
This is not something for "Android application" in the common understanding/way (java/c), focusing on Python apps.
-naif
I tried to run APAF on android during the last days. I am updating this toping with what I've discovered so far.
2012/6/21 Fabio Pietrosanti (naif) <lists lists@infosecurity.ch@lists@infosecurity.ch infosecurity.ch lists@infosecurity.ch>:
On 6/21/12 7:28 AM, meejah wrote:
meejah <meejah meejah@meejah.ca@ meejah@meejah.cameejah.cameejah@meejah.ca>
writes:
That used to be an optional dependency, so I will put that code back in (actually, probably just take out the dependency for psutils and report PIDs instead). It's really just "nice to have" anyway and users who want that could of course easily do it themselves.
I've just pushed version 0.5 which removes psutil dependencies.
After removing psutil, txtorcon was successfully imported from python.
But I see another issue, not strictly related to txtorcon but to the standard library's temp module. Here a screenshot.
http://imageshack.us/f/96/schermata2008245614120a.png/
Apparently what happens is that the tempfile module does not successfully create the temporary directory for hosting the configuration.
Note: I am using the same tor binary of Orbot, placed in data/data/org.torproject.android/app_bin/ as you can see from https://github.com/mmaker/APAF/blob/master/scripts/python.sh
Which maybe the relationship between APAF and Guardian Project's ORLib/OrBot?
I mean, we should not "reinvent the wheel", however APAF is going to provide Python developers an easy way to build desktop/server
applications.
I think we can discuss this on #tor-dev.
--
ù
Sorry for being so late, but as anticipated on the irc channel, I spent most of the last week as talk manager at Europython. During, the free time I've had the occasion to meet lots of developers of the tor community joining the tordev meeting before and during the hackfest. Concerning the project, I've organized some new cool features for services, such as the configuration class which should help the customization. Backend apis are almost finished - authentication, tor controlport, panel configuration- and covered with unit tests. There is a new dependency, pyCrypto, used for generating safely random ports on which the service should listen to on each run; I think it will be useful also for future utilities exposed from the apaf itself, as toolkit for services. I hope to merge those changes with the platform-specific modules, so that after midterm evaluations I can work on the frontend side. Anybody wanting to help with graphics?
Apologizing, Il giorno 21/giu/2012 07:41, "meejah" meejah@meejah.ca ha scritto:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
meejah meejah@meejah.ca writes:
That used to be an optional dependency, so I will put that code back in (actually, probably just take out the dependency for psutils and report PIDs instead). It's really just "nice to have" anyway and users who want that could of course easily do it themselves.
I've just pushed version 0.5 which removes psutil dependencies.
Thanks,
mike -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux)
iQEcBAEBAgAGBQJP4rGjAAoJEMJgKAMSgGmnWNUH/iyuxOC8jElse8Bb/74id6mM kRvg6QB0BhZEv4XBsXoQJAZ292CvvUqNdlnyUDibeksFv4DGfserHOKV/U41Y28b zEndr9WE8fgmRPHWOBJpB5GPSBqn/GcE8Ohmc+PskpvK6UpS8vzF+OwpH8mU31Vb E1kEW+9NL44Jn7mXB9Z2A7nQJyhcy+LKYTUr0xg1dEoqLx0m72nuFgLbkTjvI1yU C+uhP3fp3Gnm1C2jqq9DL2bJLvOahNNou9n1T8SByz1jFClG8ThyJEt0yZyGdRHa kcN2QPRgWk4fUcZOQ4nRR9WEmyVDHc0mGpGX6FX6uT3Wj4FtX8KPbJ5vWgvbVKw= =BMR2 -----END PGP SIGNATURE----- _______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
On 7/8/12 2:19 PM, Michele Orrù wrote:
Sorry for being so late, but as anticipated on the irc channel, I spent most of the last week as talk manager at Europython. During, the free time I've had the occasion to meet lots of developers of the tor community joining the tordev meeting before and during the hackfest. Concerning the project, I've organized some new cool features for services, such as the configuration class which should help the customization. Backend apis are almost finished - authentication, tor controlport, panel configuration- and covered with unit tests. There is a new dependency, pyCrypto, used for generating safely random ports on which the service should listen to on each run; I think it will be useful also for future utilities exposed from the apaf itself, as toolkit for services. I hope to merge those changes with the platform-specific modules, so that after midterm evaluations I can work on the frontend side. Anybody wanting to help with graphics?
What are we missing as graphics elements?
Making a list of the ones for which ideas had come out during time, which are made / need to be made / need to be documented?
- Application Icon Windows / OSX:
- Dock Icon / Menu OSX:
- System Tray / Menu Windows:
- Splash Screen Windows/OSX:
- DMG Packaging Finder/Background Icon OSX:
- Embedded Browser for App UI (Windows / OSX):
-naif
What are we missing as graphics elements?
Making a list of the ones for which ideas had come out during time, which are made / need to be made / need to be documented?
Application Icon Windows / OSX:
Dock Icon / Menu OSX:
System Tray / Menu Windows:
Splash Screen Windows/OSX:
DMG Packaging Finder/Background Icon OSX:
Embedded Browser for App UI (Windows / OSX):
Well, right now the most important graphic elements are system tray icons, bot for windows and osx. Since I will develop the frontend side of the apaf ater midterm evaluation, I think that is fine with these now. Otherwise, may I take vidalia's ones? At least until something comes up :)
On 7/11/12 8:26 PM, Michele Orrù wrote:
Well, right now the most important graphic elements are system tray icons, bot for windows and osx.
For icons it will be very important also to provide a clear documentation on which format must be used and how to process an existing image to convert that to create the overall sets of icons required (the one of the application, the one for windows system tray, the one for dock, etc) and/or provide a facility that you provide a single image and it will automatically convert to the proper format for all the elements on all the platforms.
Since I will develop the frontend side of the apaf ater midterm evaluation, I think that is fine with these now.
If we can also implement Splash Screen, with it's own progress bar for Startup progress, it would complete the most relevant UI elements "directly perceived" by the end-user, at installation time and at each start.
The "progress bar" below the "splash screen" maybe also very useful to let end-user report errors/diagnose issues with the application startup?
Otherwise, may I take vidalia's ones? At least until something comes up :)
May we ask Mogui to implement OSX specific stuff that he like a lot ?: - DMG packaging Finger/Background - Embedded Browser for OSX (Hellais committed some experimental code?)
-naif
Here are some updates about the APAF project.
The only thing left to do concerning APAF's REST APIs is to wrap and secure tor control port calls for providing the user's current status of anonymity.
Builds for debian are ok, both when generating the .deb and when installing with pip. OSx has its own .app, system tray icon, and the embedded browser which acts just like a simple application window. Window still has a non-working system tray (there's something wrong between the win32 gui thread and twitsted reactor), but I have a single executable, which eventually spits out the datadir/ directory by itself (see apaf.blobber).
I've performed some cleanup, and reorganized each platform's main in such a way that the relative module is dynamically imported.
During the next few days, I am going to work on the frontend part, and fix builds for windows.
Follows the report of the last 10 days.
Coding. hellais warned me about a possible xsrf vulnerability in the APAF's panel, which should be fixed with 629828d . I have also successfully implemented a working django blog engine - ZinniaBlog - on the top of apaf.
Documentation. Discussing with my mentor, it came out the need to use both sphinx and a wiki. The first one, for APIs documentation - that are not going to change often, and should be more or less stable; the second one, for tutorials and external services - more probable to change frequently, and with a more interactive support from a possible community. So, I've updated the documentation also with latest code changes, and put it on readthedocs: http://apaf.readthedocs.org/en/latest/index.html
Some random cleanup - GUI modules have been moved to a new package apaf.ui, html templates split from static/ and moved to a new templates/ directory, etc.
Frontend. Re-started the development of panel pages. using cyclone's templates. Basically, we plan to have both a javascript application taking into account tor network's delays, and a plain html for browsers having js disabled.
Unfortunately, during the last week of this session I didn't managed well the communication with hellais, because he was in a different timezone and busy with the toorcamp, and our timetables didn't really matched.
Frontend. There is a basic html version - I should say incomplete; the javascript interface, instead, was kept commented and unused, waiting for a well-defined implementation to be discussed with my mentor. Backend. Session cookies, configuration file for each service, fixed xsrf vulnerability. User Interface: I tried to create a system tray icon also for ubuntu, using pyGTK, but it's still incomplete. Mobile: finally, I got the APAF to run on my android phone.
I've also installed python and twisted on my android phone, but psutils, a dependency of txtorcon, is a wall for porting apaf into android.
Hi Michele. It looks like txtorcon is trying to make psutil an optional dependency...
txtorcon/util.py 31 try: 32 import psutil 33 process_factory = psutil.Process 34 except ImportError: 35 process_factory = int ... 82 def process_from_address(addr, port, torstate=None): ... 91 If psutil isn't installed, the PIDs are returned instead of 92 psutil.Process instances.
Though torstate.py doesn't have this sort of fallback...
txtorcon/torstate.py 2 import psutil ... 292 def guess_tor_pid(self, *args): 293 if self.protocol.is_owned: 294 self.tor_pid = self.protocol.is_owned 295 296 else: 297 self.tor_pid = 0 298 try: 299 procs = filter(lambda x: x.name.startswith(self.tor_binary), 300 psutil.get_process_list()) 301 if len(procs) == 1: 302 self.tor_pid = procs[0].pid 303 except psutil.AccessDenied: 304 pass
Considering that the method's name is "guess_tor_pid" it sounds like it should be best-effort, and fail gracefully if psutil is unavailable. It might be worth asking if this is a bug.
Personally I decided to write my own modules for this functionality [1] (including some improvements based on psutil [2]) because a C module dependency didn't feel worth this functionality - especially since pid lookup is a one-time thing, and doesn't need to be blazingly fast.
Cheers! -Damian
[1] https://gitweb.torproject.org/stem.git/blob/HEAD:/stem/util/system.py [2] https://gitweb.torproject.org/stem.git/blob/HEAD:/stem/util/proc.py
On 6/20/12 5:54 PM, Damian Johnson wrote:
Personally I decided to write my own modules for this functionality [1] (including some improvements based on psutil [2]) because a C module dependency didn't feel worth this functionality - especially since pid lookup is a one-time thing, and doesn't need to be blazingly fast.
Eh... if only tor was also available for use "as software library" (Like silvertunnel & OrLib), we would a lot of less "system integration complexity"!
There's a lot of effort in many project on "Tor integration hacks" to use it for application integration requirements.
Imho Tor should before or later really provide clear and standard guidance and path for integration in 3rd party applications:
- API to build tor within an application - API to do basic operations on Tor - Generalized Filesystem I/O (to keep all data, from configuration to hidden services up to descriptors in memory and/or in a database)
So that "anonymity" feature can be plugged in to existing applications, like it is possible to plug SSL or OTR in a Chat client.
-naif
Eh... if only tor was also available for use "as software library" (Like silvertunnel & OrLib), we would a lot of less "system integration complexity"!
Actually, tor *does* provide the pid. I added it a year or two back because getting it via psutil, proc, or system utilities all had drawbacks... https://gitweb.torproject.org/torspec.git/blob/HEAD:/control-spec.txt#l646
The only reason that stem still has pid lookup utilities is so that we can get the pid *before* connecting to tor, as a workaround for a longstanding tor bug... https://trac.torproject.org/4881
... this is now fixed in 0.2.3.11, but before that TBB provided a relative path for its authentication cookie making it a big pita for controllers to attach. I needed the pid in order to look up the cwd, so I could expand the cookie path.
On first glance txtorcon seems to be using psutil as a fallback for when 'GETINFO process/pid' is unavailable.
That said, there's lots of other useful information that it would be nice for the control socket to provide (cpu/memory usage and connection information are high on my wish list).
Cheers! -Damian
-
Damian Johnson -
Fabio Pietrosanti (naif) -
meejah -
Michele Orrù