Advogato blog for itamar

9 Apr 2004

Fri, 9 Apr 2004 18:09:55 GMT

Occassionally people will ask Twisted developers why we have a custom serialization format for our remote object protocol, instead of using Python's standard pickle format. Besides interoperability with other languages (e.g. Java), the main reason is security. Unpickling strings from untrusted sources is totally and utterly insecure.

For example, a pickle JP Calderone provided, which runs "touch blah" when unpickled:

>>> import pickle
>>> pickle.loads('c__builtin__\neval\np0\n(S"__import__(\'os\').system(\'touch blah\')"\np1\ntp2\nRp3\n.')
0
>>>
itamar@sheriffpony:~$ ls -l blah
-rw-r--r--    1 itamar   itamar          0 2004-04-09 14:02 blah

8 Apr 2004

Thu, 8 Apr 2004 16:42:31 GMT

Why I lost a job to Open Source software

In the past few months I've been looking for a job, sending out resumes, asking friends if they know of anything. Two weeks ago I got my first job offer, from a company that develops software targeted at a specific vertical business market involving quite a bit of network programming. The work sounded like fun, it was in New York City so I wouldn't have to move across the country and so I accepted. They sent me their intellectual property agreement, and invited me (after I signed, of course) to a company meeting that Friday.

This is where the trouble began. The clause that caused me problems went something like this:

"I agree that I... hereby assign to the Company... any and all inventions... which relate in any way to the actual or anticipated business, research, developments or products of the Company."

Now, I fully support the right of the company to protect its proprietary knowledge, and I would never work on something that would compete with my employer's business. However, the language employed by the agreement above meant I could not really work on Twisted, of which I am one of the core developers. Since Twisted is a networking framework, and networking being "related in some way" to the company's business, the company would own the copyright and I would be unable to contribute to the project.

I thus asked for an exemption, either for any open source projects I wished to work on, or alternatively for a mutually agreed upon list of projects. Since other clauses in the agreement would have given them legal protection if I did something competing or conflicting (not that I would), I felt the company's rights would still be protected.

To some degree the company had no problems with my doing open source development. They use some open source software internally, and will feed patches back to the developers. They did however have a problem with my working on anything that relates to what their software does, the broad interpretation of the agreement. They felt that any skills, knowledge or techniques I learned from working on the company's software might leak into Twisted, thus allowing potential competition to have a head start developing competing software.

From my perspective, I felt that their fears were overstated, since Twisted is much lower-level than their business-specific application, apparently much slower and an already existing general networking platform. The company's product has been in development for four years, by some very smart people. Twisted would only save a small amount of the work necessary to get anywhere near what they have built. Additionally, the reason they wanted to hire me in the first place was due to the experience I had gained working on Twisted (network programming, framework design, protocol design). I believed the ongoing growth in skills I would gain by continuing to work on Twisted would far outweigh any theoretical gain some competing company would get from my Twisted development work. I know of no company using Twisted that is in the same business.

We negotiated back and forth about what pieces of Twisted I might work on, an argument between my emotional investment in code I've written and fear of losing it and their emotional investment in their code and their fear of losing it. Eventually they got back to me with an offer: stop working on Twisted completely, in return for a %20 raise in salary from their initial offer.

Which I then refused. I work on open source software because it allows me to create software I will always be able to use, not leave behind when I switch jobs. Because other people can use it, like it, and occasionally even tell me that. I've learned most of my programming skills from working on Twisted, from working with some of the smartest people I know and getting bug reports, complaints, suggestions and use cases from all of our users.

Certainly I would put my job first, and I would not let my open source development get in the way of my doing my job well. Nor would I work on software that competes with my employer's business. But I am not willing to have my employer determine what I do in my free time.

I'm still looking for a job. Anyone, ideally in NYC or Boston, want to hire me?

12 Jan 2004

Mon, 12 Jan 2004 16:38:19 GMT

Further progress on the Java front. Kaffe OpenVM 1.1.3 now runs all tests (which includes being able to run Jython) for my remote object protocol library TwistedJava . Kinda slow since there is no PowerPC JIT, but I suspect that has to do with the startup process and Jython's importing system, the actual tests are pretty speedy.

Hm, maybe I should rephrase. I didn't make any changes to my code, rather Kaffe has been improving. So now there are two open source Java implementations that are good enough to run my code, which was written for the standard Sun JDK.

27 Dec 2003

Sat, 27 Dec 2003 14:57:06 GMT

IBM seems to finally have a 1.4 JDK for Linux PPC, currently the only full 1.4 JDK/JRE available for PPC. While I dislike using proprietary software, I want to try running Freemind, a free software outliner.

Writing free software for proprietary platforms like Java (unless you make sure to test with gcj or kaffe), and even more so Mac OS X or Windows, is a far cry from writing for free platforms. It will often end up forcing your end users to use proprietary systems so they can run your software, and the underlying APIs you use are not under your control (and may be removed or canceled) until an open source reimplementation comes along.

The history of mTropolis, a multimedia system that competed with Director, is instructive (copied from here):

Originally a private company, mFactory, the creators of mTropolis, were bought by Quark, makers of Quark Xpress. Not long after Quark bought mTropolis, and only weeks before version 2 was due out Quark announced that they were killing mTropolis. Due to a Herculean effort by an extremely dedicated product manager version 2 was eventually released. Currently mTropolis is still used by several hundred die-hards and its capabilities have been increased by the development of third party mods. Unfortunately, mTropolis remains in limbo to this day.

What this doesn't mention is that a group of users tried to buy mTropolis from Quark, and Quark refused to sell it.

There a number of ways to deal with this:

For Java/C#, test and make sure your code runs on free platforms (kaffe, gcj, mono).
Use a cross-platform open source platform (e.g. Python) or write portable C or C++, implement a portable backend, and write OS specific bindings integrating with each OS. Abiword and Mozilla are good examples of this.
Only use libraries that run on multiple OSes (e.g. GTK2, which runs on Windows and X11).

Of course, for platform-specific coding you're pretty much stuck if the platform is proprietary.

14 Dec 2003

Sun, 14 Dec 2003 19:50:56 GMT

Hooray! For the first time ever, my Java open source project (remote object protocol compatible with Twisted's) runs, or at least passes tests, on a totally free platform, gcj 3.3.3 pre0 in Debian.

Also playing with making native jython:

gcj -lreadline -fjni --main=org.python.util.jython /usr/share/java/servlet-2.2.jar \
    /usr/share/java/libreadline-java.jar /usr/share/java/jython.jar -o jython

seemed to do the trick on Debian. Testing it with -O2 and -maltivec to see if it's any faster.

26 Nov 2003

Wed, 26 Nov 2003 23:15:34 GMT

My first article in a series on protocol design went up. In between contract work, free software (Python VoIP phone, woo), and other stuff, I am also looking for a full time job - read my resume.

15 Jun 2003

Sun, 15 Jun 2003 15:12:34 GMT

Just got back from Usenix 2003. The paper glyph and I wrote is available: Networking For The Rest Of Us, as are the slides. I'm still not sure we got the point across well, but the points I realized I wanted to add (configuration as scripting, ability to encode common configurations in a deployment generator, unified authentication backends) wouldn't have fit in a 30 minute talk anyway. Some of this arose from the time Glyph spent presenting Twisted to various sysadmins, and our plans to reimplement the authentication infrastructure.

The conference itself, while small, was extremely interesting and educational.

5 Mar 2003

Wed, 5 Mar 2003 19:44:05 GMT

Our power grows!

Mystery Robot is written with Twisted. Mystery Robot is an implementation of Nambers, a way of making IP addresses easy to remember.

14 Jan 2003

Tue, 14 Jan 2003 18:37:47 GMT

Entrance of the Central Scrutinizer, from Frank Zappa's "Joe's Garage" album:

This is the CENTRAL SCRUTINIZER...it is my responsibility to enforce all the laws that haven't been passed yet. It is also my responsibility to alert each and every one of you to the potential consequences of various ordinary everyday activities you might be performing which could eventually lead to *The Death Penalty* (or affect your parents' credit rating). Our criminal institutions are full of little creeps like you who do wrong things...and many of them were driven to these crimes by a horrible force called MUSIC!
Our studies have shown that this horrible force is so dangerous to society at large that laws are being drawn up at this very moment to stop it forever! Cruel and inhuman punishments are being carefully described in tiny paragraphs so they won't conflict with the Constitution (which, itself, is being modified in order to accommodate THE FUTURE).

12 Jan 2003

Sun, 12 Jan 2003 17:17:17 GMT

Hack of the day - Chopshop. Rips Shoutcast radio MP3 streams into separate MP3 files on your disk. It requires Twisted CVS (or 1.0.2 once released).

There are already existing programs that do this (Streamripper sounds nice, gotta steal that mp3 silence trick), but Chopshop is probably the shortest one - I wrote about 230 lines of code, half of it added to Twisted as generic Shoutcast protocol support.