This article is advice - not legal advice: you are hereby advised to get legal advice on the following advice, and if you screw up, don't even THINK about contacting me, blaming me etc etc TOTAL disclaimer, it was Your Decision etc etc - on how to do Reverse Engineering.
In light of the recent project announcements and others, mono [good luck guys!] you may wish to follow these guidelines.
This is from experience of having done network-reverse-engineering of Samba TNG for four years. The bias is therefore specifically geared towards dealing with Microsoft, but it applies to all big companies.
0) ASK FOR THE SPECS! i know this sounds weird, but it's a very important step. most of the reverse-engineering laws (esp. the EU ones) require that you be able to PROVE that 'the information was not available by any other means'. and only THEN can you do 'interoperability' reverse-engineering.
so, you must ask. send an email to Mr Bill Gates. do a follow-up telephone call [no, i'm actually not joking, although it does sound like a funny thing to do: hi, mr gates, please give us all of your secrets we want to do as well as you]. send out letters by courier to 1 microsoft way, by email to public forums where the people developing the software are known to hang out / respond.
wait for six weeks. if you get no response, assume that [GET LEGAL ADVICE!] you've done enough, and you can proceed.
1) Buy licenses for all - ALL software - you use. Companies have been successfully sued not for reverse-engineering but for not having licenses for the software (or in some cases hardware) they were reverse-engineering.
2) Find out what the legal attitude towards reverse-engineering is in the country you intend to work in. if it's not legal there, find some employees in another country, and find out what the legal aspects are w.r.t. importing that software.
3) Find out the legality of, and follow this:
a) employ one person to do the reverse-engineering, to Hack Up Some Code.
b) employ a second person who can talk to a) who acts as a firewall, generating a specification which is published PUBLICLY and then give it to..
c) employ a third person who has never heard of, met, seen, breathed in the same room as a), to implement b's spec.
repeat until c completes the implementation. at least one large company reversed-engineered the specification for and a clean-room implementation of X-Windows this way, where ab and c were teams not people.
4) follow the interoperability / rev-eng guidelines. in the EU, as mentioned above, the laws are that you can only rev-eng for interoperability only, where info is not available by any other means [i assume that this excludes microsoft's NT source code license - available at $1m per year - because it is a read-only license...]
in the U.S. okay. the u.s. is a bitch. i cannot BELIEVE how the DCMA xxxx got through. anyway, that's been shot to pieces, discussed to death etc etc and of course, they have more money than us, so they win [oo sarky today, aren't i?]
... however: read it carefully [the DCMA]. you will note that there are some exceptions. namely, security companies and anti-virus companies. soo....
5) go work for, set up, or create, a security company. as part of the research of that company, do the following:
a) produce security reviews and audits of the program being reverse-engineered.
b) BE RESPONSIBLE. THIS IS IMPORTANT, LISTEN UP. get onto the GOOD side of the company whose product you are reverse-engineering. in microsoft's case, that's the secure@microsoft.com people. DO NOT UNDERESTIMATE HOW IMPORTANT THIS IS.
send them reports, repro code, DO NOT BLAB TO THE PRESS. follow the guidelines for bug-fixing / security-bug-fixing, namely:
- report to vendor - *full* disclosure including repro code. do NOT blab to public.
- send follow-up 2 weeks later if no response, saying, 'if no response, we must go public to get your attention, in 6 weeks time'.
- in 8 weeks time, make a RESPONSIBLE report not outlining how to create a virus or a public death-sentence program. you don't want to be known as the people who helped destroy the internet....
if at ANY time you hear from the vendor:
- do not blab to the public. work WITH them to produce a security bulletin, to do a simultaneous announcement of your report and theirs. GIVE THEM TIME TO FIX THE PROBLEM RESPONSIBLY AND PROPERLY. you don't want them to get irritated at having to pull out all the stops, come out with another security-fix later 'cos you didn't give them time enough, it will only bite you later [see Last Resort, below].
c) produce a full specification (this can be your 3a, above: and heck, you even get the good-will of the vendor, too!) ... AND PUBLISH IT. do it as a book (e.g ISBN 1578701503 - DCE/RPC over SMB: Samba and Windows NT Domain Internals, shameless-plug, shameless-plug).
6) get life insurance. oh, you think i'm kidding?? then why did four separate individuals advise me to be ever so, ever so careful, when i started in august 97 on the nt domains protocol?. why have three separate individuals sent me parts of their work asking that it not be disclosed that they wrote it? because they were afraid of the 800lb gorilla and what it might do to them, their careers etc. think about it. you're taking on a multi-million dollar company who can't _buy_ you - you're open source, for pity's sake. their alternatives involve patents and legal recourse (last resort: it would be _really_ bad press), employing the developers, or getting rid of them by other means.
7) WRT crypto:
a) if it's passwords / authentication, then [GET LEGAL ADVICE] it's acceptable to produce NON-GENERIC code that does not have a publicly accessible interface to be able to encrypt data.
b) send a copy of your code to the U.S. Dept responsible for exports. apparently, this is now sufficient [GET LEGAL ADVICE!] to export the code from the U.S.
c) if you have non-U.S. employees, GET A BXPA LICENSE. the contents of your non-U.S. employees heads are then 'countries-in-their-own-right', to which you can 'export' your valuable, and of course, U.S. developed, military-grade munitions. these employees then become subject to the same laws as U.S. citizens. they may talk to other U.S. citizens but not to anyone else.
this gets you out of the ridiculous situation where your non-U.S. employees can email people in the company, but they can't respond. why? because _you_, the non-U.S. employee, have developed some munition that is imported to the U.S. [no laws broken here, yet]. but, if those employees respond, they are EXPORTING those munitions [to your head, from whence it came...] and THAT is illegal. so don't do it.
either that, or just post absolutely bxxxxy everything onto public forums. DO NOT engage in private conversations with individuals off-list about crypto. then it's covered [GET LEGAL ADVICE] by free speech laws... hmm....
8) have all intellectual property owned by an off-shore company [this is not legal in the U.S].
so, you got onto their good side. you helped them improve the quality of their code, by helping them fix important bugs. you develop a good relationship with some of the developers, even: and they let you in on a few secrets - that some of them actually *support* what you are doing, because of the massive benefits to *their* product. ... and, get this: you know how lonely it gets, working on this stuff? it's so advanced, there are very few people out there who understand it. well, guess what? the people who originally developed it feel the same way. and despite the confidentiality clauses, actually having someone to talk to who understands and respects what you have achieved (working out what they did _and_ doing a good job), even if they _are_ a little scared of you :) is worth a thousand confidentiality clauses. a breath of fresh air is always welcome [except if it's bitterly cold air... ]
The Last Resort
... then, the Marketing Machine got Mean, decided to Take You Down. The Legal Machine Gets Involved. you end up Underground. well, if that happens, you know enough to produce some really nasty viruses. carrot didn't work, Big Stick Time...
Reverse engineering is perfectly legal in countries like France, for the means of interoperability (again get legal advice to be sure). So unless your are doing some other felony like software piracy or copyright violation, you are almost safe from being sued.
Beside cryptography which has specific rules, I don't remember any export restriction for such work.
This is probably the same situation in other European countries.
darn, i almost forgot network reverse engineering.
network reverse engineering is [GET LEGAL ADVICE] legal. why? because it's publicly accessible data. there's nothing to stop you duplicating or parroting data you can see on a network [yet. GET LEGAL ADVICE].
so what approach should be taken to do network-rev-eng? well, this is important: develop a client and a server at the same time.
why? because you can use one to bootstrap the other. it goes like this. Real means the one you wish to network-rev-eng; stub means the one you are developing:
- make a connection attempt from a Real Client to your Stub server. record the request into a file.
- plug the recorded request into your Stub Client. replay it to your Real server. the Real server will respond [hopefully! this isn't actually guaranteed, see below!] to the Stub Client. record the response into a file.
- make the same connection attempt from a Real Client to your Stub server. when your Stub server receives the request, playback the recorded response. [hopefully!] the Real Client will buy that and it will move to the *second* request, and repeat from above.
CAVEATS:
- you need to examine the data being recorded. network traces, which show really obvious things like names of servers, names of clients, names of users, and strings in general, you soon get used to 'seeing' those, and it's pretty obvious, *duur* well if there's a client name in the request, *duur* i have to send *my* client name back.
so, to help you do this: make sure that Real client and Stub client, and Real server and Stub server have names *of the same length* :) :) you can then in-line search/replace the strings of the right names :) :)
w.r.t. strings, look especially for string-lengths. w.r.t variable-data-blocks, look for data-lengths. you *can't* do network requests in structure-packing *without* having data / string lengths (maybe strings you can null-terminate) except over UDP of course. and of course except having escape-sequences like you do for xml, html, etc., you know the drill.
- checksums, digital signatures and crypto in general tend to screw up the above process. at that point, you need to follow advice laid down in this article [GET DAMN LEGAL ADVICE]
when you have encrypted data, then assuming that you have managed to crack the ecryption, the above process becomes REALLY important: staring at network traces is useless.
other advice:
- stare at network traces. i mean, stare. make several. compare them. make more doing the same thing, compare them. stare at them. network traces are your friend. it _will_ sink in - eventually. like, after two months.
i use vi at the console, so i have two text-dumps (one on each window) and i flick backwards and forwards, then page-down, alt-f1, alt-f2, alt-f1, alt-f2. page down alt-f1 page down alt-f2 alt-f1 alt-t2.
differences show up *immediately* - remember, eyes are good at detecting *changes*. make it easy to detect changes.
if using Netmon, use Netmon v1 because netmon v2's display capabilities are xxxxed up [it uses internet destroyer to do the displaying]. if you do Alt-W[indow] 1, Alt-W[indow] 2, repeat, you find that the two windows slowly migrate one pixel down at a time!!!! _not_ very helpful :)
- make only one small change in the Client / Server setup and compare the sniff results. e.g. change the name of the client (or more specifically, change the _length_ of the name of the client).
- develop an automated decoder tool. plug the above-recorded requests and responses into it.
pleeeeaaase develop the automated decoder tool into a real IDL compiler. TNG has 50,000 lines of *HAND GENERATED* structure-packing / unpacking code. it's very well structured code, but.... never mind, i just don't recommend anyone doing it, that's all :)
get specs where you can on the wire-format. write or find an IDL compiler. please. you will feel a lot better for doing so :)
anyway, if i think of anything more, i'll post it here, later.
a bold missed out there, whoops!
yes, France is part of the EU, so is covered by EU law regarding reverse engineering for 'interoperability purposes' only.
French legislation fully enables reverse engineering. This is the reason of the sybiline sentence at the end of Microsoft licence software licence (for instance), precising that the reverse engineering restrictions only apply if they are legal in the concerned country.
New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.
Keep up with the latest Advogato features by reading the Advogato status blog.
If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!
del.icio.us
Digg
Google bookmark
reddit
Simpy
StumbleUpon
Furl
Newsvine
Technorati
Tailrank