These are the latest from my hackerland journey. I feared that either me or the challenge site would go down, and in fact hacker.org is now temporarily unavailable since three days. Maybe it is under a Denial of Services attack of some user who is angry because he can't solve some challenge... Who knows?

Close to the last castle

With 190 challenges solved, I am close to the last castle on the upper left corner of the map. The challenges around it are very hard, none of them is solved, and after there will be a super-hard challenge anyway to protect this castle.

Being a dangermouse it was reasonably easy to solve some quests of the serie "Really Small Mouse". I brought the mouse down to 21 instructions, but there are some hackers who can do it in 20 and less! Cool challenges encountered on the road were 'No full ACK in SEPT' which refreshed modular algebra a bit, 'Maelstrom' to train image recognition, 'HVM cipher' and 'Really Simple Access' to revive my reminescences of cryptanalysis. While solving "Shattered and Shuffled" and "Shredded and Scrambled", I felt cold down my neck, it was the breath of the dark side who strucked me.

I suffered deceiving in "Such Much" and only an answer on the forum brought me back on track. I almost brought down hacker.org by submitting huge numbers calculated with Python! I attacked with brute force 'Soviet Intercept' and I almost locked me out of this challenge; now I have to wait one hour between submitting of answers for this particular challenge, though now I prepared a sort of key which should allow the identification of the spies.

For the cryptographic challenges involving unkown codes, I developed a tool to perform the kappa-test, and another one to perform the chi-test on vigeneres which have substitutions in columns instead of simple caesar ciphers. I wrote them according to this article. I found a good tool to perform first image analysis on Steganographic challenges here made by Caesum, the author of the Challenger's handbook. To perform coprime factorization I definitely recommend MSieve. Other discoveries were Binary Coded Decimals (BCD) and Setun, a Russian computer which was working with ternary balanced circuits :-)

deltasql 1.4.2 and beyond

1.4.2 was a tactical release, to get hands on code which I did not touch since three months and also to advertise deltasql through the Mantra of Open Source Release often, release early. The development of 1.4.3 is struggled by technical accidents (the server I am using to test went down, because the friend of mine who kindly hosts deltasql forgot to pay the bill). Additionally, I face now trouble pushing my code with git to sourceforge. But 1.4.3 will be fresh wind with several minor bugfixes, user preferences and the ability to inform users of new scripts via sendmail.

Finally, to remember myself on how to

Set default operating system in Grub

1. edit DEFAULT variable in /etc/default/grub
2. run sudo grub-update

There are many challenge sites on the net, and one to unify them all.

I decided to focus on hacker.org as there are many high quality challenges involving programming, cryptography and maths, and a bit less of ugly exploits and similar. The site organizes challenges on a map. Harder challenges are on mountains, easier ones on grass. Each time a challenge is solved, it opens the road to new challenges. Castles protect challenges that are on crossroads. Winning against a fortified castle opens new possibilities and new areas to be discovered.

From easy to hard or the other way round

Sometimes i get stuck solving a harder challenge. At first time, it looks like i am wasting my time, but the point is that if by chance i manage to solve it, several other challenges suddenly get much easier.

This was the case with the Labyrinth challenge, which forced me to develop new techniques for the HVM. In the end, I invested more than one month of time on it. Once Labyrinth was solved, I could solve the similar Deluge and the sequel "Flash Flood warmup" in one day. Other challenges like "King Mouse", "Brokenest Keys" and "Mus Minutoides" finally found a solution through new ugly techniques (mainly calls with no returns and what i call the statistical approach, to spoil them a bit).

To sharpen my weapons further, I put Caesum's Challengers handbook in my journeyer's backpack. And here I found a good explanation of the chi and kappa test, so that I added the index of coincidence to the magic spells I am aware of.

It was a great satisfaction to besiege the castle protected by 'Spiral bits'! The path following algorithm did his job well, but the decoding part was a complete nightmare, so that i turned the Lazarus GUI into a complete editor for inserting, deleting and flipping bits. I also had an integrated lens on board and a tool to compare the decoding on slightly different paths (which did not work very well though). And a good bit of luck was also part of solving it!

I then succesfully attacked 'No full ACK in HEX' and 'No full ACK in DEC', learning Knuth arrow notation, the tower of powers, and in particular learning Python's ability of computing with large numbers, which is unrivalled. Maple couldn't do what Python did. Now I know why some numeric people I worked with were so happy with Python :-)

And still, there is work to do. My armies failed several times against 'Blizzard' and 'Descartes'. The 'Captcha' prototype needs further refinement as well...

(current ranking on hacker.org is 188 with 180 challenges solved/on Khan 200 challenges solved and Copernicus badge reached, next goal is one million points)

Mauri is born! Marti now has a brother and we are all very happy to welcome him in our family :-) Might the source be with you, Luke.

In the following, some updates on our lone hackerland journey:

Runaway Robot

Probably some smart people go straight to the correct solution. For me, I walked through all stages of enlightment. I first developed a maze parser in Frepascal, then got a brute force solver in place which could solve up to about maze level 120. The solver started taking long times for each new level (up to 6 hours). I am a bit out of gear right now as we moved house, lost connection to civilization for some time, and still have to reactivate the server room, which will move from pavement to cellar. I remembered that spartacusII server got burned in a similar challenge and decided to upgrade the algorithm as soon as possible. With dynamic programming I restricted a bit the solution space. A first attempt with combinations failed, I tried to debug the way I create combinations, but could not find the mistake. I rewrote the algorithm and got a working solver for levels up to 153. Then again, the solver was taking too much time. Finally, I got a sort of Zen flash and found a way to retrieve the solution, as it would be just popping out from the maze. The new solver could solve any maze in less than a second, and easily moved up to level 513. When it solved level 514 (a level noone has solved), and was waiting the one minute delay to avoid server pounding before the solution is submitted, I got very proud. Too much proud, as the server answered: "Your solution sucked, maximum level for this puzzle is 513". This site is amazing, it does not forget to teach you to be modest :-) The whole story can be seen in this graph as well.

I then decided to attack another puzzle, Mortal Coil. Adapting the parser and getting a brute force solver in place was easy, as Runaway prepared the path. But now I am stuck at level 68. I found papers on the Hamiltionian path problem, which is NP-complete and now need to think on how to divide and conquer it.

Other challenges

On the Hack Virtual Machine, I worked out the meaning of the two commands ^ and v and such knowledge lead to some progress: by analyzing the solution of Execution Style, I finally cracked Hello Small World! Also revision 34 of my Labyrinth program could solve the three mazes, though it required several submissions. In fact, the limit of 40'000 cycles is very tight. The version which solved the challenge could solve the reference maze I was using only in about 53'000 cycles. The Labyrinth program is based on a wall following algorithm published in Nievergelt's book about Algorithms and datatypes.

I found in the depths of Internet an almost complete map of the challenges and it is clear now that to achieve some breakthrough, I need to work out Spiral Bits. The encoding is a bit hard, but by printing out the spiral on a piece of A3 paper, it is possible to work it out with some patience. The pathfinder solver I developed follows the spiral but keeps staying on the side, so I have to think a way to balance it a bit. To debug the breadcrumbs left by the program, I use this magnifying glass.

On Khan Academy, I am close to the Copernicus Badge, but the Atlas Black Hole badge moved again out of range from 250 to 500 challenges...

Some other news

Unrelated to the previous topic, a deltasql patch to branch from existing tags for 1.4.1 is published here.

And do not forget to check the Wikileaks Spy Files!

Let me finally point at a news on the Voyager probes measuring Lyman-alpha radiation (here). Amazing piece of irons, the Voyager space probes!

Win a trip in space!

I once saw a documentary about Burt Rutan's Spaceship One. I was impressed by the engineering skills of Rutan's team and by their audacity: how they tried different rockets for their spaceship (the first one blew up), the ingenious method developed for reentry in atmosphere... How during the flight to win the SpaceX price the main 3D instrument gave up and the pilot simply lead the spaceship in direction where it was more dark :-)

If you are lucky, there is now the possibility to win a suborbital flight with Space Adventures, a company which offers space trips based on the technology developed by Rutan.

If you are a database administrator living in UK, US, Canada or Germany, try Redgate contest DBA in space. You might win a trip to space! I am doing it although I am Swiss and can't win anything just for fun :-)

I noticed the contest, because Redgate is developing database control version tools similar to deltasql.

Monitor a tor node in Wikileaks style

One of the good sources feeding Wikileaks is the tor network. What happens is that lot of intelligence agencies pull sensitive stolen documents via tor, so that they can hide their tracks while stealing documents. Wikileaks volunteers might run tor exit nodes and monitor for unencrypted traffic going through it.

For example, in Gentoo (unfortunately, this time I did not write down all steps, so it is only a sketch):


emerge -av tcpdump # to monitor traffic
emerge -av openntpd # to synchronize the clock with ntpd
emerge -av tor # tor itself
torgencert # generates a certificate for your tor node
ntptime # to adjust your clock


Configure /etc/tor/torrc so that it is sound, plus add the following lines so that the tor node becomes an exit node.
The following lines make your tor node an useful node which will cause you little trouble as all exiting traffic is encrypted a part of irc.


ExitPolicy accept *:22 # ssh
ExitPolicy accept *:465 # smtps (SMTP over SSL)
ExitPolicy accept *:993 # imaps (IMAP over SSL)
ExitPolicy accept *:994 # ircs (IRC over SSL)
ExitPolicy accept *:995 # pop3s (POP3 over SSL)

ExitPolicy accept *:6660-6697 # allow irc ports, very widely
ExitPolicy reject *:* # no other exits allowed


The next two policies make your tor node useful to monitor it for stolen documents. It is best however, not to run it for too long time as you might get complaints, as unfortunately tor is not only used by charitative NGOs...

ExitPolicy accept *:22 # ssh
ExitPolicy accept *:465 # smtps (SMTP over SSL)


The final policy is

ExitPolicy reject *:* # no other exits allowed


Now try to tear up tor and listen for packets:

/etc/init.d/tor start
tcpdump -A | grep -i "WHATYOUAREINTERESTEDIN" > captured .txt


If tor does not start check /var/log/tor/tor.log for hints.

Besides this topic, I found an interesting article on how the Great Chinese Firewall works. But I think that DNS poisoning is only one of the many techniques used.

Cool hacker challenges

On hacker.org I was able to cut the head of the arithmetic snake. And I am proud on how I solved the "Cons Car" challenge with a Freepascal prototype always at the swapping limit of my old andromeda laptop. As side effect, I finally got insights into the LISP language. I am breaking my head on the late didactic crypto challenges and on "Anybody Out There": I tried to model it with two parameters of an oscillating circuit (L and C) and had a look at the hydrogen line, but it did not work out. (rank 210/solved 170)

Not much progress on Khan Academy (videos 187/2619 - solved 194/211)

"Force has no place where there is need of skill." - Herodotus

Kahn's wisdom

Khan Academy published new exercices and this revived my interest in the platform: I solved some more exercices and watched through several videos. While watching "How Earth's tilt causes Seasons" I realised a conceptual mistake in simclimate (released with GPU): I consider the angle for sunlight but not day duration of sunlight... I also watched the series on Milancovitch Precession and Obliquity, Orbit Excentricity wobbling and Perihelion precession. I had the Python introduction, courses about Human Evolution and Earth Formation, and I then decided to start a complete refresh of Linear Algebra, as the course attended at university was taken 13 years ago. And even at work, from time to time, linear algebra is lurking. (Current status: exercices: 176/188, videos: 187/2259)

Journey in Hackerland

The quests are getting definitively more difficult to solve. This month I could solve only eight challenges. Revision 54 of my Brainfuck interpreter written in Superhack language finally conquered the fortified castle. For "Spiral bits" my prototypes aren't good enough yet, so that one complete area of Hackerland remains off range. "Execution Style" and several didactic cipher challenges were solved. Also the Lawsonomy secret is unveiled. But speaking of didactic cipher challenges: brute force works only up to 4 billion keys, which can be tested in roughly a quarter hour of computations assuming printable ASCII characters. If the key space is bigger, this quote is definitely not valid: "If force does not work, you are not using enough force". For "Branches" I painfully worked out a solution, but the number I get is not accepted as solution, meaning that probably I cut off too much code or that I oversimplified the recursion. Who knows. For "Snake Arithmetic", I understood how the nominator is composed, but I still could not telescope the denominator part of the Python equation. For "Anybody Out There" I bought the book "Fourier Transform for pedestrians", but I am still waiting for it as it is out of print and the new edition should come this month. For this alien challenge, I also asked help to an old keen friend of mine who studied physics.

Through my journey, I discovered plenty of tools worth of mention: IntelliJ IDEA, Netbeans and ImageMagick. My favourite pseudorandom generator ISAAC has a companion in form of the RadioGatun hash function.
(Current status: rank 212, quests 165/277)


deltasql 1.4.1

deltasql 1.4.1 is a maintenance release. I discovered a stability problem in deltaclient (for which a dedicated patch of 1.4.0 was done), and tried to minimize the impact of undefined index errors when using strict error_reporting in php.ini. Some unused columns were dropped and the synchronization INSERT statement was simplified. This release really qualifies for the maintenance tag.

My next goal is to try to develop a plugin for the Squirrel SQL client, which is phantastic client which is able to connect to multiple database types (as it is based among other drivers on JDBC ones, a technology I am familiar with). Squirrel SQL is useful at work as well, as there I do not have administrator rights, and I am therefore prevented to install the full Oracle driver.

Linux Magazines

I am a fan of Linux Magazine. I read many articles there, including one presenting System Dynamics tools used in simulations and one about the Hercules Mainframe emulator. And all the cloud computing stuff is more than a hype. Virtualbox is definitely running on my computers. Plenty of images for Virtualbox are here.

Cables

I downloaded Wikileaks cables from Cryptome. For the moment, I decided to split them with split -b 4096k cables.csv cableparts and to search through them with fgrep -i tanzania cableparts* > tanzania.txt. Probably not the best approach, though.