Older blog entries for dangermaus (starting at number 46)

14 Feb 2011 (updated 14 Feb 2011 at 22:16 UTC) »

One of the pleasures of living in a small village in the mountains is to light the fire in the evening. Me and Marti, we like to watch flames and we straighten arms and say 'hhhhh' when we finally sense heat. Virgi is also happy to see that we managed to put fire only in the oven and not to the entire house.

Khan Academy

Salman Khan is kind of modern Prometheus who stole fire from Zeus and brought to humanity. This time, instead of a torch he brought youtube videos with him, where he explains on a blackboard calculus, algebra, analysis, physics, biology and history from the simplest concepts to the advanced topics. The videos are for free and distributed in the best 'Open' manner. You can download 30 giga of videos if you like, or watch them on youtube. I personally am working through the 'Laplace Transform" videos, stuff which I never learnt before. I think the videos (and examples to work through) are very useful both to students but also for people who already attended university and would like to fill some leaks in their knowledge.

To enter Khan Academy, click here, pick out some video, I am sure you'll enjoy the relaxed style used by Khan to teach about anything, even the most advanced topics. To read more on wikipedia about it, click here.

* I was recovered three days at the hospital to cut off the cystifellea as I had too much fast food when I was student. They should put on any fast food the photos of my cystifellea and the stones they found in it to better advertise their meals. WhenI was waiting for operation, I "repaired" the laptop of my room's neighbour by simply taking off the battery so that it could restart. He called me a "PC doctor" :-) I think it is easier to repair computers than humans. In fact they do not cry if you do something wrong, nor they need anaestesia. Anyway, this is the introduction to the next paragraph :-)

Heart Transplantation for Andromeda

The screen on my laptop "andromeda" broke about three months ago after 9 years of duty. It is an XE3 HP laptop 1GHz, 512 MB RAM, 40 GB of RAM, on which I developed most of my Open Source projects. It was one of the first laptops sold with Windows XP, on which I installed first Red Hat 8.0, Suse 9,9.2 and 9.3, Debian 5.0 and ended with Gentoo. I also used the laptop during my study in Zürich, computed stuff with Maple for physics and maths courses, wrote "Hubble in a Bottle" for the zBox supercomputer, deltasql and GPU. . By chance I found an older XE3 laptop with 900 MHz, 256 MB RAM, 30 GB hard disk and decided to execute the operation "make one out of two", which works with basically any device of the same or similar kind (my uncle did it with 2 BMWs). The main advantage is that no matter how you work you end up with lots of spare parts and the disadvantage is you need space to store them.

There were two possibilites: 1. take the screen of the older laptop and put into the newer, or 2. take memory and disk from the newer to the older. I did it in sequence, as the newer laptop is faster (although the plastics has so many rips), but when I exchanged the screen, the laptop mainboard answered with a beeeep which basically was the official last breath of the newer laptop.

I then chose the second option, not really a choice as the newer laptop died. So that now I am writing this blog entry on the older laptop, which was by the way almost never used, plastic intact and shiny. It has also a robust appearance. I hope to keep this one for another ten years :-)

The point of developing software on old things is that if you get it working reasonably on it, you know that on newer hardware it will fly ! Also it remembers you that the heuristics you choose is way more important than the hardware you are on top.

Fractal dimension of Internet

* This is left as an exercise for the reader:

Write a bash script that rolls 4 dices with 255 faces (quite a big dice!) and concatenate them with a dot. Scan the obtained random IP number to verify if there is something behind it. If you find very weak computers with IP numbers starting from 127.x.y.z try to break into them (when you realize that you are attacking your own 'localhost', you can leave this part of the exercise). From the ratio of computers which answer compute how many computers are attached to the Internet. Call this number N.

Next modify the bash script so that instead of scanning the IP number, it traceroutes the random IP number, so that you get the distance from your computer to the other computer (in 'hops'). Compute an average distance and call it 'l'.

Ok now take Mandelbrot formula N ~ l^d/d and solve numerically for d. d represents then the 'Fractal dimension' of Internet. Well, according to Wikipedia, the Internet is not fractal, only the WWW is :-(

Feynman Lectures on Physics

I bought the books three years ago, and now finally found a bit time to read among them. Some things which I understand a bit more are: how Maxwell completed Faraday law when applying it to the condensator; how Einstein looked at the problem of a moving charge feeling a magnetic force if moving in proximity of a rod with electric current: for reasons of symmetry, if the rod is moving with current and the charge stays, the charge should still feel a force, but there isn't any force in classical physics which does it. Einstein then put an electric force and explained its existence with a relativistic argument (- charge are travelling, plus charges are at rest, - charges are travelling fast, therefore there is contraction of length, which implies an overall positive charge of the rod which generates an electric field which attracts the charge). The quantization of spin discovered by the Stern-Gerlach experiment and how spin states relate to linear algebra in quantum mechanics, how Schrödinger wave equation explains the valence model, how the heat equation works, Green and Stokes theorems applied to Maxwell equations...

* and I should rewrite the heat equation in simclimate as well...

GPU II preview for developers

* This is taken from the project news:

GPU is currently undergoing its 4th rewrite, codenamed freedom (3th was named solaris which reached version 0.963, 2th polaris, 1th unnamed). The preview package has version 0.5.0. This package is solely intended for developers as there is still no client working, just a bunch of unit tests.

The package should sketch the whole idea of the next generation GPU. Some people might be disappointed that GPU moves away from peer-to-peer to become a client-server architecture, but we believe this is the only drawback. The highlights of the coming release (probably 6 months from now) are:

- client/server architecture should allow scaling of nodes up to several thousands of clients

- client is in Freepascal and can be compiled on Windows, Linux and Mac OSX

- Server is in PHP/mySQL and runs everywhere where LAMP stack runs

- All software used to develop GPU II is Open Source and available freely

- The plugin architecture of GPU I is retained, although the plugin layer has been rewritten (available in preview) and plugins need to be ported as well

- Communication is entirely over HTTP, workunits can be downloaded, and resulting workunits can be uploaded via HTTP back to the server (available on GPU preview package)

- HTTP makes GPU transparent to most proxies, so that clients can run also in companies and not only at home

- GPU II retains features like channels (to run whiteboard, chat and sensor networks), job management, clustering services, all rewritten in light of the experiences done with GPU I (available on GPU preview package)

- GPU II will stay a multipurpose distributed computing architecture, so that any client and any server will support any computation (unlike the BOINC architecture)

- Each server should be able to support 40-100 clients according to some initial computations, the network should scale with number of servers

- One superserver will manage the whole network of servers

The package contains most of the server, communication layer client and server side, plugin layer clientside. we hope the release will fuel some more interest into the GPU project. Have fun!

* I read the book Crypto by Dan Brown and end up thinking how I would build my 'Digital Fortress' algorithm

Digital Fortress à la dangermouse

Depending on what to cipher, I would first choose the best compression algo, e.g. for text .zip, for pictures PNG or JPG (with loss) and apply compression to the 'plain text'. Then generate a hash of 'plain text' and append it. With ISAAC, a good pseudorandom algorithm I had the chance to analyze (initialized with some PBKDF2 of the 'key') I would generate a pseudorandom sequence of the same length of the compressed 'plain text' plus hash. Then XOR 'plain text' plus hash and pseudorandom sequence and finally execute a set of predefined statistical tests to ensure that at least the standard properties of white noise hold.

* To conclude I would cipher 'I love you' and send it to Virgi so that she can cranck on it :-)

31 Dec 2010 (updated 28 Jan 2011 at 09:31 UTC) »

arxiv Droid is my favourite Android app to see science progress. I like the 'Astrophysics All' and 'Computer Science All' categories where the latest papers on those topics are submitted. It is cool to see papers on exoplanets, gravitational anomalies and crazy algorithms costantly flowing :-)

New sync algo for deltasql

Wednesday of last week I had a WebEx session with three people in a company in India who are actively using deltasql. For me it was 7pm, for them it was 10:30pm, and they still were in the office! For me it was a honour that they waited so long to speak with me. Also strange that with India I have three a half hours time shift. I never hard of half an hour time shifts before.

The company they work for does software for hospitals in India and the US, and has as an inspiration a spiritual leader named Mata Amritanandamayi or Amma for friends, and she has many. Among big hospital in India, their company serves also an hospital of a nuclear facility.

The software they ship is a J2EE/mySQL framework based on Jboss, roughly the same architecture used in the company I was previously working for. Being all technology based on Open Source software, for them it was natural to use deltasql to manage the burden of keeping database schemas in sync. The deltasql server was customized with Amma's face and had 2070 scripts in it when they showed it to me in the Webex session.

Their software has a complex history of branching and the source code is managed by Subversion. They pointed out, that in deltasql the information from which a branch is originating is missing, which was the real reason why SQL synchronization on branches of branches never worked. The change in datamodel and the changes in the synchronization algorithm scared me first, beside the fact that I want to keep focus on the GPU project, which at the moment undergoes a full rewrite. Therefore I stated: I will think about it, maybe for end of 2011...

One and a half week later deltasql version 1.3.0 is published: it has a brand new synchronization algorithm which is implemented as a tree traversal from leaf to root recording each segment, and a tree traversal back from root to leaf generating the synchronization scripts for each segment. The final step collates the scripts generated in each segment and presents them to the user. Documentation and FAQs are updated as well.

Starwars ASCII

To complete the underground trip in the last blog entry, the ASCII movie of Star wars:


telnet towel.blinkenlights.nl
30 Oct 2010 (updated 30 Oct 2010 at 08:17 UTC) »

Martina is now 10 month old and while Virgi is at work on weekends I have to look at her. Fortunately, she is now opening a drawer and reordering its contents (mainly my pijamas), so that I have time to blog this entry! 8-)

git is faster than light

git is really fast, faster than my shadow and faster than cvs. I therefore moved deltasql repository on git and created a new git repository for what will become GPU II.

On gentoo, emerge git will install it. On ubuntu, sudo install apt-get gitcore will do it. On Windows, I use git extensions.

A good introduction to git is on the progit book. There is also a git plugin for eclipse, which I did not test until now.

Continouus database integration with deltasql

I finished the bash client for deltasql server, which can be used for continouus database integration. The bash client included in Release 1.2.0 supports Oracle and mySQL. The next release should support postgreSQL as well. I did an additional effort to document deltasql, trying to explain why there is need of projects and modules and how the synchronization mechanism works.

Going underground...

After reading the book, I went underground for a while, and I discovered these interesting things:

1. Facebook anno 1987

If you want to see how facebook looked like in 1987, telnet to the server mail.encompasserve.org and register an account for free following the given instructions carefully. It's an old but respectful VAX machine (OpenVMS Alpha Operating System). Then go and read the mails in the forum with these commands:


telnet mail.encompasserve.org
NOTES
OPEN WHO_AM_I
READ 4
READ 5
READ 6
READ 7
READ ...

To disconnect, follow this sequence:


exit
exit
disconnect

2. The Rogue game

The precursor of Dungeon and Dragons which was also featured in Scientific American. In the Italian version 'Le Scienze' I found the article for April 1985, edition number 200. I remember playing a version of this game named Moria on MS/DOS, when I was young and healthy 8-).

With this command sequence you can watch people and robots playing the game:


telnet nethack.alt.org
w
a

To exit, press three times 'q' on your keyboard.

3. Retrieving positions from celestial objects from NASA server

Instead of hacking into NASA as suggested by the book, one might enter NASA by its front door. Assume we want to know how far from the Sun the most far thing built by mankind is, then we need to ask HORIZONS.


telnet ssd.jpl.nasa.gov 6775
Pioneer 11
... scroll down until 100% with Enter ...
e
v
500@Sun
y
eclip
2010-Oct-30
2010-Oct-31
1d
y
... scroll down until you see ...
... JDCT  ...
... X  Y  Z
2455499.500000000 = A.D. 2010-Oct-30 00:00:00.0000 (CT)
   1.538965002087235E+01 -7.644892272720375E+01  
1.995301567624495E+01

It means that Pioneer 11 is distant from sun 15.38 AU in X direction, -76.45 AU in Y direction and 19.95 AU in Z direction as of 30 October 2010 (today). Calculating the distance with Pythagoras gives a distance from the Center of the Sun of 80.49 AU. An AU (Astronomical Unit) is defined as the average distance between Earth and Sun, it's about 100 million km. So Pioneer 11 is distant from us 80.49 * 100E6 km, 8.049 billion kilometers!

Sending a signal to Pioneer 11 should take 8049E9 m / (3E8 (m/s)) = 26830 s = 7.45 hours of travel in the empty darkness of space until it reaches the space probe. So as last exercise, execute this command, but you need to wait 2*7.45 hours until you get an answer as the signal needs to travel back:


telnet spaceprobes.jpl.nasa.gov 6775
enable redshift adjust
connect Pioneer 11
echo "hi little green man, please do not touch the 
antenna, thanks!"

Just a joke ;-)

I found the Underground book of wikileaks founder online in text format. Cool!

Volcano Eyjafjallajoekull

I finished coding the climate simulation frontend for the GPU project. It has some neat features like ashes of Volcano Eyjafjallajoekull which in fact cover Europe during Summer and CO2 cycle based on population density data of U.N. The nuclear war feature still needs some improvements, though.

The atmospheric model is quite simple and do not now if it can stand scientific simulations. The frontend has plenty of parameters which affect the simulation. Inverting earth rotation will also invert coriolis effect on winds, for example. It is possible to put Earth in Mercury or Iupiter orbit and see what happens. Go stop earth rotation or revolution around Sun, or change water density!

To test the climate simulation, it is suggested to download GPU and upgrade it via menu point "Tools->Autoupdate to latest version".

This frontend is completely written in Lazarus/Freepascal. To me, it looks like the Lazarus project is close to production stage, and it is even more powerful than Delphi itself.

ASURO robot

I had a chance to play with the Asuro robot. It was my first experience with the 8bit Atmel ATmega8L microprocessor. Until now I only looked at the PICs microprocessor family without doing anything concrete with it.

One thing to know about ASURO is the behaviour of the PollSwitch() function. The switch sensors return unreliable data. In forums, it is possible to read that only if PollSwitch() returns several times the same value, this value is reliable. Therefore our C-code changed like this:


unsigned char test1, test2, test3, test4;


while (1) {

test1 = PollSwitch(); // retrieve data from LED test2 = PollSwitch(); // retrieve data from odometric sensors test3 = PollSwitch(); // do some more logic test4 = PollSwitch();

if ((test1==test2) && (test2==test3) && (test3==test4)) { if (test1>0) { // at least one of the six sensors is colliding with an object } else { // no collision }

}

In case the code mentioned above does not work, than try this piece of code we found on the net by hai1991. The code checks if resistors in front of the switches are soldered correctly. Pressing the switches lights LED lamps in a funny way.


#include "asuro.h"
unsigned char taste;
 
int main(void)
{


Init(); while(1) { do { taste = PollSwitch(); } while( taste != PollSwitch());

switch(taste) { case 1: StatusLED (GREEN); break; case 2: StatusLED (RED); break; case 4: FrontLED (ON); break; case 8: BackLED (OFF,ON); break; case 16: BackLED (ON,OFF); break; case 32: BackLED (ON,ON); break; default: StatusLED(OFF); BackLED(OFF,OFF); FrontLED(OFF); break; } }

return 0; }

Geocaching

About one year ago, Virginia and me hided this geocache called Lago di Poschiavo close to where we live. Until now it was found 36 times. I wonder if Martina (now 7 months old) will like Geochache. I asked her, she answered me with "ta ta ta", but I am not sure of what it means :-)

23 Jun 2010 (updated 23 Jun 2010 at 14:23 UTC) »
Texturing a sphere in OpenGL?

I am writing a simple climate simulation which will be released soon as frontend for the Global Processing unit project.

One of the tasks is about plotting the Earth in OpenGL, for this I have bitmaps with continents, clouds, temperatures, etc, which I would like to project on the sphere.

I read through documentation on how to texture a sphere. After a while, I decide to go for this approach instead (in Freepascal, which is a phantastic Open source substitute for Delphi):

I create first a "sphere grid" in an array called sphere3d:


procedure init3DGrid(var w : TWorld; var clima : TClima);
var i, j : Longint;
    p1   :T3DPoint;
    lat, lon, altitude : Extended;
begin
 for j := 0 to 179 do
     for i := 0 to 359 do
       begin
         lat := YtoLat(j);       //-90..+90
         lon := XtoLon(i);       //-180..+180
         lat := lat/90 * Pi/2 + Pi/2; //0..180
         lon := lon/180 * Pi + Pi;  //0..360
// adding half a degree to longitude for triangles
// in each second row
         if (j mod 2 = 0) then
               lon := lon + 1/720 * 2 * Pi;
         p1.x := - radius * sin(lat) * cos(lon);
         p1.z :=   radius * sin(lat) * sin(lon);
         p1.y := - radius * cos(lat);
         sphere3D[i][j] := p1;
       end;
end;


function XtoLon(x : Longint) : Double; begin if (x<0) or (x > 359) then raise Exception.create('x on array has to be between 0 and 359 but was '+IntToStr(x)); Result := x - 180; end;

function YtoLat(y : Longint) : Double; begin if (y<0) or (y > 180) then raise Exception.create('y on array has to be between 0 and 180 but was '+IntToStr(y)); Result := 90 - y; end;

And in the Paint method of my extended TOpenGLControl I do a call to plot3d with my color bitmap and the vertex array computed in the previous procedure Init3dgrid:


procedure plot3d(vertex : P3DGrid; colors : PGridColor);
var i, j,
    target_i,
    target_j : Longint;
    p1,p2,p3,p4 : T3DPoint;
    r, g, b : Extended;
begin
  for j := 0 to 179 do
     for i := 0 to 359 do
       begin
          target_i := i+1;
          target_j := j+1;
          if (target_i>359) then target_i := target_i-359;
          if (target_j>179) then target_j := target_j-179;
          p1 := vertex^[i]  [j];
          p2 := vertex^[target_i][j];
          p3 := vertex^[i][target_j];
          p4 := vertex^[target_i][target_j];
          r := Red(colors^[i][j])/255;
          g := Green(colors^[i][j])/255;
          b := Blue(colors^[i][j])/255;
          glBegin(GL_TRIANGLES);
          glColor3f(r,g,b);
          glVertex3f( p1.x, p1.y, p1.z);              
          glVertex3f( p2.x, p2.y, p2.z);              
          glVertex3f( p3.x, p3.y, p3.z);              
          glEnd();
          glBegin(GL_TRIANGLES);
          glColor3f(r,g,b);
          glVertex3f( p2.x, p2.y, p2.z);              
          glVertex3f( p3.x, p3.y, p3.z);              
          glVertex3f( p4.x, p4.y, p4.z);              
          glEnd();
       end;
end; 

For my limited knowledge of OpenGL, this is easier to setup (and for me to understand) than loading a texture and using a cubic or cilindric projection on a sphere. The performance of this solution is also acceptable, at least for my Earth surface with 129600 (360x180*2) triangles.

On Monday, February 10, I was for working reasons in Milan. In the evening, I randomly walked around the city (following a M sign, thinking it was Mc Donalds, but it was M like Metropolitana, the tube). Close to the Duomo, I read a sign about a conference on micro credit made by Professor Yunus and I remembered reading an article about micro credit in some newspaper (maybe La Repubblica?) ...

I searched for 5 minutes on the map to locate "Teatro dal Verme" where the conference was scheduled and rushed to Castello Sforzesco. In front of the theater there was plenty of police. As I was a little bit late, I decided for the strategy "walk through as you would know where you are headed", because I am equipped with a Swiss army knife and I did not want to be searched and loose the knife and precious time.

The entry was free, I came at the top of the stairs of a huge theater and was very happy to see that what was happening on the stage was projected on the wall, too.

A young girl, Erica Mu, played her song with a guitar using a technique which remembered me of an oscillator where the signal is reinforced by a resonating channel. Then, Paola Turci played her song "Rwanda" which is never broadcasted on the radio, unfortunately. And Paola is a famous Italian singer, too.

A world without poverty

Professor Yunus came on the stage followed by a train of photographers, and explained the principles of micro credit with these words.

"I looked at how a conventional bank works" - he said - " and I did exactly the opposite!" He explained: "First, a conventional bank lends only to rich people, and the richer the more they lend. We lend to the poor people, the poorer, the more we lend. Secondly, a conventional bank lend to men. We lend to women :-)"

He then added "Conventional banks build their buildings in the center of the city, we build our banks in the villages, not in the cities. Customers have to go to the bank, we go to our customers. And conventional banks lend huge amounts of money, we lend only little amounts".

He then recalled a poor woman who received from Yunus 30$ as a credit. She was trembling as she never had touched money in her life. So, she kindly denied, but Yunus insured her that she would pay back only if the eggs of the hen she would buy would give her some kind of return. Yunus soon learned that poor people are honest, they pay back if only they can! (Ehm, just by side as my humble remark: can we say the same of very rich people when they generously decide about their own bonus?)

These principles were the spark to build the Grameen bank, which is a micro credit foundation that helped many Bangladesh people to come out of poverty. With many, we say more than 100 million.

Another fundamental thought in Yunus is that being poor does not mean at all being stupid or less in any way than someone who is rich. In the speech he explained that he was delighted when he met the daughter of a poor woman which had nothing, who managed to become a physician respected by the whole village. Everyone is intelligent and can come out of poverty, if the system does not prevent him to do so.

He extended the concept of micro credit to education and helped students in funding their studies. However, when students finished their education, they came to him and said: "Yunus, unfortunately, there are no jobs where I could apply what I hardly learned!". Yunus explained them, that they should not stand up each morning and think about job seeking, they are powerful enough to think about job giving, to create companies and to create jobs.

In particular, if a student had a good idea, Yunus encouraged him to leave the study and establish his own company. A diploma is a mere piece of paper which gets meaningless if there are no jobs in the economy.

In one of his experiments which keeps running, professor Yunus gave beggars small useful things (like lighters and similar) and asked them instead of begging on the same place, to walk house by house and try to sell these things. Yunus feared beggars did not feel motivated, but he was wrong: they were very happy to act as traveling salesmen. Yunus allowed them to beg in case they were not able to sell anything.

"Funnily enough, as time went by, beggars knew exactly which houses were good to sell things, and which ones were good to beg. Even if they did not own a master degree at Harvard, beggars understood the concept of market segmentation very well!", he added.

In his talk, Yunus criticized the neo-liberal thinking and questioned the principle that being selfish makes everyone richer. This attitude is one of the reasons of the current financial crisis. Yunus believes that being selfish is fine and is related to the instinct of conservation, but it does not have to become a reason of life. Everyone likes to help others and gains positive feelings about it.

In this sense, he called big companies to "social responsibility". With Danone, he created a particular cheap yogurt and marketed it in Bangladesh. A little child that eats for several months this yogurt will not suffer of some of the diseases created by malnutrition. With Adidas, he tries to sell cheap shoes that prevent people to go round barefoot and catch other diseases. These are challenges were profit is not the focus, the focus is something like "create a good cheap shoe for less than 1$". And of course, the image of such responsible companies help them in the conventional economy.

Professor Yunus looks very young for his age, he is 70! And he keeps creating this kind of positive powerful never ending cycles :-)

Finally, an example to say how real the thing might get: I got as a gift for my little child born at the beginning of January a band to carry her around. On the package containing the band, there was a little round stick saying "Grameen Foundation".

Einstein radically upset the way scientists looked at physics, Yunus will radically change the way we understand micro and macroeconomics.

More on Professor Yunus can be found on wikipedia at http://en.wikipedia.org/wiki/Muhammad_Yunus

One solution to escape the cage of firewalls around you and reach your beloved server is

Install Ajaxterm on Gentoo

These are the steps I followed, they should complement the explanation of the wiki of Ajaxterm, created by Antony Lesuisse.

1. setup a Dyndns account and ddclient

Publish the IP address of your router first with these steps:

Visit Dyndns, register an account and your host. Choose a freely available DNS name like delta.dyndns.org. On your Gentoo box, install ddclient


USE="ssl" emerge ddclient
su
vi /etc/ddclient/ddclient.conf

In ddclient.conf enter your account details for Dyndns.

Launch and register ddclient with


/etc/init.d/ddclient start
rc-update add ddclient default

2. install Ajaxterm


ACCEPT_KEYWORDS="~x86" emerge ajaxterm
/etc/init.d/ajaxterm start

If launching ajaxterm fails with [!!], do the following:


vi /etc/services

At the bottom of the file services add


ajaxterm     8022/tcp
ajaxterm     8022/udp

If launching ajaxterm still fails with [!!], in /etc/init.d/ajaxterm, change the DAEMON variable to /usr/bin/python2.5.

Once ajaxterm starts with a green [OK], point a Javascript enabled browser (the text based links or lynx do not work!) to http://localhost:8022/. If your Gentoo box does not have Gnome or KDE on it, you can also point a browser on the same network segment, e.g. to http://192.168.1.33:8022/. Ajaxterm will welcome you with the login window :-).

It's time to register ajaxterm to the default runlevel with


rc-update add ajaxterm default
so that it survives reboot...

3. Emerge and configure Apache

In the wiki of Ajaxterm, it is suggested to use Apache as SSL tunnel to Ajaxterm, this for security reasons.

Let's build an apache with some modules ;-)


su
echo 'APACHE2_MODULES="authz_host actions alias auth_basic
auth_digest authn_anon authn_dbd authn_dbm authn_default
authn_file authz_dbm authz_default authz_groupfile
authz_host authz_owner authz_user autoindex cache dav dav_fs
dav_lock dbd deflate dir disk_cache env expires ext_filter
file_cache filter headers ident imagemap include info
log_config logio mem_cache mime mime_magic negotiation proxy
proxy_ajp proxy_balancer proxy_connect proxy_http rewrite
setenvif speling status unique_id userdir usertrack
vhost_alias"' >> /etc/make.conf


USE="ssl" emerge -av apache

Make sure that D_PROXY and D_PROXY_HTTP are defined in the apache2 file:


vi /etc/conf.d/apache2
[...]
APACHE2_OPTS="-D DEFAULT_VHOST -D INFO -D SSL -D
SSL_DEFAULT_VHOST -D PROXY -D P
ROXY_HTTP" 
[...]

Configure the SSL tunnel with


vi /etc/apache2/vhosts.d/00_default_ssl_vhost.conf
[...]
VirtualHost *:443>
       ServerName localhost
       SSLEngine On


ProxyRequests Off <Proxy *> Order deny,allow Allow from all </Proxy> ProxyPass /ajaxterm/ http://localhost:8022/ ProxyPassReverse /ajaxterm/ http://localhost:8022/ </VirtualHost>

[...]

Start and register Apache2 with


/etc/init.d/apache2 start
rc-update add apache2 default

4. create a certificate for Apache

Here the steps to create a self-signed certificate are described:


openssl req -new > new.cert.csr
When asked for the CommonName, give the DNS name you chose (e.g. delta.dyndns.org). This is important to make the self-signed certificate valid.

Remove the passphrase with


openssl rsa -in privkey.pem -out new.cert.key

Convert the request into the signed certificate:


openssl x509 -in new.cert.csr -out new.cert.cert -req
-signkey new.cert.key -days 3650

Indicate to apache where to find the signed certificate and the key:


vi /etc/apache2/vhosts.d/00_default_ssl_vhost.conf
[...]
SSLCertificateFile /path/to/certs/new.cert.cert
SSLCertificateKeyFile /path/to/certs/new.cert.key
[...]

Source for these steps: Apache SSL docs

5. configure NAT on your router

Login to your router, go to the NAT section and forward the port 443 to the server where Ajaxterm is running...

That's all, from a caged environemnt, simply point your browser to a https://delta.dyndns.org/ajaxterm/ and add a certificate exception, as the certificate was self signed and not issued by a CA authority.

Have fun!

People who like quiz like

Cescozen

Cescozen is a game invented by Cesco. If you point your browser to the initial page of the game, you'll encounter very difficult enigmas that wait for a solution (note: the game is in Italian). To answer each quiz, you put the solution on the URL, so that you move one directory down the tree. To me and Virgi, it took about a week investing 2-3 hours each day to get through all levels. Sometimes, the quiz requires that you search something in Google, or you have a satellite image and you need to figure out where it is. Very impressive, amazing game!

People who like quiz like also to follow howto instructions to get

The full Map Stack up and running

I am fascinated by the Open Source technology behind OpenStreetMap, which I believe will soon be succesful as Wikipedia is.

After importing the dump of Switzerland in the old Gentoo server, I showed the result to my father and he gave me another piece of iron to reach the next level of this particular game: it is an IBM with 3.2 GHZ and 1 GB of RAM. I installed Ubuntu Jaunty (9.04) on the server and repeated the PostgreSQL and mapnik install following these howto instructions made by Johnny Dobbins. This tutorial additionally explains how to install the WMS server called ogcserver and how to put a cache in front of it, so that already computed tiles do not need to be recomputed but can just be served directly from disk (or from memcache or even from Amazon S3).

Just some remarks on how the job was done:

Choosing a bounding box

the server has 74 GB of disk, so based on this specs I decided to import Europe and part of Asia till longitude of Moskau and not the whole planet. For the whole planet you probably need at least 200 GB of disk.

To be precise I chose this rectangle with coordinates taken from an old fashioned Atlas: Top left corner: 75 N, 35 W Bottom right corner: 20N 60 E

This command extracted from the whole planet file the bounding box and created a new dump called eurasia.osm.bz2. The decompression and compression is done with bzcat and bzip2 piped together.


bzcat ../../planet-latest.osm.bz2 | ./bin/osmosis --read-xml
enableDateParsing=no file=/dev/stdin --bounding-box left=-35
bottom=20 right=60 top=75 --write-xml file=- | bzip2 >
../../eurasia.osm.bz2

The execution lasted for about 14 hours. Out of the 7.4 GB of planet-latest.osm.bz2, about 2.2 GB of data where extracted for eurasia.osm.bz2. After the import with osm2pgsql in Postgres database, the bounding box takes about 40 GB of disk space.

Minor remarks on the howto of Johnny

I did not find the lines in cgiserver.py, so I left this step out.

For the file wms.py: you need to completely replace the existing file with the lines shown in the howto.

Setup of ogcserver

When browsing to http://localhost:8080, I got an error. Looking at /var/log/apache2/error.log,

I noticed this error:


IOError: sys.stdin access restricted by mod_wsgi           
                      

I then added at the bottom of /etc/apache2/sites-available/default outside the VirtualHost section the following two lines:


WSGIRestrictStdout Off
WSGIRestrictStdin Off

Tilecache: Seeding the cache

After setting everything up, pieces of the map started appearing. However, often the server got overloaded, if it had to compute tiles which where full of features (streets and rivers). Especially the tile with Germany never appeared as it was too feature rich. Eventually, the server hang up when even the swap cache got full.

The solution to this performance problem was to seed the cache. I chose a point in the center of Europe (46 N, 10 E) and a radius of 30 degrees.

I then issued in the directory ogcserver/tilecache-2.10/ the following command:


./tilecache_seed.py basic
46,10,30
[press Control D on the keyboard]

and I let compute map tiles for about 48 hours.

The result

Here it is :-). Happy browsing, press twice the plus button first, to reach the correct zoom level. The speed in serving tiles might vary, as I have an ADSL and use dynamic DNS...

11 Dec 2009 (updated 11 Dec 2009 at 13:29 UTC) »

I managed to install Openstreetmap on Gentoo (status: December 2009) and I could generate some interesting tiles of Poschiavo, which is a little valley in the mountains where I live. My server is an AMD k6-600 MHZ with 600 MB of RAM, so not a real power horse but quite an old scrap iron thing. Of course, I did not import the whole planet, but only a dump of Switzerland.

Your own Openstreet map tile server on Gentoo

I wrote down some steps that differ from the excellent tutorials here and here, because my installation is performed on Gentoo instead of Ubuntu: of course, this is working now as for December 2009, but an emerge --sync in the future might solve some of these problematic steps (maybe someone more knowledgeable than me will read this tutorial and fix some ebuilds :-).

Step: Subversion and Java

These one were easy and you probably figured out faster than me: emerge -av subversion
emerge -av sun-jdk

Step: Installing Postgresql with Postigs spatial extensions

osm2pgsql does not work with Postgresql 8.1 which is marked stable in Gentoo. The reason is that Postgresql 8.1.11 does not know about the command 'DROP TABLE IF EXISTS' which is used in oms2pgsql. Defining a stored procedure that does the same and modifying the source code of osm2pgsql did not work for me.

Therefore I chose another way, which goes as follows: you should add at the end of /root/.bashrc the following alias:

alias aemerge='ACCEPT_KEYWORDS="~x86" emerge '
source /root/.bashrc will load this setting for the first time. The setting is kept after, as .bashrc is read at every login. From now on, you can use "aemerge" to emerge unstable packages instead of "emerge".

Then, you can install an unstable Postgresql with
aemerge -av postgresql

This will install a Postgresql 8.2.14 which is fine for our purposes.

Install also Postgis:
aemerge -av postgis
Postgis-1.4.0 is then installed.

Configure the database postgis extensions not by using emerge --config as suggested at the end of emerge, but use the instructions provided in this tutorial. Remember to follow the instructions as normal user and not as superuser (a part of the inital step of setting up the database).

As noted in the previous comment, lwpostgresql.sql does not exist in postgis anymore, it was renamed in postgis.sql, you should therefore use
psql -d gis -f /usr/share/postgresql/contrib/postgis.sql

Step: Installing mapnik
For mapnik I had problems, it did not compile from source as described in the tutorial, it was failing when compiling the postgis plugin for linking reasons against the library libpq.la, though the library was present on the proposed path.

I did the following trick:

USE="postgresql" aemerge -av mapnik

emerged and installed a mapnik with all dependencies, which however was not able to recognize the postgis extensions on the 'gis' database.

In fact, when running ./z0_generate_tiles.py, mapnik was failing with:

Could not create datasource. No plugin found for type 'postgis' (encountered during parsing of layer 'leisure') when connecting to the imported database.

Then I did the following: I unmerged the unstable libpq library (libpq is a library used by postgresql used as connection layer.
emerge --unmerge libpq
I unmerged mapnik without its dependencies with
emerge --unmerge mapnik
I installed the stable libpq library with:
emerge -av libpq
Then I installed mapnik from source with the following steps
mkdir src
cd src
svn co http://svn.mapnik.org/trunk/ mapnik
cd mapnik
scons/scons.py -c install
scons/scons.py PGSQL_LIBS=/usr/lib/postgresql/ PGSQL_INCLUDES=/usr/include/postgresql/
sudo scons/scons.py install PGSQL_LIBS=/usr/lib/postgresql/ PGSQL_INCLUDES=/usr/include/postgresql/
ldconfig

Therefore: postgresql is compiled against an unstable version of libpq, while mapnik needs a stable one to compile. Of course, postgresql will survive the restart with
/etc/init.d/postgresql restart
:-)

Have fun with your maps! Import the planet on your favourite Gentoo box!

I was browsing through the apache logs with cat /var/log/apache2/access_log and I wondered how many crawlerbots are visiting our webserver... No wonder that the visitor counts are always showing way too many visitors :-D

Coffee Break Planner

To increase the number of visitors to our webserver, I finally found a killer application. I developed it as a Google Gadget and is called the Coffee Break Planner. This gadget helps you to meet with your friends at the right time for a tasteful coffee... It also increases the overall performance of your company, as in coffee breaks most of the time people speak about work anyway :-)

Again, the gadget is only an xml envelope, while PHP and mySQL do the hard work.

37 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!