Older blog entries for dangermaus (starting at number 40)

One solution to escape the cage of firewalls around you and reach your beloved server is

Install Ajaxterm on Gentoo

These are the steps I followed, they should complement the explanation of the wiki of Ajaxterm, created by Antony Lesuisse.

1. setup a Dyndns account and ddclient

Publish the IP address of your router first with these steps:

Visit Dyndns, register an account and your host. Choose a freely available DNS name like delta.dyndns.org. On your Gentoo box, install ddclient


USE="ssl" emerge ddclient
su
vi /etc/ddclient/ddclient.conf

In ddclient.conf enter your account details for Dyndns.

Launch and register ddclient with


/etc/init.d/ddclient start
rc-update add ddclient default

2. install Ajaxterm


ACCEPT_KEYWORDS="~x86" emerge ajaxterm
/etc/init.d/ajaxterm start

If launching ajaxterm fails with [!!], do the following:


vi /etc/services

At the bottom of the file services add


ajaxterm     8022/tcp
ajaxterm     8022/udp

If launching ajaxterm still fails with [!!], in /etc/init.d/ajaxterm, change the DAEMON variable to /usr/bin/python2.5.

Once ajaxterm starts with a green [OK], point a Javascript enabled browser (the text based links or lynx do not work!) to http://localhost:8022/. If your Gentoo box does not have Gnome or KDE on it, you can also point a browser on the same network segment, e.g. to http://192.168.1.33:8022/. Ajaxterm will welcome you with the login window :-).

It's time to register ajaxterm to the default runlevel with


rc-update add ajaxterm default
so that it survives reboot...

3. Emerge and configure Apache

In the wiki of Ajaxterm, it is suggested to use Apache as SSL tunnel to Ajaxterm, this for security reasons.

Let's build an apache with some modules ;-)


su
echo 'APACHE2_MODULES="authz_host actions alias auth_basic
auth_digest authn_anon authn_dbd authn_dbm authn_default
authn_file authz_dbm authz_default authz_groupfile
authz_host authz_owner authz_user autoindex cache dav dav_fs
dav_lock dbd deflate dir disk_cache env expires ext_filter
file_cache filter headers ident imagemap include info
log_config logio mem_cache mime mime_magic negotiation proxy
proxy_ajp proxy_balancer proxy_connect proxy_http rewrite
setenvif speling status unique_id userdir usertrack
vhost_alias"' >> /etc/make.conf


USE="ssl" emerge -av apache

Make sure that D_PROXY and D_PROXY_HTTP are defined in the apache2 file:


vi /etc/conf.d/apache2
[...]
APACHE2_OPTS="-D DEFAULT_VHOST -D INFO -D SSL -D
SSL_DEFAULT_VHOST -D PROXY -D P
ROXY_HTTP" 
[...]

Configure the SSL tunnel with


vi /etc/apache2/vhosts.d/00_default_ssl_vhost.conf
[...]
VirtualHost *:443>
       ServerName localhost
       SSLEngine On


ProxyRequests Off <Proxy *> Order deny,allow Allow from all </Proxy> ProxyPass /ajaxterm/ http://localhost:8022/ ProxyPassReverse /ajaxterm/ http://localhost:8022/ </VirtualHost>

[...]

Start and register Apache2 with


/etc/init.d/apache2 start
rc-update add apache2 default

4. create a certificate for Apache

Here the steps to create a self-signed certificate are described:


openssl req -new > new.cert.csr
When asked for the CommonName, give the DNS name you chose (e.g. delta.dyndns.org). This is important to make the self-signed certificate valid.

Remove the passphrase with


openssl rsa -in privkey.pem -out new.cert.key

Convert the request into the signed certificate:


openssl x509 -in new.cert.csr -out new.cert.cert -req
-signkey new.cert.key -days 3650

Indicate to apache where to find the signed certificate and the key:


vi /etc/apache2/vhosts.d/00_default_ssl_vhost.conf
[...]
SSLCertificateFile /path/to/certs/new.cert.cert
SSLCertificateKeyFile /path/to/certs/new.cert.key
[...]

Source for these steps: Apache SSL docs

5. configure NAT on your router

Login to your router, go to the NAT section and forward the port 443 to the server where Ajaxterm is running...

That's all, from a caged environemnt, simply point your browser to a https://delta.dyndns.org/ajaxterm/ and add a certificate exception, as the certificate was self signed and not issued by a CA authority.

Have fun!

People who like quiz like

Cescozen

Cescozen is a game invented by Cesco. If you point your browser to the initial page of the game, you'll encounter very difficult enigmas that wait for a solution (note: the game is in Italian). To answer each quiz, you put the solution on the URL, so that you move one directory down the tree. To me and Virgi, it took about a week investing 2-3 hours each day to get through all levels. Sometimes, the quiz requires that you search something in Google, or you have a satellite image and you need to figure out where it is. Very impressive, amazing game!

People who like quiz like also to follow howto instructions to get

The full Map Stack up and running

I am fascinated by the Open Source technology behind OpenStreetMap, which I believe will soon be succesful as Wikipedia is.

After importing the dump of Switzerland in the old Gentoo server, I showed the result to my father and he gave me another piece of iron to reach the next level of this particular game: it is an IBM with 3.2 GHZ and 1 GB of RAM. I installed Ubuntu Jaunty (9.04) on the server and repeated the PostgreSQL and mapnik install following these howto instructions made by Johnny Dobbins. This tutorial additionally explains how to install the WMS server called ogcserver and how to put a cache in front of it, so that already computed tiles do not need to be recomputed but can just be served directly from disk (or from memcache or even from Amazon S3).

Just some remarks on how the job was done:

Choosing a bounding box

the server has 74 GB of disk, so based on this specs I decided to import Europe and part of Asia till longitude of Moskau and not the whole planet. For the whole planet you probably need at least 200 GB of disk.

To be precise I chose this rectangle with coordinates taken from an old fashioned Atlas: Top left corner: 75 N, 35 W Bottom right corner: 20N 60 E

This command extracted from the whole planet file the bounding box and created a new dump called eurasia.osm.bz2. The decompression and compression is done with bzcat and bzip2 piped together.


bzcat ../../planet-latest.osm.bz2 | ./bin/osmosis --read-xml
enableDateParsing=no file=/dev/stdin --bounding-box left=-35
bottom=20 right=60 top=75 --write-xml file=- | bzip2 >
../../eurasia.osm.bz2

The execution lasted for about 14 hours. Out of the 7.4 GB of planet-latest.osm.bz2, about 2.2 GB of data where extracted for eurasia.osm.bz2. After the import with osm2pgsql in Postgres database, the bounding box takes about 40 GB of disk space.

Minor remarks on the howto of Johnny

I did not find the lines in cgiserver.py, so I left this step out.

For the file wms.py: you need to completely replace the existing file with the lines shown in the howto.

Setup of ogcserver

When browsing to http://localhost:8080, I got an error. Looking at /var/log/apache2/error.log,

I noticed this error:


IOError: sys.stdin access restricted by mod_wsgi           
                      

I then added at the bottom of /etc/apache2/sites-available/default outside the VirtualHost section the following two lines:


WSGIRestrictStdout Off
WSGIRestrictStdin Off

Tilecache: Seeding the cache

After setting everything up, pieces of the map started appearing. However, often the server got overloaded, if it had to compute tiles which where full of features (streets and rivers). Especially the tile with Germany never appeared as it was too feature rich. Eventually, the server hang up when even the swap cache got full.

The solution to this performance problem was to seed the cache. I chose a point in the center of Europe (46 N, 10 E) and a radius of 30 degrees.

I then issued in the directory ogcserver/tilecache-2.10/ the following command:


./tilecache_seed.py basic
46,10,30
[press Control D on the keyboard]

and I let compute map tiles for about 48 hours.

The result

Here it is :-). Happy browsing, press twice the plus button first, to reach the correct zoom level. The speed in serving tiles might vary, as I have an ADSL and use dynamic DNS...

11 Dec 2009 (updated 11 Dec 2009 at 13:29 UTC) »

I managed to install Openstreetmap on Gentoo (status: December 2009) and I could generate some interesting tiles of Poschiavo, which is a little valley in the mountains where I live. My server is an AMD k6-600 MHZ with 600 MB of RAM, so not a real power horse but quite an old scrap iron thing. Of course, I did not import the whole planet, but only a dump of Switzerland.

Your own Openstreet map tile server on Gentoo

I wrote down some steps that differ from the excellent tutorials here and here, because my installation is performed on Gentoo instead of Ubuntu: of course, this is working now as for December 2009, but an emerge --sync in the future might solve some of these problematic steps (maybe someone more knowledgeable than me will read this tutorial and fix some ebuilds :-).

Step: Subversion and Java

These one were easy and you probably figured out faster than me: emerge -av subversion
emerge -av sun-jdk

Step: Installing Postgresql with Postigs spatial extensions

osm2pgsql does not work with Postgresql 8.1 which is marked stable in Gentoo. The reason is that Postgresql 8.1.11 does not know about the command 'DROP TABLE IF EXISTS' which is used in oms2pgsql. Defining a stored procedure that does the same and modifying the source code of osm2pgsql did not work for me.

Therefore I chose another way, which goes as follows: you should add at the end of /root/.bashrc the following alias:

alias aemerge='ACCEPT_KEYWORDS="~x86" emerge '
source /root/.bashrc will load this setting for the first time. The setting is kept after, as .bashrc is read at every login. From now on, you can use "aemerge" to emerge unstable packages instead of "emerge".

Then, you can install an unstable Postgresql with
aemerge -av postgresql

This will install a Postgresql 8.2.14 which is fine for our purposes.

Install also Postgis:
aemerge -av postgis
Postgis-1.4.0 is then installed.

Configure the database postgis extensions not by using emerge --config as suggested at the end of emerge, but use the instructions provided in this tutorial. Remember to follow the instructions as normal user and not as superuser (a part of the inital step of setting up the database).

As noted in the previous comment, lwpostgresql.sql does not exist in postgis anymore, it was renamed in postgis.sql, you should therefore use
psql -d gis -f /usr/share/postgresql/contrib/postgis.sql

Step: Installing mapnik
For mapnik I had problems, it did not compile from source as described in the tutorial, it was failing when compiling the postgis plugin for linking reasons against the library libpq.la, though the library was present on the proposed path.

I did the following trick:

USE="postgresql" aemerge -av mapnik

emerged and installed a mapnik with all dependencies, which however was not able to recognize the postgis extensions on the 'gis' database.

In fact, when running ./z0_generate_tiles.py, mapnik was failing with:

Could not create datasource. No plugin found for type 'postgis' (encountered during parsing of layer 'leisure') when connecting to the imported database.

Then I did the following: I unmerged the unstable libpq library (libpq is a library used by postgresql used as connection layer.
emerge --unmerge libpq
I unmerged mapnik without its dependencies with
emerge --unmerge mapnik
I installed the stable libpq library with:
emerge -av libpq
Then I installed mapnik from source with the following steps
mkdir src
cd src
svn co http://svn.mapnik.org/trunk/ mapnik
cd mapnik
scons/scons.py -c install
scons/scons.py PGSQL_LIBS=/usr/lib/postgresql/ PGSQL_INCLUDES=/usr/include/postgresql/
sudo scons/scons.py install PGSQL_LIBS=/usr/lib/postgresql/ PGSQL_INCLUDES=/usr/include/postgresql/
ldconfig

Therefore: postgresql is compiled against an unstable version of libpq, while mapnik needs a stable one to compile. Of course, postgresql will survive the restart with
/etc/init.d/postgresql restart
:-)

Have fun with your maps! Import the planet on your favourite Gentoo box!

I was browsing through the apache logs with cat /var/log/apache2/access_log and I wondered how many crawlerbots are visiting our webserver... No wonder that the visitor counts are always showing way too many visitors :-D

Coffee Break Planner

To increase the number of visitors to our webserver, I finally found a killer application. I developed it as a Google Gadget and is called the Coffee Break Planner. This gadget helps you to meet with your friends at the right time for a tasteful coffee... It also increases the overall performance of your company, as in coffee breaks most of the time people speak about work anyway :-)

Again, the gadget is only an xml envelope, while PHP and mySQL do the hard work.

Webserver Gate Gadget

I created this gadget to upload files to my webserver as ftp is firewalled in the company I am working for. I prepared it so that other people can install on their webserver as well, if they read the Readme on the gadget :-)

To work on the webserver I use this service, which costed me 15 euro for one year. It is a Unix shell via HTTPS that can be opened with a normal browser. Tunneling with httptunnel or corkscrew or any other trick did not work for me... Too stupid to break out without commercial support, ehm!

Joomla

Since beginning of the year I manage the local boy scout homepage with Joomla and I am very happy with this solution, last but not least released as Open Source.

Android emulator v2.0

I am experimenting with Eclipse, Android v2.0 and a tutorial a friend did... But I still do not have meaningful results. Some Java exception crashes the small application and the emulator does not work behind the proxy, even if the -http-proxy option is set in the launch parameters. Notice that the mysql trick works only with Android 1.6. Probably the proxy filters also on the browser identification string... and the emulator does not mimic a Mozilla or IE as identification... And the proxy is difficult to overcome...

23 Oct 2009 (updated 23 Oct 2009 at 07:11 UTC) »

My fifth and sixth gadgets are born. Their internals are quite the same: they have a DOM parser and they parse RSS feeds. One grasps news of my valley, the other one retrieves news of the local boy scout group in Poschiavo. I submitted five out of six gadgets to the Google directory, and they start appearing in searches, including description, screenshot and thumbnail.

Riding the Wave

So, these six gadgets are leading me through a learning path for Javascript, I assume that if I combine Javascript and XML this might lead me one day to AJAX technologies. Also, it remembers me a little bit the crazy Java Server Faces experience I had in my previous job. JSF is a braindead technology, in my humble opinion. And Javascript understanding might help me in setting up a BOINC project for GPU. Another nice thing is that I got an invitation to Google Wave and gadgets are a building block of Wave. I can include my own gadgets to waves where others partecipate :-) The wave thing really has some appeal and might replace one day email traffic. In the beginning, I was impressed by Wave, now I miss a little bit people to interact with... And I still cannot invite other people to Wave. Probably there is some timer in there and I have to wait some more time.

End of Geocities?

I was surprised when visiting my page on Tuesday of this week, that geocities will close my homepage, unless I pay a monthly fee. I had this page since July 1998, and though it was plagued by advertisement, I kept it updated with the stuff I produced in free time. Therefore, I had to download one by one each html file and picture to save it. I was almost done, but a download limit kept me to finish this tiding work. 5-6 hours later I could download the rest, and now the homepage rests silently on my hard drive. Crazy that Yahoo chose to hardwire advertisement directly in my pages, so that also while browsing offline I see it. I am planning to rewrite the homepage anyway... I assume Yahoo is quite in financial trouble, if they had to take this step...

15 Oct 2009 (updated 15 Oct 2009 at 14:56 UTC) »

I like the iGoogle start page. Last week, I was browsing a little bit bored through the Google Gadget API document. Without thinking too much, as usual, I added the Developer Gadget on my start page.

This week, I realised what the "Cached" checkbox on the Developer gadget means. Basically, you need to disable the cache, else while developing a gadget, you will not see any progress :-). At the same time, I also tried to add a predefined ohloh gadget, but that did not work well.

A gadget story

This was the spark in my mind to develop my own ohloh gadget to show my CVS statistics on the Open Source projects I am working. It is here. Okay, nothing special, it is an XML shell with an HTML image and a reference tag. But it works and rewards me.

Another thing to know is that I live in a small valley in the mountains, and on Monday it was snowing. Another gadget was born. It is also an image with a reference tag. The image is from the webcam on the Bernina pass. Just to know when I have to mount snow chains on my car. By the way, I should mount winter tires as soon as possible :-) In the night, this gadget turns black, but it is not a bug as sometimes there are spotlights of cars.

I participate also in some Open Source projects. On Tuesday evening I wrote the GPU Google Gadget showing how many computers are online on the internet cluster. This time I made use of the content "type=url" tag, to link to a php page on my webserver. I changed an existing php page with a .css to have much smaller fonts and added some handy links to the gadget.

With the content "type=url", you are not tighten to XML, HTML and Javascript: you can reference any dynamic content. I like php, but you can do Ruby on Rails or a cgi script in C++ or some .NET asp page, at your wish.

Yesterday evening, with the same "rocket science" I wrote the deltasql Latest Scripts gadget.

And I added also many other gadgets on my start page: one shows my BOINC stats, one stock market information and one currency exchanges.

Gadgets are funny! With the Developer gadget you can go browse any gadget, modify it, publish it on your scrappy box and develop your owns :-)

I bought a Rollin KVM switch with 8 outputs (VGA, mouse, keyboard) for 218 swiss franks (=140 euro). Additionally, 8 cables with VGA, mouse and keyboard are needed (each one costs about 15 euro). The switch is the key component of the newly born...

S.C.R.A.P.

With this KVM switch, I connected one LCD monitor, one keyboad and one mouse to my Gentoo server, to a XP cube bought second hand for less than 200 swiss franks and to an old computer of my father. The switch allows me to use up to 8 computers with one console. Network connectivity is provided by an old Ethernet 10 Mbit/s switch attached to a router with wireless. The 10Mbit/s switch still features the 10T connector for coax on board :-).

Some of you will argue that a VNC server running on each computer would do... Yes, of course, I will also install a VNC server on each computer for when I am working remotely, but when I am at home the KVM switch is much more practical and allows me to setup new computers that normally still do not have a working VNC server on them... Additionally, the KVM switch is robust and stable, it never crashes! By pressing twice the CTRL key, it pops up a menu that allows me to name each computer of the cluster, to cycle through the computers, to go to a computer. However, to change computer, I still find easier to press on a button on the KVM switch.

I also have a Sun Ultrasparc III around, but the Rollin KVM switch is not compatible to Sun hardware. So I will run the Sun as a standalone component of my cluster. Two laptops I have around will connect to the cluster simply via Ethernet. Some applies for the Commodore C64, equipped with an Ethernet cartdrige and the Contiki operating system. The OLPC laptop bought in the G1G1 of December 2007 will connect to the cluster via the wireless router.

My father runs a computer shop and software house, so I think as soon as I have some place I'll extend the cluster up to 8 old boxes (+C64 and +Sun +2 laptops +OLPC = 13 nodes). To find them, it will not be a problem, friends and customers keep substituting their computers even if in principle there is no need...

To have an own supercomputer is a dream of mine since I am a child. Last but not least, I forgot to explain what S.C.R.A.P. means: Super Cluster Ready at Processing.

On the cluster, I'll experiment with GPU. With the cluster, I hope to develop GPU further and to make it more platform independent. Additionally, I'll run some BOINC instances. My favourites BOINC projects are Seti@home and Climateprediction.net.

When running in full the cluster might consume more than 13*300W + 200 W (screen) = 4100W. I need care on attaching the nodes on different phases. I do not plan to run all nodes all the time, power is still not cheap enough for my pockets!

Wow I am so happy, also the server is setup with X and Gnome now under Gentoo.

I used the laptop to help the server compile as the server is only an AMD 700 MHz with distcc.

Gnome distcc monitor did not work for me, I used a simple


tail -f /var/log/syslog | grep distcc
to monitor the laptop in helping with gcc jobs. As a distributed computing fan it warmed my heart looking at two computers helping each other :-)

Ah another thing who almost drove me crazy was a bug when starting X on the server. X worked, but the keyboard and the mouse freezed, the entire computer blocked. Just two xterm on the screen and nothing else... System non-reactive, only pressing power down removed the server from Nirvana. I tried two keyboards (USB and PS/2), two different mouses (one USB and one PS/2), three different video cards (old nVidas, old ATI Radeon, an S3 Virge cannibalized from a Windows 95 computer) and several configurations made by X -configure and xorgcfg...

I was almost giving up and thinking about trashing the <guevara> server by throwing it out of the window of the first floor, when my girlfriend Virgi (virus for the friends) found this parameter for xorg.conf:


Section "ServerFlags"
        Option  "AutoAddDevices" "False"
EndSection

With this option everything worked fine.

Gentoo Linux is running fine on the laptop with Gnome. Now I am setting up also <guevara>, the server. I'd like to get the following services running on the server: distcc, apache2 with php, mysql and also a rsync daemon, so that I can do emerge --sync once on the server and later synchronize directly the laptop with the server :-)

Long live Gentoo!

31 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!