Older blog entries for yeupou (starting at number 134)

Release: SeeYouLater 1.1

I’ve just released SeeYouLater 1.1 (fetch a list of IP or known spammers and to ban them by putting them in /etc/hosts.deny). This is a small cleanup release, now it avoids duplicates in both database and hosts.deny.

You can obtain it on the Gna! project page using SVN or debian packages.


Syndicated 2010-10-08 14:49:33 from # cd /scratch

Being warned of pending packages upgrades with apt-warn

I started using GNU/Linux with RedHat 5.2. It cames with plenty of packages (GNOME 0.20, Linux 2.0.36, etc) and I was quite happy to deal with RPM (RedHat Package Manager, hum) telling me which package is required to install another one, which package contains which files. You simply had to go to RPMFind.net to get missing packages. If no package was available, you could write a clean RPM spec to build one or use checkinstall to build RPMs on the fly when doing make install. It was more than ten years ago, still, nowadays Microsoft Windows XP (sorry, I never used Vista/7) have no clean packaging system that I know of; you have a clumsy list of installed software (InstallShield, whatever it means), no clear idea of dependancies, you can remove pieces of software required by other still installed software and there are plenty of installed pieces of software that you have no way of clearly listing.

At that time, I had a Pentium II 350 MHz and a Pentium 200 MMX as workstations and a Pentium 133 MHz as home server. I, soon enough, had the idea to write a script to produce a list of installed packages readable over intranet and so I published a BASH-based script to output an HTML view of a RPM database called pdbv, standing for Package DataBase View, the first version 1.0.0 being released in June 2002. On the Gna! project page, when listing pros and cons of pdbv, the first pro that came up was “it does not require lucid/gtk+/qt or other big libs”: nowadays, GTK+ and Qt probably no longer strike the mind of anyone as “big (bloated) libraries” and I assume Lucid is no longer even installed on most GNU/Linux systems. Later, I rewrote pdbv in Perl which made if was faster and lighter. Here are demos of pdbv: pdbv 1.x with French locale, pdbv 2.x.

As you can see browing pdbv’s demos, it obviously supports also dpkg (Debian Package, duh). I gradually switched over Debian GNU/Linux for two reasons: apt-get and the branching stable/testing/unstable. Apt-get was the end of wasting time on RPMFind. Debian stable offers astonishing stability for servers while testing/unstable provides brand new desktop software in a timely fashion.

Nowadays, I spend less time dealing with computers and I no longer rely much on pdbv. Due to lack of support (I guess I’m to blame; but KPackage or Synaptic are surely more useful to endusers anyway), it will be removed from Debian at its next stable release (it is still in Debian lenny but no longer in testing). I no longer care much about which software is installed, I use debfoster to keep clean my systems (I know, just like apt-get, debfoster is deprecated in favor of aptitude, but I cannot help using it instead).

However I’d like to know which upgrades are pending. For this reason (and I’m quite sure I’m reinventing something that already exists, but I failed to find it and I wanted it my way), I wrote a small script called apt-warn that will run apt-get update and then warn you of pending updates (only if it has not warned you already about them). It requires Apt::Pkg. It is supposed to be installed a cronjob in /etc/cron.daily. Running on my workstation this morning, it outputs:


Follows 4 newly updated package(s) that you could upgrade on bender:
hicolor-icon-theme (0.11-1 -> 0.12-1)
sudo (1.7.2p7-1 -> 1.7.4p4-2)
xserver-common (2:1.7.7-4 -> 2:1.7.7-6)
xserver-xorg-core (2:1.7.7-4 -> 2:1.7.7-6)

Follows 5 recently updated package(s) that you also could upgrade:
autopoint (0.18.1.1-1 -> 0.18.1.1-2)
gettext (0.18.1.1-1 -> 0.18.1.1-2)
gettext-base (0.18.1.1-1 -> 0.18.1.1-2)
login (1:4.1.4.2-1 -> 1:4.1.4.2+svn3283-1)
passwd (1:4.1.4.2-1 -> 1:4.1.4.2+svn3283-1)

Autopoint, gettext, login and passwd pending upgrades were already warned about yesterday. A second run will return no output since there is no other available upgrade not already warned about.


Syndicated 2010-09-19 10:19:39 from # cd /scratch

Getting MPlayer to cope cleanly with redshift

Redshift is a nice tool that adjusts the color temperature of your screen according to your surroundings. As result, your eyes hurt less if you are working in front of the screen at night.

It is easy to set up, it is for instance already packaged for Debian (package redshift). Once installed, you have to determine longitude and latitude of your position – googling around should do. And you can made some test to defines which range of temperature you want redshift to work with – I like it cold, so I go from 6500 to 9300. And you add it in autostart, the way you want.

In my case, I added redshift -l 48.799:2.505 -t 6500:9300 & just before startkde in my ~/.xsession

Easy, isn’t it? Sure. But when I watch TV with MPlayer or any video with SMPlayer, especially around 01 AM, I’d like color temperature back to normal. And, no, I’m not fond of the idea of doing a killall each time I start watching a video and then a call to redshift afterwards when I’m done.

Configure SMPlayer to use this MPlayer wrapper that kills and starts redshift at the right time

So I wrote a wrapper that send SIGTERM to any redshift process when starting, call MPlayer then, when over, restart redshift. It uses basic perl functions so it has no dependencies. You may however edit it to the set latitude/longitude and temperature range to whatever you like.

It should be just as if you were using MPlayer, so you can configure SMPlayer, or any MPlayer frontend, to use this wrapper. Obviously, this wrapper could be modified to work with vlc, xine or any else video rendering engine.


Syndicated 2010-09-09 14:55:00 from # cd /scratch

Slaying Spams with both Bogofilter and SpamAssassin embedded in exim

Ads are spam. Good thing with the internet’s ads is that you can set up countermeasures.

(Disclaimer: yes, there is nothing new here, just an example of setup)

I have plenty of email addresses from different providers, some are definitely history. I could go through the websites of all of these and set up forwarding for the one I no longer use but still want to be able to get mail from, just in case. Well, I would do that if I was using my mail client to fetch mails – because otherwise fetching mails would actually take ages.

But, as I have a local home underclocked :) server, I find way easier and potent to, instead, use ESR’s fetchmail to download them all to a single account that is accessed by my mail client through IMAPS. I have a /etc/fetchmailrc like:

poll pop.free.fr with proto POP3
user 'XXX' there with password 'XXX' is 'localuser' here
poll imap.gmail.com with proto IMAP
user 'XXX@gmail.com' there with password 'XXX' is 'localuser' here with ssl
user 'XXZ@gmail.com' there with password 'XXZ' is 'localuser' here with ssl

Fetchmail download mails than then relies on the installed SMTP, which is Exim, to deliver it to end user account mailbox accessible through IMAPS.

What’s so nifty nifty about? Well, mails will also be filtered for spam. As it happens on the local home server, it will be unnoticeable for the end user that is me. We’ll use several anti-spam tools, not caring about redundancy and time-consumption: DNSBLs, Bogofilter, SpamAssassin, razor2.

So, here we go. Note that Exim (exim4) in Debian use the user Debian-exim. localuser is the recipient end-user.
We will create a system group dedicated to spamchecking to easily share bayesian databases:

# addgroup --system spamslayer
# adduser Debian-exim spamslayer
# adduser localuser spamslayer

* Bogofilter is a bayesian spam filter . It is said to be faster and lesser time consuming than the SpamAssassin’s own bayesian filter so will run mails through it first. It is installed with the debian package.

Edit /etc/bogofilter.cf as follows:

bogofilter_dir=/var/lib/bogofilter
db_transaction=yes

The bayes directory must be created by hand:

# mkdir /var/lib/bogofilter
# chgrp spamslayer /var/lib/bogofilter
# chmod 2777 /var/lib/bogofilter

* SpamAssassin is a powerful, at the cost of time-consumption, spam-killer. It is installed with the debian package.

In the following site-wide config /etc/spamassassin/local.cf, I use bayesian filters, razor2, several DNSBLs and I adjust some tests according to my needs:

# Save spam messages as a message/rfc822 MIME attachment instead of
# modifying the original message (0: off, 2: use text/plain instead)
#
report_safe 1
# Set which networks or hosts are considered 'trusted' by your mail
# server (i.e. not spammers)
#
trusted_networks 192.168.1.
# Locales
#
# (I only receive mails in English or French)
ok_locales en fr
# Set the threshold at which a message is considered spam (default: 5.0)
#
required_score 3.3
# Use Bayesian classifier (default: 1)
#
# (I created the relevant directory)
use_bayes 1
bayes_file_mode 0777
bayes_path /var/lib/spamassassin-bayes/bayes
score BAYES_20 0.3
score BAYES_40 0.5
score BAYES_50 0.8
score BAYES_60 1
score BAYES_80 2
score BAYES_95 2.5
score BAYES_99 6
# Bayesian classifier auto-learning (default: 1)
#
# (I may change that, not sure about it)
bayes_auto_learn 1
# Set headers which may provide inappropriate cues to the Bayesian
# classifier
#
bayes_ignore_header X-Bogosity
bayes_ignore_header X-Spam-Flag
bayes_ignore_header X-Spam-Status
# use razor
# (/etc/razor is the standard debian path)
use_razor2 1
razor_config /etc/razor/razor-agent.conf
score RAZOR2_CF_RANGE_51_100 3.2
# some rbl checks are already made by exim, at RCPT time, not all.
skip_rbl_checks 0
rbl_timeout 30
score RCVD_IN_SBL 15
score RCVD_IN_XBL 15
score RCVD_IN_SORBS_HTTP 15
score RCVD_IN_SORBS_SOCKS 15
score RCVD_IN_SORBS_MISC 15
score RCVD_IN_SORBS_SMTP 15
score RCVD_IN_SORBS_ZOMBIE 15
# adjust some tests scores: lower DUL test
score FROM_ENDS_IN_NUMS 0.2
score FROM_HAS_MIXED_NUMS 0.2
score FROM_HAS_MIXED_NUMS3 0.2
score RCVD_IN_NJABL_DUL 0.1
score RCVD_IN_SORBS_DUL 0.1
# lower stupid test
score DNS_FROM_SECURITYSAGE 0.0
# adjust some tests scores
score FAKE_HELO_HOTMAIL 3
score FORGED_HOTMAIL_RCVD 3
score HTML_FONT_BIG 2.4
score NO_REAL_NAME 2
score RCVD_IN_BL_SPAMCOP_NET 3
score SUBJ_ILLEGAL_CHARS 4.8
score EXTRA_MPART_TYPE 2.8
score SUBJ_ALL_CAPS 2.6
# increase all scores related to drugs: what do I care, duh
score DRUGS_ANXIETY 5
score DRUGS_ANXIETY_EREC 5
score DRUGS_ANXIETY_OBFU 5
score DRUGS_DIET 5
score DRUGS_DIET_OBFU 5
score DRUGS_ERECTILE 5
score DRUGS_ERECTILE_OBFU 5
score DRUGS_MANYKINDS 10
score DRUGS_MUSCLE 5
score DRUGS_PAIN 5
score DRUGS_PAIN_OBFU 5
score DRUGS_SLEEP 5
score DRUGS_SLEEP_EREC 5
score DRUGS_SMEAR1 5
# same goes for porn
score AMATEUR_PORN 5
score BEST_PORN 5
score DISGUISE_PORN 5
score DISGUISE_PORN_MUNDANE 5
score FREE_PORN 5
score HARDCORE_PORN 5
score LIVE_PORN 5
score PORN_15 5
score PORN_16 5
score PORN_URL_MISC 5
score PORN_URL_SEX 5
score PORN_URL_SLUT 5

The bayes directory must be created:

# mkdir /var/lib/spamassassin-bayes
# chown Debian-exim /var/lib/spamassassin-bayes
# chmod 0777 /var/lib/spamassassin-bayes

Obviously, it implies that razor2 must be properly installed. We install the debian package then set it up. Remember it must run with user Debian-exim, so we do:

# chown -R Debian-exim:spamslayer /etc/razor
# su Debian-exim
$ razor-admin -home=/etc/razor -register
$ razor-admin -home=/etc/razor -create
$ razor-admin -home=/etc/razor -discover

To save ressources, we start SpamAssassin as a daemon (spamd), that will be called using its specific client (spamc). Before using the initd script, edit as follows /etc/defaut/spamassassin:

# Change to one to enable spamd
ENABLED=1
# SpamAssassin uses a preforking model, so be careful! You need to
# make sure --max-children is not set to anything higher than 5,
# unless you know what you're doing.
OPTIONS="--create-prefs --max-children 5 --helper-home-dir -u Debian-exim -g spamslayer"
# Cronjob
# Set to anything but 0 to enable the cron job to automatically update
# spamassassin's rules on a nightly basis
CRON=1

All that being do, you’ll want to (re)start the daemon with the relevant initd script (/etc/init.d/spamassassin restart here).

* Now we’ll tune Exim to call all by himself first Bogofilter and then SpamAssassin, if necessary only. We use splitted configuration in /etc/exim4/conf.d/. That is debian-specific I think but it does make any difference anyway.

First we define useful transports in /etc/exim4/conf.d/transport/35_spamblock (the name 35_spamblock is arbitrary and the number does not matter here):

spamslay_bogofilter:
driver = pipe
command = /usr/sbin/exim4 -oMr spamslayed-bogofilter -bS
use_bsmtp = true
transport_filter = /usr/bin/bogofilter -l -p -e
home_directory = "/tmp"
current_directory = "/tmp"
# must use a privileged user to set $received_protocol
# on the way back in!
user = Debian-exim
group = spamslayer
log_output = true
return_fail_output = true
return_path_add = false
message_prefix =
message_suffix =
#
spamslay_spamd:
driver = pipe
command = /usr/sbin/exim4 -oMr spamslayed-spamd -bS
use_bsmtp = true
transport_filter = /usr/bin/spamc
home_directory = "/tmp"
current_directory = "/tmp"
# must use a privileged user to set $received_protocol
# on the way back in!
user = Debian-exim
group = spamslayer
log_output = true
return_fail_output = true
return_path_add = false
message_prefix =
message_suffix =

Second we define routers, here in /etc/exim4/conf.d/router/350_spamblock – the order matters, here it is just after 300_exim4-config_real_local and before 400_exim4-config_system_aliases:

# first bogofilter
spamslay_router_bogofilter:
# When to scan a message :
# - it isn't already flagged as spam
# - it has not yet been spamslayed at all
condition = "${if and { {!eqi{$h_X-Spam-Flag:}{yes}} {!eq {$received_protocol}{spamslayed-bogofilter}} {!eq {$received_protocol}{spamslayed-spamd}} }}"
driver = accept
transport = spamslay_bogofilter
#
# second spamd
spamslay_router_spamd:
# When to scan a message :
# - it isn't already flagged as spam
# - it has not yet been spamslayed with SA
condition = "${if and { {!eqi{$h_X-Spam-Flag:}{yes}} {!match{$h_X-Bogosity:}{^Yes}} {!eq {$received_protocol}{spamslayed-spamd}} }}"
driver = accept
transport = spamslay_spamd
#
# This route will send any mail that got here to the devnull alias, that
# should be configured in /etc/aliases to be a real link to /dev/null.
# This route should get only mails that have spam score higher than 14.
# This will affect users mails!
spamslay_killit:
condition = "${if ge{$h_X-Spam-Level:}{\*\*\*\*\*\*\*\*\*\*\*\*\*\*} {1}{0} }"
driver = redirect
data = spam
file_transport = address_file
pipe_transport = address_pipe

* Next step, now that spams are flagged, it makes sense to put them apart. I do this with procmail. Here’s the relevant bit /home/localuser/.procmailrc:

IMAPDIR=$HOME/.Maildir/
ISDIR="/"
DOT="."
# tagged by hand, to be learned from by both SpamAssassin and Bogofilter
spam=$IMAPDIR$DOT"Poubelle.Spam"$ISDIR
# by spamd
spamBySA=$IMAPDIR$DOT"Poubelle.SpamSA"$ISDIR
# by bogofilter
spamByBg=$IMAPDIR$DOT"Poubelle.SpamBg"$ISDIR
expirable=$IMAPDIR$DOT"Poubelle.Expirable"$ISDIR
#
:0
* ^X-Spam-Status: Yes
$spamBySA
:0
* ^X-Spam-Flag: YES
$spamBySA
#
:0
* ^X-Bogosity: Yes
$spamByBg

* Training bayesian filters.

Now that spam ended up in a specific mailbox/maildir, both SpamAssassin and Bogofilter bayesians filters can be trained to be effective. We add the following in /etc/cron.d/bayes:

# trains bayesian filters
BASEDIR="/home/localuser/.Maildir"
SPAMDIR_MANUAL="$BASEDIR/.Poubelle.Spam/cur/ $BASEDIR/.Poubelle.Spam/new/ $BASEDIR/.Poubelle.Spam"
SPAMDIR_SA="$BASEDIR/.Poubelle.SpamSA/cur/ $BASEDIR/.Poubelle.SpamSA/new/ $BASEDIR/.Poubelle.SpamSA"
SPAMDIR_BG="$BASEDIR/.Poubelle.SpamBg/cur/ $BASEDIR/.Poubelle.SpamBg/new/ $BASEDIR/.Poubelle.SpamBg"
#
# spamd: can handle easily bogofiltered found spams
25 * * * * localuser /usr/bin/sa-learn --spam $SPAMDIR_MANUAL $SPAMDIR_BG >/dev/null
#
# bogofilter: not sure how it would cope with spamd headers so we'll avoid them
# for now
# (-u was not set as it is discouraged perf-wise in bogofilter's manual)
# Dirty hack to cope with rights issues: running as root - not great
28 * * * * root /usr/bin/bogofilter --register-spam -B $SPAMDIR_MANUAL $SPAMDIR_BG && chown Debian-exim -R /var/lib/bogofilter

Obviously, if you want it to learn from plenty of different users, you’ll have to think of something more elaborate :)
Anyway, regarding plenty of users, it would actually probably wise to think twice about the whole concept of sharing bayesian filters that may not at all be accurate for very differents users.

I’m not very happy with the handling of bogofilter files read/write access, it remains to be cleaned up. Obviously, one alternative would have been to avoid meddling with Exim and to run both bogofilter and spamd via procmail. Sure, it would not have been site-wide setup but for a few users, ~/.procmailrc can be replicated easily. But actually I enjoy messing with Exim, that’s kind of a hobby. I skipped here the part where we call DNSBLs in Exim (working out-of-the-box anyway). And on a production server, with the SMTP wide opened to the web, it is possible to follow this approach just to shut off spammers at SMTP-time -which induces a huge resources gain- and even ban them.


Syndicated 2010-08-13 14:56:03 from # cd /scratch

Underclocking, going backward?

Do you remember back in the days when a Pentium III doing 600GHz was awesome? At that time, when guys at Intel were foretelling that the increase of the processors clock rate will have no end, or at least none that they could possibly envision, you’d see that oath as testimony of the faith in a future of endless possibilities, gaming-wise.

Weirdos...

Later on, Intel went as far as publishing a Pentium 4 which was degraded version of the Pentium III. Less complex, less instructions, it was able to go higher in clock rate than any Pentium III, something 1.8GHz easily. It went on. People even bought laptop with Pentium 4 2.6GHz. And then people start noticing: hey! it’s winter, it’s freezing damn cold outside but I’m not even forced to turn the heat on! Or funnier, hey, why do my brand new laptop is making more noise than a vacuum cleaner? And what black magic made power supplies became a noticeably costly component of a computer?

Well, that’s all about physics. And there’s not much to do about. The faster the computer processing unit run, the more energy it will burn, the hotter it will get.

AMD was smart enough to soon start shipping processors with lower clock rate than Intel ones for the same effective potential. It was also smart enough to brand them accordlingly, branding them for instance something like 3200+ to tell they would be as potent as a Pentium 4 3.2Ghz, while they had a way slower clock rate.

Intel could surely not completely obviously go backward – and publicly recognize AMD wittier. But they could not loose the growing market of the laptops, where the heat issue (not to mention the impact on the batteries life) was too much of a problem with Pentium 4, so they invented the Pentium M… based on the Pentium III, of course.

Considering the unavoidable antagonism beetween fastness and energy consumption, the best idea that someone (who, I do not know) came up with was to enable the operating system to set the clock rate according to the current need. It comes with many different names (Cool & Quiet, whatever) and I believe is it now available with most recent processors. On Debian, you just have to install cpufredutils and load the relevant kernel module cpu (powernow_k8 for instance on my AMD Athlon 64 X2 Dual Core workstation) and then pick a policy. Yes, you have to pick a policy, like on demand, performance, etc.
Obviously, there is a performance loss (hence the name -performance- of the policy which actually only set the clock rate to max) since you are not always running the fastest possible: there is always a delay needed for the operating system to understand that now you need full speed when it was idle just before. The different policies purpose is to optimize this delay – tuning inertia, in which regard on demand is simply harsher than conservative.
Next step would be to have the operating system guessing if you’ll need full speed or not according to what you are actually doing (which software do you run, etc) and what you are about to do according to past usage (yes, logging what you use to do and making guess).

So currently, on a workstation like mine, using cpufreq on demand is probably a wise choice. Most of the time, it will run slower than it could, because you do not need full power of a recent processor to browse over the web, reply to mails and whatever crap like that you may want to do. And when you’re compiling a piece of code, when you are encoding a piece of music, then you’ll have full power. I never or rarely use GNU/Linux to play games so inertia is not a crucial issue – however, to play games, it would surely be best to set the policy to performance, even if after the game is started it will likely, anyway, request full power (surely, you configure your games to the best resolution, anti-aliasing, etc, that your box can handle, don’t you?).

(Not to mention that, gaming-wise, graphical cards now do a big part of the job, the most important anyway, making CPU less important by comparison to so-called GPU… but that’s another story)

So, now, I’ll get straight to the point. I run also a little shuttle box as local server. It serves files, it is up 24h/24 and do plenty of small things. It comes with a Celeron 2.6GHz but it surely would do as well with a slower clock rate. With in mind the idea of reducing the heat of this processor as much possible, I searched over the web on the subject of underclocking. The mainboard of the shuttle, by design, does not allow to make this processor run slower than it does. There is no possibility of playing with cpufreq or alike with a Celeron – which is actually a crappy Pentium (no L2 cache, less instructions, etc).

Pentium 4 2.80GHz running at 1.40GHz

I found however interesting the idea of buying a processor designed to run with a faster front side bus than the actual mainboard we have. It focus on the fact that the processor clock rate is actually determined by both the clock multiplier and the front side bus (FSB).
This shuttle front side bus runs at 400MHz. If I pick a processor, says, designed for a 800MHz front side bus, which is usual of Pentium 4 around 3GHz, it will run twice slower.

So I spent nine euros to get a (used - but the Celeron 2.6Ghz is not brand new either) Pentium 4 2.8GHz. And now, my shuttle runs 1.4GHz. Processor temperature is around 35°C, and the sole fan of the box is around 1300RPM. Nice side effect, this processor got Intel’s Hyper-Threading (simili multiprocessor), which is definitely good for a server.

The only remaining thing to do is to undervolt it now.


Syndicated 2010-08-04 14:58:50 from # cd /scratch

Videos misc scripts: saving space, resetting subtitles

These days, while according to neutral sources movies industry has never been so juicy so it made obviously necessary to restrict freedom in France in the name of its survival, I thought nice to share two small scripts handy when dealing with videos on your harddrive.

Surely, you wouldn’t store videos downloaded over the internet that you haven’t paid for. I guess that’s immoral since in 2009 the US box office top $10bn for the first time in history, during worldwide economic downfall . An advertisement paid by these guys that made these $10bn, while bankrupcy was really an option for major financial institutions and eviction just the same for poor tenants, said it is piracy – while it is still hard for me to envision how it relates to these events occuring on regular basis in Malaysia, Cameroon, Red Sea, etc.

And if you are not concerned by the moral issue (a communist like Jesus Christ, aren’t you?), maybe you are afraid to get caught anyway. Well, it is unlikely that police would come to your house with a search warrant looking for piracy evidence. Mostly because there is no such thing in French law as a search warrant. Indeed, police is entitled to enter in your house in some cases: in three cases only. First, there is what is named commission rogatoire, an order given to a policemen by a judge to do something specific in his name such as search in your house. Looks obviously like an US/UK common law search warrant but it is not: sure it gives the same rights to the police, but it is not an usual procedure in France as it applies only to criminal investigations (information judiciaire), not for trivial misdemeanour/regulatory offences. Second, in the case of a enquête de crime ou délit flagrant (felony or misdemeanour punishable by jail time that just occurred), police can enter your house without your consent. Three, in what is actually the only case that would allow police to enter your house for a regulatory offence (which what this piracy is actually more or less about) is the enquête préliminaire – funny, in this case, police requires your (written) consent to enter. If guess that if I had this kind of piracy evidence at home and the police coming to my doorstep asking to enter in an enquête préliminaire, I would probably not consent.

And I would not even dare to bring the issue of the fine endorsed by this HADOPI law. It is said that, as a friendly reminder of your place you should not have forgotten so easily (customer, yes, that’s you – nothing else – even if you do no harm, it is not up to you to proceed otherwise as you’d wish), by Law, your Internet access will be discarded while you’ll still pay for it. Well, French penal Law states also that “Nul n’est responsable que de son propre fait”, meaning that you can only be punished for your own doings. Sure, there are exceptions (boss that somehow forced employee to misbehave, etc). But none that I think of such as the case of two persons living in the same house and being punished together for the actions of only one of them without the knowledge and consent of the other. By principle, this idea is outlaw, a regression of two thousands years of penal law, dropping us back in the days when you were entitled to take possessions (by force if necessary) of the belongings of a spartian you’ve just met because, as athenian (or whatever), you were recently spoiled by another spartian, no matter that they had no ties aside their citizenship.

1. So there is a script called dir2x264.sh that I wrote for the purpose of saving harddisk space by cleanly converting .avi and .mpg files to an x264-encoded .mp4 file.

It could surely be tuned – I noticed issue when dealing with .mkv files. So far it uses mencoder (mplayer’s encoder) with lavc as audio codec and x264 as video codec. So obviously it requires mencoder, with support of these two (usual) codecs.

To use it, go in the directory where you have avi or mpg files, put the script in there and call it (it will always process all the files in the current `pwd`).

$ cd ~/myvideos
$ chmod +x dir2x264.sh # (if not made executable already)
$ ./dir2x264.sh

It will log work being done in dir2x264log, to easily evaluate the harddisk space saved.

2. In case you cannot find (Have you tried SMPlayer?) the exact correct subtitle file for your video but found one that is just fine except there is a delay between the sound/image and the subtitle, the subtitle_reset.pl could do the trick for you. It depends on Subtitles.pm (libsubtitles-perl in debian).

This script takes two command line arguments: –file and –time (in seconds, positive value or negative), so the usage is quite obvious.
It will make a backup of the original file. If you run it several times to finely adjust your file, it will always restart from the backup file, unless removed obviously.

Note that however this script will not help if the matter is that the video and subtitles file frame-rate differs. You may want to give a try to subs, a script that is now shipped with Subtitles.pm (that was not there or that I missed when I first wrote this one).


Syndicated 2010-07-30 20:08:39 from # cd /scratch

RSS feeds: HTML output with rawdog from akregator’s OPML

Akregator, a KDE RSS aggregator

RSS feeds are probably one of the most useful tools of nowaday’s internet. Obviously, it is not really complicated to find interesting pages over the web. It is way harder to keep up to date, however. These feeds fix that issue. I will not explain what RSS fields are but will focus on how I use them.

On my main workstation, with KDE, I use Akregator that aggregates all the feeds. It is nicely integrated in the enviroment: in Konqueror, with one click, I can add whatever RSS field is mentionned in the headers of a HTML page. After adding RSS fields, I can sort them by categories I defined.

Akregator's feeds in rawdog's HTML output

It happens from time to time that I want to access my RSS fields on another computer over the network or even with my laptop over the web. Here comes rawdog, a “RSS Aggregator Without Delusions Of Grandeur”. I picked it because it is easy to set up and lightweight (unlike TinyRSS etc). This aggregator is installed on my local network server and uses akregator list of feeds and produce a multicolumn HTML output that apache serves.

First, on my main workstation (on which one I use Akregator): I set up a cronjob that copies Akgregator’s feeds list to my user account on the server. Note that I use SSH with a key with no passphrase to do so.

/etc/cron.d/rawdog:

25 * * * * user if [ -e ~/.kde/share/apps/akregator/data/feeds.opml ]; then scp ~/.kde/share/apps/akregator/data/feeds.opml server:~/ 1> /dev/null ; fi

(It is nice that Akgregator use the OPML format and not a specific config file)

Next, on server side, on which one rawdog has been installed (nothing specific here, it is shipped by Debian), I created a rawdog user account then made a symlink from /home/rawdog to /var/www/rss.

We need first to provides rawdog with the Akregator’s OPML – it does not support it. To do this, I fetched a perl scripted made by Tero Karvinen that I edited so it support categories. It results in the following opml_to_rawdog.pl stored in /home/rawdog/scripts (/var/www/rawdog/scripts directory access over http being forbidden by Apache).

We set a cronjob to produce a feeds list that rawdog can handle, /etc/cron.d/rawdog:

# Make sure we have the latest feeds, imported from akregator
# if there is a user/feeds.opml, compare it with the current one
30 * * * * root cd /home/rawdog && if [ -e /home/user/feeds.opml ]; then if [ ! -e feeds.opml ] || [ "`diff /home/user/feeds.opml feeds.opml`" != "" ]; then scripts/opml_to_rawdog.pl /home/user/feeds.opml > feeds ; fi ; fi

I put a rawdog config file in /home/rawdog and I edited it to suit my needs. Most notably, I edited as follow /home/rawdog/config:

maxarticles 50
datetimeformat %d %B, %Hh%M
template templates/page
itemtemplate templates/item
outputfile index.html
showfeeds false

feeddefaults
killtags true
truncate 220

# this is the file generated by opml_to_rawdog.pl
include feeds

It relies on two templates pages stored in templates/ directory: templates/page and templates/item. Not surprisingly, the layout is based on a CSS file called style.css – you will have to edit it to match categories names. It also requires the truncate plugin, to be stored in the plugins directory.

The last step is to update /etc/cron.d/rawdog to actually generate the HTML output:

# Run every 9 minutes
*/9 * * * * rawdog cd ~/ && /usr/bin/rawdog -d ~/ -u -w 2>> errors

# once per month, clean up the errors list
10 2 2 * * rawdog rm -f ~/errors

That’s all folks (even if there is room for improvement)!


Syndicated 2010-04-07 22:37:54 from # cd /scratch

Hardware support: Radeon HD 4870 with Xorg

Hardware support is a big deal. When you buy a piece of junk, you expect it to work. And when this piece of junk is required to run software, you need it to work. Nowadays, there is no salvation without a graphics card, not only for games but even for desktop apps that are designed with composite-effects supports in mind etc.

Apple always targetted wealthy customers by providing rock-solid software designed to run on rock-solid hardware. It is not quite a challenge, actually, to provide unbuggy software when you only support your own hardware (and at the same time when you claim one cannot -legally- run MacOS X on anything else than an Apple).

It is way harder to support plenty of hardware, including that you have no clue about the internals design. But that is what provides freedom to the user: freedom to change piece of hardware whenever he wants, freedom to select which piece he wants – not the kind of freedom you enjoy at AppleStore where you have only two costy options of harddisks. Apple never thought of democratizing their stuff, that’s not their business plan.

Still, hardware support is probably what’s harming Libre Software OS most. The issue is still the same over the years, hardware producers count on Libre Software community to provide drivers for Libre Software environment -at no point they deliver by themselves Libre and quality drivers-, which implies Libre Software is lagging to provide efficient specific hardware support.

I experienced it this morning. Yesterday, I upgraded my workstation. I was using xserver-xorg-video-radeonhd drivers until today with the Radeon HD 4870 (RV770) I have here. Next time I booted the workstation, I got a blank screen when Xorg started. At some point, I found out it had to do with firmware-linux-nonfree, I filled a bug and was advised to use xserver-xorg-video-radeon instead of xserver-xorg-video-radeonhd. I did that and now it works (with xserver-xorg-video-radeon 6.12.192-2: DRI/hardware acceleration, composite effects, etc), with the latest kernel available (linux 2.6.32-4-amd64).

Here’s the content of my /etc/X11/xorg.conf that works:

Section "InputDevice"
Identifier "Generic Keyboard"
Driver "kbd"
Option "AutoRepeat" "500 30"
Option "XkbRules" "xorg"
Option "XkbModel" "pc105"
Option "XkbLayout" "fr"
EndSection

Section "InputDevice"
Identifier "Configured Mouse"
Driver "mouse"
Option "Protocol" "IMPS/2"
Option "Device" "/dev/input/mice"
Option "ZAxisMapping" "4 5"
EndSection

Section "Device"
Identifier "Sapphire RADEON HD 4870"
Driver "radeon"
Option "DRI" "on"
Option "VideoOverlay" "on"
Option "EnablePageFlip" "on"
Option "DDCMode" "on"
EndSection

Section "Monitor"
Identifier "LG L194WT"
Option "DPMS"
EndSection

Section "Screen"
Identifier "Default Screen"
Device "Sapphire RADEON HD 4870"
Monitor "LG L194WT"
DefaultDepth 24
EndSection

Section "Extensions"
Option "Composite" "Enable"
EndSection

It may seems easy-going, nonetheless, it took my hours to find out the exact cause of the problem, googling about, doing tests by reverting my system to previously installed package, etc.

But that’s still easier than coping with issues like one can suffer on a proprietary operating system, with no way to find out what exactly changed on this computer during the latest upgrade. For the record, I have no way to access the ATI graphics card configuration tool (called Catalyst Control Center, CCC.exe) on Microsoft Windows XP since its latest upgrade. But I won’t even bother to try to work out the undecipherable, I’ll just wait passively for a new CCC.exe release.


Syndicated 2010-03-24 14:45:50 from # cd /scratch

Counter-Strike/Day Of Defeat userconfig.cfg and the neverending quest for a Libre Software business model regarding games

Baby can come along too, but I'm getting drunk without or without you! - Gettin' Drunk by the Beat Farmers

I played to most of the famous first-person shooter (FPS) of the 1990’s and some of the 2000’s: Wolfenstein 3D, Doom, Doom II, Duke 3D, Quake, Half-Life, Medal of Honor, even the less famous like Redneck Rampage (shipped with a wonderful country/rock-a-billy/psychobilly soundtrack; still not kidding, I would even consider going to Reverend Horton Heat gig that will take place in France in a few days if I was rich, which I’m not) etc… But I played regularly for years only to a few of them.

Duke 3D was unmistakenly fun and definitely made the best out of numerous hours of LAN games. Counter-Strike’s (CS) gameplay, however, remains unique. Obviously, the rendering engine was technically modern enough (Duke 3D is fun, but you do not need to aim target in 3D, you can destroy a target up in the air while aiming the boardwalk ; in Doom II, you cannot even jump). But that’s not just it, no, the game really leads to team play. The two teams, counter-terrorists or terrorists, both have an obvious objective (blowing up a bomb, freeing hostages, etc) and each teammate that goes down is out of the game for the round. So unlike in Medal of Honor, Quake Arena, etc (where one player does not equal one another), any player matters. To some players, this a drawback, they rather have unlimited lives, moving fast, dying fast, respawning fast, moving fast, dying still fast. To me, it is part of what makes this game aside from the others FPS. Based of the same rendering engine (the one of Half-Life), Day of Defeat (DoD) and Team Fortress (TF) both provides some kind of team play also well thought but way more conventional (capture the flag, mainly), with respawn. Day of Defeat takes place during World War II while Team Fortress is kind of cartonish/fun, somehow reminding me of Duke3D.

Counter-Strike with Source engine, still not outdated


Half-Life² was eventually published with a new rendering engine called Source. And it is a good thing they also released CS, DoD and TF renewed with this new engine, so these good games kept up with the trend. Despite the fact that the map oilrig (a mode where you had to protect a somekind of VIP) disappeared during the transition to Source, CS gameplay was unaffected. And I continued playing from time to time to CS. I still do.

Today, I found out that I no longer had my userconfig.cfg. In CS, you need to buy weapons at the begin of each round. And, clearly, when a round begins, you have no time to fool around, wasting precious seconds by looking at each possible weapon. You have to take the best you can get in the fastest way possible, which meaning having shorcuts. Such shortcuts are not provided, so you have to write them in a userconfig.cfg file. As, unfortunately, we are not speaking of Libre Software here, everything is really poorly documented, you have to google around. So here comes my cstrike userconfig.cfg:

// ---------------------------------------------------------------
// shopping: commands

// gun
alias buy_gun "buy p228; buy secammo; buy vesthelm;"
// shotgun
alias buy_shotgun "buy xm1014; buy primammo; buy vesthelm;"
// cheapeast decent
alias buy_cheapest "buy mp5; buy primammo; buy vesthelm;"
// assault rifles
alias buy_cheap "buy galil; buy famas; buy primammo; buy vesthelm;"
alias buy_expensive "buy ak47; buy m4a1; buy primammo; buy vesthelm;"
alias buy_scope "buy sg552; buy aug; buy primammo; buy vesthelm;"
// equipment
alias buy_protection "buy vesthelm; buy vest"
alias buy_fb "buy flashbang; buy flashbang"
alias buy_he "buy hegrenade"
alias buy_smoke "buy smokegrenade"
alias buy_defuser "buy defuser"
alias buy_ammo "buy primammo; buy primammo; buy secammo"
alias buy_equipment1 "buy_protection; buy_defuser ; buy_ammo"
alias buy_equipment2 "buy_protection; buy_fb; buy_smoke; buy_he"

// shopping: binds

bind "F1" "buy_equipment1"
bind "F2" "buy_equipment2"
bind "F3" "buy_gun"
bind "F4" "buy_shotgun"
bind "F5" "buy_cheapest"
bind "F6" "buy_cheap"
bind "F7" "buy_expensive"
bind "F8" "buy_scope"

// ---------------------------------------------------------------
// communication: commands
// unfortunately, we have no variable to say exactly where we are
// so it is meaningless to provide much details

alias waitalot "wait;wait;wait;wait;wait;wait;wait;wait;wait;wait;wait;"
alias w "waitalot;waitalot;waitalot;waitalot;waitalot;waitalot;"
alias msg_enemies_none "radio3; w; slot4"
alias msg_enemies_few "radio3; w; slot2"
alias msg_in_a "say_team (A !!!)"
alias msg_in_b "say_team (B !!!)"
alias msg_bomb_here "radio3; w; slot3; w; say_team The bomb is here!"
alias msg_yes "radio3; w; slot1"
alias msg_no "radio3; w; slot8"
alias msg_follow_me "radio1; w; slot5"

// communication: binds

bind "KP_END" "msg_enemies_none"
bind "KP_DOWNARROW" "msg_enemies_few"
bind "KP_PGDN" "msg_enemies_plenty"
bind "KP_LEFTARROW" "msg_in_a"
bind "KP_5" "msg_in_b"
bind "KP_RIGHTARROW" "msg_bomb_here"
bind "KP_HOME" "msg_yes"
bind "KP_UPARROW" "msg_no"
bind "KP_PGUP" "msg_follow_me"

// EOF

In Day of Defeat, the userconfig.cfg matters less, as you do not have to buy weapons. However, the communication shortcuts can be of use, especially as in DoD:S, unlike in CS:S, you have access to the variable providing your location on the map. Here comes my dod usercfg.cfg making use of the location %l variable:

// ---------------------------------------------------------------
// communication: commands
// at the contrary of CS:S, we have the location
// variable (%l) that enables us to
// provide detailed info

alias msg_enemies_none "voice_areaclear; say_team %l clear"
alias msg_enemies_ahead "voice_enemyahead; say_team Enemy ahead %l"
alias msg_enemies_behind "voice_enemybehind; say_team Enemy behind us at %l"
alias msg_enemies_left "voice_fireleft; say_team To my left at %l"
alias msg_enemies_right "voice_fireright; say_team To my right at %l"
alias msg_enemies_machinegun "voice_mgahead; say_team MG at %l"
alias msg_enemies_sniper "voice_sniper; say_team Sniper at %l"
alias msg_enemies_grenade "voice_grenade"
// do not use voice_backup here because it may provide to much insight to the other team
alias msg_needhelp "say_team I NEED BACKUP AT %l"
alias msg_comeon "voice_gogogo"

// communication: binds

bind "KP_END" "msg_enemies_left"
bind "KP_DOWNARROW" "msg_enemies_ahead"
bind "KP_PGDN" "msg_enemies_right"
bind "KP_DEL" "msg_enemies_behind"
bind "KP_LEFTARROW" "msg_enemies_machinegun"
bind "KP_5" "msg_enemies_sniper"
bind "KP_RIGHTARROW" "msg_enemies_grenade"
bind "KP_HOME" "msg_needhelp"
bind "KP_UPARROW" "msg_comeon"
bind "KP_PGUP" "msg_enemies_none"

// EOF

And this dichotomy between CS:S and DoD:S, both based on the same rendering engine, definitely illustrates how frustrating proprietary software can be. If %l exists in DoD:S, it cannot be complicated to implement in CS:S. It is not really an issue of time, if anybody was able to read the code, edit it and redistribute the modifications made, there would have been an user willing to spend this time. I know I would.
But it is not and so we just have to rely on CS:S publisher that does not show much interest in developing further CS:S.

eduke3d and STALKER COP: lights, shadows, objects, textures, no room for comparison

Clearly, there is no recent and great piece of FPS that is Libre Software that I know of. We just get old rendering engine GPLed: Ken Silverman’s Build (Duke 3D/Shadow Warrior/Redneck Rampage/etc engine), Id Software’s id Tech 3 (Quake III, MOHAA, Call of Duty/etc). Even if it is nice that this code is freed at last, I have the sad feeling that they would not be freed if their authors felt their code could be someday improved enough to compete with their own new rendering engine. So no business can be based on these rendering engines, and we’re stuck with Libre Software looking like cheap obsolete products. It may sound harsh, don’t get me wrong: I enjoy the work being done on eduke32, that’s nice, and I’m not criticizing the efforts made. Still, it cannot compete with what is being published as proprietary sofware. Even games with somehow outdated rendering engines like STALKER Call of Pripyat are way ahead.
When Half-Life² was published, there was hope: their business model seemed to implies releasing frequently data (episodes – levels, including artwork). They could have freed their engine. But they didn’t and I guess they just failed to see any profit in doing so – well, I could say that now CS:S would handle %l :)

I’m wondering if we’ll ever see brand new games published as Libre Software. The only people that have the cash flow and the motives to do so could be ATI or NDVIDIA, the hardware producers. But they may has well just continue to work as they currently do.

(PS: and I’m not even going as far as expecting these to run under a GNU-based operating system)


Syndicated 2010-03-19 03:59:34 from # cd /scratch

Securing and improving internet services, including SSH and SMTP, using xinetd

As stated by its manpage, xinetd performs the same function as inetd: it starts programs that provide Internet services. Instead of having such servers started at system initialization time, and be dormant until a connection request arrives, xinetd is the only daemon process started and it listens on all service ports for the services listed in its configuration file. When a request comes in, xinetd starts the appropriate server. Because of the way it operates, xinetd (as well as inetd) is also referred to as a super-server.

The X in the name stands for extended. Which means the following is really for xinetd, not openbsd-inetd :)

Still according to its manpage, so far, the only reason for the existence of a super-server was to conserve system resources by avoiding to fork a lot of processes. While fulfilling this function, xinetd takes advantage of the idea of a super-server to provide features such as access control and logging. Some people will say, and they’ll be right, that running all services through a wrapper implies, instead of conserving resources, somekind of overhead: in conserves resources since it avoids running concurrently as many services as available on the server, right, but this makes no sense on a server with a wide audience that actually have enough users so all services are anyway almost always running concurrently. In this case, some people would be correct to assume more efficient to use standalone servers for each service .

But, and that’s the point, access control makes a difference. Sure, standalone servers have also access control. OpenSSH does ; and using OpenSSH via xinetd should not discourage to look into /etc/ssh/sshd_config. Nonetheless, xinetd access control applies to any service running through it. And that’s prett-ay, pretty good. For instance, you do not need to configure each standalone server to be hardened enough against DDoS, if xinetd is, you should be fine.

So let’s get to business. We assume here you have xinetd up an running. Shouldn’t be a big deal, xinetd is standard on many GNU/Linux systems.

Normally, you should have a /etc/xinetd.d where you can add bits of config for xinetd (if it does not exists, well, you could still use the default config file /etc/xinetd.conf).

There you have basic standard stuff: chargen, daytime, discard, echo, time. If you do not want to provide these, sure make sure each entry in these files got the line:

disable = yes

For each following example to work, you must indeed shut down the standalone server, otherwise the service port won’t be available to xinetd. Also, in the following examples, you’ll have to edit the IPs according to your network.

This is for OpenSSH, with a specific port for root login (probably a nuisance a distant server supposed to be frequently accessed as root – but a safe pick for a local network server rarely accessed as root from the web):

# To work, sshd must not run by itself,
# so /etc/ssh/sshd_not_to_be_run
# should exists

# allows unrestricted SSH only to local network
service ssh
{
socket_type = stream
protocol = tcp
wait = no
user = root
bind = 192.168.0.1
only_from = 192.168.0.0
server = /usr/sbin/sshd
server_args = -i
}

# allows SSH from the web but restricted to users listed
# (root being forcefully disallowed)
# restrict also to only 5 connections per IP (per-source)
# and limit the rate of incoming connections (cps)
service ssh
{
socket_type = stream
protocol = tcp
port = 22
wait = no
user = root
bind = 88.???.???.???
server = /usr/sbin/sshd
server_args = -i -o PermitRootLogin=no -o AllowUsers=thisuser
cps = 30 10
per_source = 5
log_on_success = HOST USERID
}

# allow SSH from the web only for root, on port 33333
# requires /etc/services to include lines:
# rootexternalssh 33333/tcp
# rootexternalssh 33333/udp
service rootexternalssh
{
socket_type = stream
protocol = tcp
port = 33333
wait = no
user = root
server = /usr/sbin/sshd
server_args = -i -p 33333 -o AllowUsers=root
cps = 30 10
per_source = 3
log_on_success = HOST USERID
}

This is for Dovecot, an IMAPS server. This setup listen on the local network. You can easily tune it following the example given above:


service imaps
{
socket_type = stream
protocol = tcp
wait = no
user = root
bind = 192.168.0.1 127.0.0.1
only_from = 192.168.0.0 127.0.0.1
server = /usr/lib/dovecot/imap-login
flags = IPv4
server_args = --ssl
}

I won’t provide an exhaustive list of services that you can run with xinetd. You can surely find for yourself what suits you best! :) But the presentation wouldn’t be complete if I missed traps. Yes, you can set up traps with xinetd. For instance if you do not use ftp, irc, telnetd, etc, you can safely assume that someone trying to connect on these services ports is trying to do something he shouldn’t. And you can then decide to disallow further connections.


# bind must be set so we do not shut off clients from the
# local network that made dumb scan
service ftp
{
socket_type = stream
wait = no
user = root
bind = 88.???.???.???
flags = SENSOR
type = INTERNAL
log_on_success = HOST PID
deny_time = 1440
}

service sftp
{
socket_type = stream
wait = no
user = root
bind = 88.???.???.???
flags = SENSOR
type = INTERNAL
log_on_success = HOST PID
deny_time = 1440
}

service irc
{
socket_type = stream
wait = no
user = root
bind = 88.???.???.???
flags = SENSOR
type = INTERNAL
log_on_success = HOST PID
deny_time = 1440
}

service telnet
{
socket_type = stream
wait = no
user = root
bind = 88.???.???.???
flags = SENSOR
type = INTERNAL
log_on_success = HOST PID
deny_time = 1440
}

These are basic examples. You can do more.

For instance, I wrote SeeYouLater, denying access to spammers with hosts.deny, a production-ready perl software that looks in the SMTP daemon logs for identified spam sources IPs and then ban them via /etc/hosts.deny (which xinetd handles).
With the SMTP daemon run through xinetd, any identified spam source will no longer even be able to connect (which prevents plenty of advantages).

I’d like to mention that I ran xinetd on Gna! main servers (for mail with exim, or for CVS/SVN/Arch, etc – mail server was moved and I did not follow the way it is set up) and the overhead mentioned above was unnoticeable while the number of connections per minutes was quite higher than what you would expect on a small business network server or whatever.


Syndicated 2010-03-14 21:02:05 from # cd /scratch

125 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!