Advogato blog for aleix

Handling external dependencies with Mercurial

Mon, 17 Mar 2008 19:09:41 GMT

In order to add external repository dependencies in Mercurial, I have created an extension similar to the multi-project management feature of GNU arch. This can be probably done with other existent extensions, but I have not been able to found one that exactly met my needs (the Forest extension can be used for non-versioned Mercurial dependencies, and, in fact, they could be used together, but it is not exactly the same). So please, if anyone knows similar extensions, I would happy to know it. My friend Lluís gave me the idea to also handle non-Mercurial dependencies, so thanks from here!

To download it just type:

hg clone http://hg.hacks-galore.org/aleix/hgdeps/

You can also browse it online.

The extension consists of two commands: deps and depsclone. Below, I am pasting the help of the deps command.

This extension is useful when a repository might depend on some external dependencies: other repositories (Mercurial, CVS, Subversion…), files not handled by a SCM tool or some other dependencies.

To enable this extension, add this to your global .hgrc file (or to your repository .hg/hgrc file):

[extensions]
hgext.deps =
# or, if deps.py is not in the hgext dir:
# deps = /path/to/deps.py

In order to define these external dependencies, you need to add a section [deps] in your repository .hg/hgrc file (or in the global .hgrc) where you will define how to find them. As an example, let’s say that our repository depends on two external libraries managed by different SCM tools: ‘libfoo’ managed by Mercurial and ‘libbar’ managed by CVS.

[deps]
aliases = libfoo, libbar
alias.libfoo = /path/to/libfoo
alias.libbar = :pserver:anonymous@cvs.server.org:/sources/bar
alias.libbar.command = cvs -z3 -d$source co -r $rev -d $dest bar

Two aliases have been defined, one for each library, and also the sources where the libraries can be obtained. ‘libbar’ has two options: the library location and the CVS command to use in order to get the library. For ‘libfoo’ only a path to the Mercurial repository has been specified. $rev, $source, and $dest are substitution placeholders and will be explained later.

Once the location of the depedencies have been defined, dependency lists can be created. A depedency list has an associated name. Following the example above, two depedency lists will be created, one for version 0.9.1 of our repository and another for version 1.0. 0.9.1 and 1.0 will be used as the name of the lists.

[1.0]
f24139319bdb    libfoo    lib/foo
v0r8_0          libbar    lib/bar

[0.9.1]
3a9b061bada1    libfoo    lib/foo
v0r7_8          libbar    lib/bar

A dependency is defined by three fields: a revision name, the alias of the external dependency and a destination directory. The alias must be one of the aliases defined in the [deps] section (see above). The revision name will be substituted for the placeholder $rev also seen before. For Mercurial, the revision name will just be a Mercurial revision of the external repository. Finally, the destination directory will be substituted for $dest.

To facilitate version control, distribution, and merging of dependencies, they are stored as a file named ‘.hgdeps’ which is managed similarly to other project files and can be hand-edited if necessary.

From GNU Arch to Mercurial

Thu, 31 Jan 2008 14:09:26 GMT

Over the last two and a half years we (at work) have been using baz as our distributed version control system. Baz is based on tla, the original GNU arch implementation by Tom Lord. After all this time we have applied more than two thousand patches (distributed in various projects). We also have been using Xtla (a wonderful Emacs front-end to GNU Arch) and lately DVC (its successor, which handles GNU Arch, Bazaar, git, Mercurial and Monotone). After some months thinking about it, I decided that it was time for a change, baz was too slow working with this magnitude of patches and even slower in my PowerBook. I also wanted a fast web interface and ArchZoom was not an option as it took ages to show changes between two revisions and mostly in any operation. So, I watched the Mercurial Google TechTalk and I also read some blog entries about git and Bazaar. I decided to give Mercurial a try as it also had complete DVC support and a great web interface (like the darcs one).

One of the things I needed was a tool to convert our baz repositories to Mercurial. As far as I knew, there was no such a tool, so I thought I could use arch2darcs to convert the repositories to darcs and then use the Mercurial convert extension to convert them to Mercurial. I tried it but it didn’t work. The Mercurial convert extension worked really well with darcs for my personal projects, so, why not write a GNU Arch converter?

After some days trying it, I have posted an issue to Mercurial BTS with a GNU Arch converter. It currently does not support directory renaming, but all other things I have tried work well. The converter is slow as I didn’t know how to compute a complete changeset between patches easily (any help?). The log files might not provide all the information needed (added, removed, modified files), so I decided to use the delta command which computes all this information, and this takes a bit of time.

To use the converter (in 0.9.5) copy it to the convert extension directory (hgext/convert) and edit the __init__.py file following, for example, the darcs converter (you’ll easily see how to add it). Use the converter at your own risk, as I am sure it needs more work. I would appreciate any comments and bug reports until it is included in Mercurial (if ever).

Baz 1.4.2 in Fink

Thu, 3 Jan 2008 14:07:14 GMT

Yes, Baz is deprecated, but we are still using it at work, so to make my life easier I have created the Fink package for the latest version (probably there won’t be more). Download this file and, as usual, type:

$ tar zxvf baz-1.4.2-fink.tar.gz
$ sudo cp baz-1.4.2-fink/*
          /sw/fink/dists/local/main/finkinfo/
$ fink index; fink rebuild baz; fink install baz

Baz depends on some cryptographic libraries (gpgme, libgcrypt and libgpg-error). The updated packages are also included (until updated in Fink) in the provided file.

Please, note that when the official Baz Fink package is updated you will have to remove your local copy (from /sw/fink/dists/local/main/finkinfo/).

ECL 0.9j in Fink

Thu, 3 Jan 2008 08:05:54 GMT

ECL has a new release, and thus a new Fink package update. It is still under validation, so until it is committed you can use this file. To install it, just type:

$ tar zxvf ecl-0.9j-fink.tar.gz
$ sudo cp ecl-0.9j-fink/*
          /sw/fink/dists/local/main/finkinfo/
$ fink index; fink rebuild ecl; fink install ecl

Please, note that when the official ECL Fink package is updated you will have to remove your local copy (from /sw/fink/dists/local/main/finkinfo/).

Screen Sharing with Mac OS X and GNU/Linux

Wed, 12 Dec 2007 11:16:30 GMT

Today, I discovered how to share my Mac OS X (Leopard) screen with another non-Mac OS X computer. Not a big deal, as it is almost automatic, but I’m posting it here as I did a search and I didn’t find how to do it.

What you need to do in your Mac OS X box is enabling the Screen Sharing service from the Sharing options in the System Preferences.

Before establishing the connection you need to setup a password. This is done in the “Computer settings…” button, just like this:

Do not avoid the password step, if you do it the connection will not be established.

Finally, you only need a VNC client in your GNU/Linux box. Not all the clients work, one that it does is the TightVNC viewer. In Debian, you can just type:

apt-get install xtightvncviewer

Once installed, you only need to run it and supply the IP address of your Mac OS X box and the password you configured.

Enjoy it!

Erlang R12B-0 in Fink

Tue, 11 Dec 2007 12:09:30 GMT

Erlang has a new release, and thus a new Fink package update. It is still under validation, so until it is committed you can use this file. To install it, just type:

$ tar zxvf erlang-otp-R12B-0-fink.tar.gz
$ sudo cp erlang-otp-R12B-0-fink/*
          /sw/fink/dists/unstable/main/finkinfo/languages
$ fink index; fink rebuild erlang-otp; fink install erlang-otp

Building applications for ERC32

Tue, 2 Oct 2007 20:09:46 GMT

A while ago, I answered a message, in the RTEMS users mailing list, about how to build raw applications for the ERC32 architecture. With raw applications I mean applications that are not dependent of RTEMS, which is the usual RTOS for this architecture. I am now writing about it just to make it easier for people to find it and to add some information to the explanation.

Before starting with the process, you need to get and build the RTEMS toolchain in order to compile your future applications (I used RTEMS 4.6.1). This is because the specific toolchain for ERC32 non-RTEMS applications is no longer maintained. Note, that you don’t need to install RTEMS itself, but you will need the source code. This video shows, step by step, how to download and build these tools for the i386 BSP, which should be almost identical for ERC32.

I’m not sure if what I’m going to explain is the best way to do it, but it has worked for me, and I have ran programs generated this way in a real board for a long time. I also think this may be useful for other architectures as well.

1. You need to get the following source files from the ERC32 BSP and copy them to your application’s directory:

    cpukit/score/cpu/sparc/asm.h
    cpukit/score/cpu/sparc/rtems/score/sparc.h
    cpukit/score/cpu/sparc/rtems/score/cpu.h
    c/src/lib/libbsp/sparc/shared/start.S
    c/src/lib/libcpu/sparc/reg_win/window.S
    c/src/lib/libcpu/sparc/syscall/syscall.S

2. Modify start.S so it does not call any RTEMS code, that is, commenting these lines:

/*
    call    __bsp_board_init
    nop

    set     (SYM(rdb_start)), %g6   ! End of work-space area
    st      %sp, [%g6]

    set     (SYM(Configuration)), %l1
    ld      [%l1+4], %l3            ! work_space_size
    sub     %sp, %l3, %sp           ! set %sp to area below work_space
    andn    %sp, 0x0f, %sp          ! align stack on 16-byte boundary
*/

3. Compile start.S, window.S and syscall.S. You can remove many things from the header files, but that’s up to you.

sparc-rtems4.6.1-gcc -DASM -c -o start.o start.S

4. Compile your C files (e.g. test.c).

sparc-rtems4.6.1-gcc -O4 -Wall -mcpu=cypress -c -o test.o test.c

5. Link your program. It will not work right now, as you need to generate two final files, my_bsp_specs and linkcmds.

sparc-rtems4.6.1-gcc -mcpu=cypress -Betc -specs my_bsp_specs -o test
window.o syscall.o test.o

Note the arguments -B and -specs. -specs is used to specify a file with options to override the default switches passed to cc1, cc1plus, as, ld… By default, RTEMS uses the file bsp_specs (found in c/src/lib/libbsp/sparc/erc32). You can edit it and see that it automaticaly links with the RTEMS libraries and adds its own start.S object file. The -B flag specifies where to find the executables, libraries, include files, and data files of the compiler itself. I have used an etc directory where I have stored my_bsp_specs. If any of the needed files is not found by the compiler it will automatically search its own paths.

So, what we need to do is to create our own file, my_bsp_specs, which will not use any RTEMS library and which will point to our compiled start.S object file.

Basically,

*endfile:
crtend.o%s crtn.o%s

*lib:
-T linkcmds

*startfile:
start.o crti.o%s crtbegin.o%s

*link:
-dc -dp -N -e start

The %s indicates to search the file in the compiler specific directories. I have removed the %s from start.o so it will search for it in the current directory.

Another important thing to note in this file is the -T linkcmds option. This indicates the linker script, which describes where the code and data for the application will reside in memory, to be used by the GNU linker. Fortunately, we can use the one that comes with RTEMS (found in c/src/lib/libbsp/sparc/erc32/startup), so copy it to the etc directory.

Finally, note that when linking, we have not specified the start.o file as it is already done in the my_bsp_specs file.

You might also find more information in the RTEMS BSP and Device Driver Development Guide.

Code size optimizations

Fri, 28 Sep 2007 09:08:14 GMT

During the last two years I have had the opportunity to write embedded software for the SPARC ERC32 architecture. The first piece of software we have written is an application that needs to fit in a 64 Kbytes PROM with a minimal running system that can execute periodic tasks, send packets over a network, load another application stored in an EEPROM that can be remotely patched and many other things. Before starting to write it from scratch we tried to fit RTEMS (a Real-Time Operating System) in the 64 Kbytes. We achieved a running RTEMS in 55 Kbytes, but there was only 10 Kbytes left (note that modern versions of RTEMS are smaller).

The following tips are the ones I have found useful to decrease my application size (they might not work in all architectures). Note that all the tips are C oriented and that gcc has been used.

I would really appreciate if you can share your tricks and I will be glad to include them in the list.

Compress application code

As the application is stored in a PROM, you will normally have a minimal code to perform some initializations and checks (e.g. RAM tests), and the rest of the code for the application itself. The main application will run from RAM, so you could compress the application using a simple compression algorithm such as LZSS, uncompress the application to RAM and run it. The decompression algorithm will be obviously uncompressed.

If you are working with an ERC32 processor check the MKPROM-ERC32.

Use inline functions judiciously

As it is well-known, performance issues should not be treated while developing. So, writing a lot of inline functions might not have great performance effects, but could have big size penalties if you don’t use them judiciously.

Compiler optimization flags

If performance is not important in your application (i.e. you have no hard real-time requirement), you can use the size optimization options provided by your compiler. For example, in gcc, you can use the -Os option to decrease the executable size. We reduced 5 Kbytes.

Note, that this will avoid inline functions optimization, but you can still tell gcc to include them passing the -finline-functions option.

Split functions into separate files

If you are developing different applications that might share most of the code, you may think on writing an static library. The point is that having a reusable library might affect the executable size. This is because, probably, not all the functions compiled will be used in all the applications. When gcc compiles a source file, all the functions in it will be linked in the final binary even if your code does not use them. Of course, if non of the functions of the source file is being used, they will not be linked. Splitting functions into separate files might solve the problem.

There is one way to avoid this (I haven’t tried it) with gcc using the -ffunction-sections option. This creates a separate section for every function. The linker can then use the –gc-sections option to discard unused sections.

Separate debug and release code

While writing embedded software you may need to have debug and release code. Clear examples are the printing or logging functions that might not be necessary in production code. If this is the case, separate the debug and release code. In C, you could have a serial port printing function in debug and an empty macro in release. This can be achieved via a building system, separating the definitions in different files (which I recommend), or using macros.

Do not use packed structures extensibly

By definition, packed structures are not aligned, this means that, depending on the architecture, the compiler needs to generate extra code to get members of structures. So, using packed structures extensibly may increase your application size and may decrease the performance.

Another reason for not using packed structures is portability.

Enumerations vs. constant objects

As described in this article, constant objects, such as:

unsigned int const ITEM_ID = 0×01200020;

may increase your application size. So, if you really have a few bytes left, you might be interested in changing the definition by using an enumeration:

enum { ITEM_ID = 0×01200020 };

Bomb Practical Common Lisp

Thu, 20 Sep 2007 16:08:55 GMT

I started reading PCL some time ago (only time for 10 chapters). This is my first Common Lisp book, and by now I’m not ready to write a great article about Common Lisp details, but if you are looking for a Common Lisp tutorial, this could be one.

Bomb it!

Integer comparisons

Sun, 16 Sep 2007 18:12:41 GMT

This may be something basic, but I lost a bit of time last week trying to find a bug at work, so I thought it was worth mentioning it.

When comparing signed and unsigned expressions of the same size, the compiler produces what it might be unexpected results. Suppose you have this code:

#include <stdio.h>

int
main (void)
{
  unsigned short int a = -12;
  signed short int b = -12;

  printf ("%s\n", (a == b) ? "OK" : "failed");

  return 0;
}

Here, you would probably expect an “OK” output, but surprisingly you get “failed“. Why is this? If you know about integer promotion, you may skip to the end if you don’t want to follow all the process.

Fortunately, I received a copy of K&R last week, so I started digging into the issue to try to understand why this was happening.

About equality operators, K&R section A7.10 reads

The equality operators follow the same rules as the relational operators…

Section A7.9, about relational operator, reads

The usual arithmetic conversions are performed on arithmetic operands…

So, how these arithmetic conversions work?

This is also clearly explained in K&R section A6.5 (the same rules apply to the C99 standard, section 6.3.1.8, Usual arithmetic conversions). The part that interests us is after having evaluated all the real type conversions, when it says

Otherwise, the integer promotions are performed on both operands…

So, before performing the comparison of our operands, both undergo integer promotion. Integer promotion (K&R, section A6.2) says that

If an int can represent all the values of the original type, then the value is converted to int; otherwise the value is converted to unsigned int.

An int can represent all the values of our operands, so after the integer promotion is performed, both operands have int type. Having a look back to our operands, we know that the a variable holds the value 0xFFF4, and after applying the integer promotion, it maintains the value. The same happens with variable b, that holds 0xFFFFFFF4 to represent -12. Clearly, both values are different and the check fails.

At the end of K&R section A6.5 this is explicitly explained

The new rules are slightly more complicated, but reduce somewhat the suprises that may occur when an unsigned quantity meets signed. Unexpected results may still occur whan an unsigned expression is compared to a signed expression of the same size.

Note, that this issue does not occur with 32-bit values, as both operands would be “0xFFFFFFF4″.

So, be careful when comparing unsigned and signed types.