Older blog entries for aleix (starting at number 53)

BitPacket: Python 2.x and 3.0 compatibility

Lately, I’ve been porting BitPacket to Python 3.0. I wanted to keep backwards compatibility with Python 2.6 (which is the 2.x I have in my Debian) and, thankfully, I only had to fix three minor issues:

  • Unicode strings
  • Dictionary keys
  • Bytes vs. strings

StringIO and unicode strings

If you have ever used the StringIO module you should be familiar with this:

try:
    from cStringIO import StringIO
except ImportError:
    from StringIO import StringIO

In Py3k the StringIO is located under the io package, so you should changed the above by:

from io import StringIO

which is also compatible with Python 2.6.

Once I did the change my code only worked in Py3k, Python 2.6 complained when trying to use the write method with a simple string:

>>> from io import StringIO
>>> stream = StringIO()
>>> stream.write("test")
Traceback (most recent call last):
  File "", line 1, in
  File "/usr/lib/python2.6/io.py", line 1515, in write
    s.__class__.__name__)
TypeError: can't write str to text stream

You should note that in Py3k all the strings are unicode strings by default. This is not true in Python 2.6, so my first approach was the following:

>>> stream.write(u"test")
4

Unfortunately, this only worked in Python 2.6. Py3k does not recognize the unicode prefix “u“, giving you this error:

>>> stream.write(u"test")
  File "", line 1
    stream.write(u"test")
                       ^
SyntaxError: invalid syntax

I googled a bit and find out a Making code compatible with Python 2 and 3 post (from the guy that finished all the SICP exercices) where it explained some similar issues, so I came up with this solution:

def u_str(string):
    if sys.hexversion >= 0x03000000:
        return string
    else:
        return unicode(string)

>>> stream.write(u_str("test"))
4

In Py3k, unicode does not exist but as that line is never executed we don’t get any error.

Even that worked well, I was not very happy with it. It was too slow and I had to use the custom u_str function everywhere. So, I googled a bit more and I found a nice pycon 2009 talk about Python 3.0 compatibility. Finally, I had which I think is the best solution (for both speed and clearness):

try:
    # This will raise an exception in Py3k, as unicode doesn't exist
    str = unicode
except:
    pass

So, instead of defining a new u_str function, the str type is re-defined as unicode for Python 2.6. Then, I only had to update all the strings in the code to use str:

>>> stream.write(str("test"))
4

Note: I put this code in a compatibility.py file and import it everywhere I need it.

Dictionary keys

The next problem was reported by the 2to3 tool that comes with Py3k.

-                for k in field.keys():
+                for k in list(field.keys()):

Basically, it told me that the dictionary keys() method returns a view in Py3k not a list, so it needs to be converted to a list as explained here:

dict methods dict.keys(), dict.items() and dict.values() return “views” instead of lists. For example, this no longer works: k = d.keys(); k.sort(). Use k = sorted(d) instead (this works in Python 2.5 too and is just as efficient).

Bytes vs. strings

Finally, the last issue was about the difference between strings and bytes in Python 2.x and 3.0. In Python 2.x, bytes is just an alias for str:

>>> bytes
<type 'str'>

In Py3k, bytes and str are different classes and behave differently, see below:

>>> s = "AB"
>>> s[0]
'A'
>>> s[1]
'B'
>>> b = b"AB"
>>> b[0]
65
>>> b[1]
66

This means that one needs to take care of functions returning bytes (e.g. struct.pack) and the operations performed with the returned data, in my case a call to the ord function, that failed with the typical error message:

TypeError: ord() expected string of length 1, but int found

So, following the approaches mentioned above I added the following function to my compatibility.py:

def u_ord(c):
    if sys.hexversion >= 0x03000000:
        return c
    else:
        return ord(c)

which I used instead of the built-in ord in the struct.pack case.

Hope this helps to someone.

Happy hacking!


Syndicated 2010-10-14 19:09:39 from axelio

Snow town!

I just wanted to share with you some pictures of my town during the unusual and unexpected snow… Enjoy!

IMG_0393.JPG

IMG_0394.JPG

IMG_0382.JPG

Syndicated 2010-03-08 17:32:51 from axelio

16C950 UART 9-bit and custom baud rates support for Linux


Last friday I found myself patching the Linux kernel sources to be able to add custom baud rates support for 16C950 UART cards. I needed to communicate, via a serial port, with one of the devices built by the hardware guys at work. Unfortunately, a non-standard baud rate and 9-bits were needed.

The Sealevel Ultra COMM+422 PCI card we are using is already provided with a patch that adds 9-bit support. However, I was not able to found how to change the baud rate to a non-standard one.

Following the 9-bit patch approach, I added extra requests to ioctl in order to modify the 16C950 registers needed to achieve custom baud rates. Here is the list of added extra ioctl requests (also the 9-bit ones) with the necessary parameters and updated/accessed 16C950 registers:

Request Parameters 16C950 registers Description
TIOC9BITGET out: integer NMR Get current 9-bit status
TIOC9BITSET NMR Enable 9-bit support
TIOC9BITCLR NMR Disable 9-bit support
TIOCPRESCALERGET out: integer EFR, MCR Get prescaler status
TIOCPRESCALERSET EFR, MCR Enable prescaler
TIOCPRESCALERCLR EFR, MCR Disable prescaler
TIOCDIVLATCHGET out: integer LCR, DLL/DLM Get divisor latch register
TIOCDIVLATCHSET in: integer LCR, DLL/DLM Set divisor latch register
TIOCSAMPLINGCLKGET out: integer TCR Get sampling clock
TIOCSAMPLINGCLKSET in: integer TCR Set sampling clock
TIOCPRESCALERCLKGET out: integer CPR Get prescaler clock
TIOCPRESCALERCLKSET in: integer CPR Set prescaler clock

I’m not sure if this is the best way to do it, but it works. So, if you need 9-bit and custom baud rates support, apply one the following patches to the kernel (I have aslo updated the Sealevel patch for older and newer kernels than the one provided):

9-bit 2.6.24 2.6.26 2.6.31 2.6.32
9-bit and baud rate 2.6.24 2.6.26 2.6.31 2.6.32

I am not responsible for any damage that these patches can cause to your hardware. No warranty is provided, so use them at your own risk.

Syndicated 2010-02-21 08:06:06 from axelio

SCEW 1.1.1 released


Finally, character escaping has been added to SCEW. This new release only features this and fixes output on Windows console for UTF-16 characters.

Stay tuned, major BitPacket updates come next!

Happy hacking!

Syndicated 2009-12-11 19:43:35 from axelio

SCEW 1.1.0 released


I’m pleased to announce SCEW 1.1.0. This is a minor release including two minor bug fixes and improvements for XML tree and element comparisons. Users can now provide their own XML tree and element comparison hooks.

Check out the release notes.

Happy hacking!

Syndicated 2009-11-30 20:29:35 from axelio

SCEW 1.0.0 released


I’m pleased to announce SCEW 1.0.0. It has been a long time since the last release in May 2004.

This new release includes a lot of improvements: unit tests, homogenized API, a lot of new functions, support for custom I/O sources, documentation updates, Windows support improved and many others.

Please, have a look at the changes for more details.

Savannah mirrors are being updated, so until then use the no-redirected download.

As usual, bugs, comments, criticisms, etc. are more than welcome.

Happy hacking!

Syndicated 2009-10-30 17:08:10 from axelio

Handling external dependencies with Mercurial


In order to add external repository dependencies in Mercurial, I have created an extension similar to the multi-project management feature of GNU arch. This can be probably done with other existent extensions, but I have not been able to found one that exactly met my needs (the Forest extension can be used for non-versioned Mercurial dependencies, and, in fact, they could be used together, but it is not exactly the same). So please, if anyone knows similar extensions, I would happy to know it. My friend Lluís gave me the idea to also handle non-Mercurial dependencies, so thanks from here!

To download it just type:

hg clone http://hg.hacks-galore.org/aleix/hgdeps/

You can also browse it online.

The extension consists of two commands: deps and depsclone. Below, I am pasting the help of the deps command.

This extension is useful when a repository might depend on some external dependencies: other repositories (Mercurial, CVS, Subversion…), files not handled by a SCM tool or some other dependencies.

To enable this extension, add this to your global .hgrc file (or to your repository .hg/hgrc file):

[extensions]
hgext.deps =
# or, if deps.py is not in the hgext dir:
# deps = /path/to/deps.py

In order to define these external dependencies, you need to add a section [deps] in your repository .hg/hgrc file (or in the global .hgrc) where you will define how to find them. As an example, let’s say that our repository depends on two external libraries managed by different SCM tools: ‘libfoo’ managed by Mercurial and ‘libbar’ managed by CVS.

[deps]
aliases = libfoo, libbar
alias.libfoo = /path/to/libfoo
alias.libbar = :pserver:anonymous@cvs.server.org:/sources/bar
alias.libbar.command = cvs -z3 -d$source co -r $rev -d $dest bar

Two aliases have been defined, one for each library, and also the sources where the libraries can be obtained. ‘libbar’ has two options: the library location and the CVS command to use in order to get the library. For ‘libfoo’ only a path to the Mercurial repository has been specified. $rev, $source, and $dest are substitution placeholders and will be explained later.

Once the location of the depedencies have been defined, dependency lists can be created. A depedency list has an associated name. Following the example above, two depedency lists will be created, one for version 0.9.1 of our repository and another for version 1.0. 0.9.1 and 1.0 will be used as the name of the lists.

[1.0]
f24139319bdb    libfoo    lib/foo
v0r8_0          libbar    lib/bar

[0.9.1]
3a9b061bada1    libfoo    lib/foo
v0r7_8          libbar    lib/bar

A dependency is defined by three fields: a revision name, the alias of the external dependency and a destination directory. The alias must be one of the aliases defined in the [deps] section (see above). The revision name will be substituted for the placeholder $rev also seen before. For Mercurial, the revision name will just be a Mercurial revision of the external repository. Finally, the destination directory will be substituted for $dest.

To facilitate version control, distribution, and merging of dependencies, they are stored as a file named ‘.hgdeps’ which is managed similarly to other project files and can be hand-edited if necessary.

Syndicated 2008-03-17 18:08:51 from axelio

31 Jan 2008 (updated 20 Mar 2008 at 10:31 UTC) »

From GNU Arch to Mercurial


gnuarch.pngOver the last two and a half years we (at work) have been using baz as our distributed version control system. Baz is based on tla, the original GNU arch implementation by Tom Lord. After all this time we have applied more than two thousand patches (distributed in various projects). xtla.pngWe also have been using Xtla (a wonderful Emacs front-end to GNU Arch) and lately DVC (its successor, which handles GNU Arch, Bazaar, git, Mercurial and Monotone). After some months thinking about it, I decided that it was time for a change, baz was too slow working with this magnitude of patches and even slower in my PowerBook. I also wanted a fast web interface and ArchZoom was not an option as it took ages to show changes between two revisions and mostly in any operation. So, I watched the Mercurial Google TechTalk and I also read some blog entries about git and Bazaar. dvc.png I decided to give Mercurial a try as it also had complete DVC support and a great web interface (like the darcs one).

One of the things I needed was a tool to convert our baz repositories to Mercurial. As far as I knew, there was no such a tool, so I thought I could use arch2darcs to convert the repositories to darcs and then use the Mercurial convert extension to convert them to Mercurial. I tried it but it didn’t work. The Mercurial convert extension worked really well with darcs for my personal projects, so, why not write a GNU Arch converter?

After some days trying it, I have posted an issue to Mercurial BTS with a GNU Arch converter. It currently does not support directory renaming, but all other things I have tried work well. mercurial.png The converter is slow as I didn’t know how to compute a complete changeset between patches easily (any help?). The log files might not provide all the information needed (added, removed, modified files), so I decided to use the delta command which computes all this information, and this takes a bit of time.

To use the converter (in 0.9.5) copy it to the convert extension directory (hgext/convert) and edit the __init__.py file following, for example, the darcs converter (you’ll easily see how to add it). Use the converter at your own risk, as I am sure it needs more work. I would appreciate any comments and bug reports until it is included in Mercurial (if ever).

Syndicated 2008-01-31 13:51:31 from axelio

Baz 1.4.2 in Fink


Yes, Baz is deprecated, but we are still using it at work, so to make my life easier I have created the Fink package for the latest version (probably there won’t be more). Download this file and, as usual, type:

$ tar zxvf baz-1.4.2-fink.tar.gz
$ sudo cp baz-1.4.2-fink/*
          /sw/fink/dists/local/main/finkinfo/
$ fink index; fink rebuild baz; fink install baz

Baz depends on some cryptographic libraries (gpgme, libgcrypt and libgpg-error). The updated packages are also included (until updated in Fink) in the provided file.

Please, note that when the official Baz Fink package is updated you will have to remove your local copy (from /sw/fink/dists/local/main/finkinfo/).

Syndicated 2008-01-03 14:03:50 from axelio

ECL 0.9j in Fink


ECL has a new release, and thus a new Fink package update. It is still under validation, so until it is committed you can use this file. To install it, just type:

$ tar zxvf ecl-0.9j-fink.tar.gz
$ sudo cp ecl-0.9j-fink/*
          /sw/fink/dists/local/main/finkinfo/
$ fink index; fink rebuild ecl; fink install ecl

Please, note that when the official ECL Fink package is updated you will have to remove your local copy (from /sw/fink/dists/local/main/finkinfo/).

Syndicated 2008-01-03 07:27:50 from axelio

44 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!