Older blog entries for conrad (starting at number 63)

Iteratees at Tsuru Capital

Tsuru Capital is a small company. We build our internal systems for live trading and offline analysis in Haskell, and we're proud to be sponsoring ICFP 2011. We use iteratees throughout our systems, and have actively encouraged all our staff to contribute changes upstream and participate in community design discussions. By being part of the open source community and taking part in peer-review, we all end up with better software.

Over time various Tsuru staff members have worked on tools using iteratees, including (grepping the CONTRIBUTORS files): Bryan Buecking, Michael Baikov, Elliott Pace, Conrad Parker, Akio Takano, and Maciej Wos. There's been some lively discussions and many small patches providing functions that we use in production every day.

Last year Conal Elliott provided some mentoring to Tsuru staff, during which we worked through a denotational semantics for iteratees. This resulted in discussions on both the iteratee project list and haskell-cafe about Semantics of iteratees, enumerators, enumeratees.

By using iteratees in production we've contributed various simple but practical functions, including:

  • enumFdFollow, an enumerator (data source) which allows you to process the growing tail of a log file as it is being written.
  • ioIter, an iteratee that uses an IO action to determine what to do. Typically this is action involves some user interaction, such as a user issuing commands like play/pause/next/prev.
  • ListLike functions last (an iteratee that efficiently returns the last element of a stream), mapM_ and foldM.
  • mapChunksM_, a more efficient version of mapM_ that operates on the underlying chunks, eg. logger = mapChunksM_ (liftIO . print).
  • takeWhile, and its enumeratee variant takeWhileE


  • endianRead8, an iteratee for reading 64bit values with a given endianness. I've used this in ght as well as an internal project.
Stream conversion We've done quite a bit of work on stream conversion, as we use a few different layers of data processing. The iteratee architecture allows you to isolate the data source, conversion and processing functions; much of what we've worked on involves ensuring the converters (enumeratees) can control or translate control messages, so that commands like "seek" do not get lost. We've also built combinators to simplify the task of creating new stream converters.
  • convStateStream, which converts one stream into another while continually updating an internal state. Importantly for variable bitrate binary data, it can produce elements of the output stream from data that spans stream chunks.
  • (>) and (. These allow stream converters to be composed without rewriting boilerplate. Jon Lato gives a good example using these in the StackOverflow answer to Attoparsec Iteratee.
  • zip, zip[345], sequence_ for using multiple iteratees to process a single stream instance, and (for zip*) collecting the results.
  • eneeCheckIfDone*: This family of functions (eneeCheckIfDoneHandle, eneeCheckIfDonePass, eneeCheckIfDoneIgnore) can be used with
    unfoldConvStreamCheck to make a version of unfoldConvStream which respects seek messages.


Parallel stream processing We often want to do multiple unrelated analysis tasks on a data stream. Whereas sequence_ takes a list of iteratees to run simultaneously and handles each input chunk by mapM across that list, psequence_ runs each input iteratee in a separate forkIO thread. For a real-world example, see Michael Baikov's post about psequence, psequence_, parE, parI.

Thanks

Thanks to John Lato for consistently and reliably maintaining the iteratee package, providing thoughtful feedback and graciously suggesting improvements.

Syndicated 2011-09-18 09:10:00 (Updated 2011-09-18 09:10:31) from Conrad Parker

A Haskell template for GTK, Glade, Cairo apps

I just uploaded cairo-appbase to Hackage. This is a template for building new GUI applications using GTK, Glade and Cairo.

To install it:

  
$ cabal update $ cabal install gtk2hs-buildtools $ cabal install cairo-appbase

Then, run cairo-appbase:

The GTK widget layout is done via a Glade XML file which can be edited visually using glade. This template includes working callbacks to handle the File and Help menus and File Save/Open dialogs, with dummy handlers for selecting filenames and the Edit menu's cut/copy/paste. The main canvas uses Cairo for graphics rendering, and includes example code from the cairo package.

To build your own application on top of this, first grab the code. You can either grab it from hackage with cabal unpack cairo-appbase, or clone the git repo:

To add widgets, install glade from your distro system and run glade data/main.glade. Note that you must run cabal install to put the glade file in the correct place for your application to pick it up. To modify the code, edit src/cairo-appbase.hs. Hooking up functions to widgets is very simple: get a widget by name (which you set in glade file), and hook one of its signals (which you found in the Signals tab in glade) to an IO () action:

  
cut1

The template code includes a trivial definition of myCut:

  
myCut :: IO () myCut = putStrLn "Cut"

A real application will want to pass data to the callback. In C, this is fairly tedious as you only have a single void * to pass to callbacks as "user_data", and applications typically do lots of marshalling and unmarshalling to pass data around. In Haskell however, you can make yourself a more complex callback handler and use a curried version of it in each instance:

  
cut1 MoonPhase -> LuckyNumber -> IO () myCut project phase num = do let selection = currentSelection project when (phase == Full) howl when (num /= 7) fail doActualCut selection

Erik de Castro Lopo discussed how currying at length in his April 2006 post, GTK+ Callbacks in OCaml. The Haskell GTK+ bindings have been around a long time, but were only recently cabalized and uploaded to Hackage. I put together cairo-appbase in August 2006 when I was playing with it, but now that I have more time for Haskell I've updated it and uploaded it to Hackage. Enjoy, and hack away!

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!