Older blog entries for rlk (starting at number 7)

CMYK!

Yesterday I helped someone print a CMYK tiff to an Epson Stylus Pro 7000 printer under Linux.

That doesn't sound like a big deal, you say. Gimp-print has printed to Epson Stylus printers for a few years now, and who really cares about CMYK? Answer: the Stylus Pro printers are Epson's high end professional printers, and a lot of professionals like the extra control that CMYK gives them. Not to mention, Epson's own driver doesn't handle CMYK input.

We did this using CUPS. I had to fix a few bugs in CUPS; one of them may have been a bug in libtiff; the fix was a one liner, to enable CUPS to handle the CMYK TIFF. The other one was more substantive -- CUPS was converting the CMYK to RGB, only to convert it right back to CMYK. Note that CMYK is four channels, and RGB is only three. That meant that information was getting lost. In this case, what the file in question did was print 100% C, M, Y, and K. The conversion to RGB made it 0% R, G, and B; and the conversion back made it 100% K, and 0% C, M, and Y. Not an insignificant difference.

This is seriously cool to professional users. There used to be an affordable RIP (raster image processor) called Adobe PressReady, but it never supported the high end printers, and Adobe has since discontinued to it (speculation was that some of Adobe's partners, who sell much more expensive RIP's, didn't like the competition). Whatever the case may be, there's at least the germ of another way to do it. Ghostscript is, after all, a Postscript RIP (among many other things), and if it can do CMYK, it means that we, the free software community, can now start to compete in the graphic arts field.

That's not to say that we're anywhere near all the way there. Without color management, we still miss something critical -- the ability to map input colors to screen colors to output colors, and close the loop. So we're currently operating open loop. But we're making strides.

Gimp-Print 4.2 is released!

Gimp-Print 4.2 is finally released. Of course, it took longer than expected; there were the usual last minute bugs.

It has certainly turned out to be a much bigger release than I had planned on, though. My expectation was that we'd clean up a few of the nastier problems in 4.0, prior to moving on to rearchitecting it. What's actually happened is somewhere in between. We didn't redo the color and dither code from scratch, and redesign how metadata is handled, but we did considerably more work than I envisioned. So I suppose that's why it took 13 months rather than 6-9 months. Was it worth it? Probably. Is 4.2 a good release? I certainly think so.

Dave Winer doesn't care for Richard Stallman

There are a number of things about this article in Scripting News that rub me the wrong way:

  • The claim that Stallman doesn't push the envelope. Exactly what does Stallman have to do to be taken seriously as a developer? I mean, he's only been the principal architect behind multiple versions of Emacs, the driving force behind gcc, the GNU project in general (which may be an indirect derivative of UNIX, but the tools are much better), and who knows how much else that I can't think of. Does he have to produce some grandiose new piece of software, ab initio, every year? I read an article in Circuit Cellar (which focuses heavily on embedded development) that was an introduction to the GNU toolchain. It sure seems like a lot of people in the embedded space like a compiler that's identical across so many platforms.

  • Open source projects are essentially open debating societies. I guess this one really sticks in my craw, because I've put no small amount of effort into building Gimp-Print into a real project. People do real stuff, be it coding or documentation, it gets into the source base, and people work together to improve things. KDE, meanwhile, does something truly amazing: they release a couple of major versions of a whole desktop environment, with applications, each year. And let's not forget the likes of Debian. Meanwhile, I think we've all heard of or seen plenty of commercial projects dissolve into bickering.

  • His evident resentment that Stallman won the Takeda Award. I guess RMS did something that the Takeda Foundation liked enough.

I get the feeling that there are a fair number of people out there who think that if you aren't out to make money, you should stand aside for people who are. Certainly Microsoft is explicit enough about that, but then there are family members who think I'm insane for not patenting (!) Gimp-print and making a fortune (!!) out of it. I guess the thought of actually wanting to give something back to the community doesn't cross people's minds.

Free software isn't going to destroy programmer's livelihoods. Maybe it will make it harder for people who want to turn out shovelware and make a quick buck, but somehow that doesn't bother me. Programmers should applaud the fact that they don't have to keep reinventing the wheel, and can build on the work (reciprocally) others do. Instead, people keep angling for barriers they can raise against competition. The success of software is measured by how rich it can make a handful of proprietors.

It's been a while

Gimp-print 4.2 is nearing release. It's hard to believe that it's been more than a year since gimp-print 4.0 came out; I didn't expect it to take as long. It's going to be a good release; we have a nice mix of new printers (particularly Epsons, since Epson is so much more helpful with what we really need), improved quality, and improved architecture. Roger Leigh, who joined after 4.0 came out after I found that he had done some work on an Epson utility, did a huge job creating a proper library from the core. And finally, Andy Stewart wrote a user's manual. That's one of the biggest improvements in 4.2. Pity docbook's so fragile, but it's worth it.

I would have liked to have done more architectural work, but 4.2 wasn't planned to be a major release, so we didn't expect to accomplish much. We accomplished more than we really planned, but we took a lot longer (I envisioned 6-9 months, and we're currently over 12). Some of it has been feature creep, but I don't think that that's the main problem; we've stayed within reasonable bounds. The big problem is that everything happens slowly, because nobody can really commit to anything, because paid work and such get in the way. The big architectural change happened early in the cycle, and all of the fallout was really over by early spring. Cleaning up the engineering of the documentation has taken some time, and a lot of that came in late, but that's as good a reason to hold a release as any.

We have 15 bugs open right now. The bug rate shot way up after beta-4, although some of that's simply pickier reporting on my part. There's one really critical one (more below), one that concerns me but I need to know how to try to reproduce it, two more easy fixes that I have out for review but that should never have gotten as far as they did before being found, and another one that just needs testing on a BSD platform. There are some other bugs that are definitely fixed, some that are low enough priority so we can go release without them (we've planned to, in fact), and so forth. So it's not quite as bad as it looks, but we do have a few stoppers yet.

Translation of PPD files

The nasty bug involves translation of PPD files. Another thing that we've done this time around is that we're translating Gimp-Print into a number of languages. One of the core gimp-print applications is a driver for CUPS driver. CUPS uses PPD files, and these files should be in the correct language for the user.

The PPD files are created at compile time, not at run time, and therein lies at least part of the problem. The internationalization mechanism (GNU gettext, or anything compatible) likes to find its message catalogs in certain places, and of course at compile time they aren't there. The POSIX-standard stuff seems to only like working in terms of valid locales, and there's no guarantee that the name of a language (e. g. "sv") will be the name of a valid locale. If it is, everything works; if not, it usually doesn't. The GNU gettext has an escape hatch: if you set the LANGUAGE environment variable, it just uses that without worrying about anything else.

The standard configure package has a --with-included-gettext option to use the gettext bundled with the package (in this case, we're bundling 0.10.40). I don't really like this, since it forces a fairly important decision on the user based on something that's only needed once (at compile time), but this would probably be acceptable, if it worked. Problem is, at least on the FreeBSD machine at Sourceforge, it doesn't! Even with --with-included-gettext, even with static linking, the PPD files just don't get translated! This causes six identical copies of the PPD files to get translated. When there are 133 PPD files per language (totalling 1 MB per language compressed), and when every one of these shows up in a menu for CUPS users, it's really quite unpleasant.

It's probably quite obvious from this that I am not an expert in this particular area. If anyone is, and would like to help us out, the bug in question is here.

Gimp-print 4.0

We're on the 5th minor release of 4.0 (4.0.4). There have been two bugs that we've fixed in the core (one for the HP 1200C, and one in the color conversion code); the rest of the bugs have been in the Ghostscript or CUPS drivers. That's about what I expected; the core has had a lot of work, while the drivers haven't been exercised as much. What's annoying is that we could have caught a lot of these through better testing. There were a few fairly nasty ones.

One issue that we're considering with 4.0 is how much to back port from the development mainline. There's one particular improvement (support for C-RET mode on higher end HP inkjets) that quite a few people want, but it needs more work on the mainline. Another issue is the color quality. There are some fairly serious color issues with 4.0: reds are rather orange, blues are rather violet, and greens are somewhat straw-colored. The improvement on the mainline is somewhat ad-hoc: a hue map. This does corrections in HSB space rather than RGB or CMY, but it does a remarkably good job of correcting the gross color errors. It won't correct other errors, though.

Gimp-print development

I consider the hue mapping described above to be purely experimental color correction. We really need a better color model. This will presumably happen over time. Nonetheless, reaction to even this has been very positive. I've really come remarkably close to matching my display.

Andy Thaller has been putting quite a bit of work into the Canon driver. He's borrowed the ink model from the Epson driver. As long as we can find enough testers, we should be able to greatly improve that driver. It still won't match the Epson driver very soon, because Canon won't give out the kind of information we need to put the printer in the very highest quality modes. Enough people have commented on the printer vendor situation, so I'll simply reiterate: printer vendors who hand out information will find that the free software community will supply high quality drivers for their printers, while printer vendors who don't will find the world passing them by.

We're also talking with the Omni folks from IBM about their approach to the metadata problem. There's a huge amount of information that goes into printing something, and we only expose a very small part of that to users. This is unfortunate; it makes it much harder for users to tune their printout, using all of the settings that the dither and color code export. The Omni team has put a lot of effort into the architecture of their system, and not as much into quality; we've put tremendous effort into quality, but our architecture is weak. There should be a lot of possibilities for synergy here.

We're also trying to decide what 4.2 should look like. I had envisioned it as a fairly large release, converting to an internal CMY(K) model, with an external metadata representation, and the like. I'm having a bit of a change of heart about this. Perhaps it might be better to make it a fairly conservative release, doing something (not necessarily the ultimate) about the color quality, fleshing out the drivers, and such, and hold the really big stuff for another round.

gtaylor is working on the Foomatic stuff so that we can version it. We've already diverged enough from 4.0 so that the 4.0 Foomatic database won't work with the mainline. What I really want, though, is to be able to generate the foomatic data files programmatically, rather than having to download the tarball from linuxprinting.org. In addition to being slow, and bulking up the release, it's not very flexible.

Gimp-print 4.0 is finally out there

It feels like it's taken forever to get this one out. Cleaning up the last few things is always a slow process, but it feels good to have it out there. Of course, now there will be a lot more users and a lot more bugs...

I think I need to slow down a bit from the pace of the past year. It was incredibly intense at times, but with a great group of people we got something big accomplished. Not the biggest thing anyone's ever done, but it now means that free software has access to the kind of print quality that didn't exist before, so there are new areas opened up in the graphic arts. That's Good!

I cut gimp-print 4.0b2 tonight. Hard to believe that the next release will probably be the real thing! This was 2 weeks after 4.0b1, which was just about how long I expected the gap to be.

I originally figured it would ship some time around late summer, in the August timeframe. So it looks like it's going to be about 2 months later than original plan. I'm quite satisfied with that; we made most of our goals from our development road map from January, except for 16-bit support. However, since there's no common Linux application with 16-bit color support, that's really quite OK.

It's also arguable to what degree we accomplished goal 8, better separation between the rendering engine and the printer drivers. The interface is better, but plenty of people remind me that it can be a lot better. That's true, and perhaps we'll do that in 4.1/4.2.

However, I can realistically say at this point that we've accomplished far more than I could have possibly dreamed, with very high quality (in terms of both bugs and print quality) within 30% of my estimated time goal. So I feel good about this project!

The one major concern I have is that a few people are reporting poor quality of some kind, and I can't tell (because I haven't received printouts from them yet) if there really are some oddball color problems I don't know about (possibly due to code bugs/compiler interactions, like we had during early alpha) which would be bad, or because we're attracting people who are incredibly picky about their color, which would be good. There are also a few reports of poor density with the Stylus Color 900 at 720 DPI. This does concern me a bit, because the 900 has a slightly oddball way of getting 3 picolitre dots. The other reports all seem to be bad gamma values brought on by a bug.

There's no way I'll be able to work this hard on 4.1. It will be necessary for more people to step up, or 4.1/4.2 will be a much slower process.

I would definitely like to be able to work on free software full time at some point in my life...

Well, one thing's for certain: we have lots of tuning left to do on gimp-print. The good news is that a lot of it might just be parameter tuning rather than a massive overhaul.

Niels Gram Jeppesen forwarded me a photo of a friend's car. Now, said car is a beautiful red Alfa Romeo, and he was justly disappointed with the somewhat orange cast the car was taking on. He didn't notice (or took pity and didn't comment on) the rather strange color of the grass. I tried fiddling around, boosting the saturation and decreasing the yellow, and got slightly better results.

This morning I printed out a photograph of my wedding (the same photograph that's on my home page) to see how far things have come since 3.0 (or maybe an early 3.1, I don't know how long ago I printed it). Well, in terms of smoothness of dithering, we've certainly come a long way. However, it was definitely oversaturated, and had a somewhat yellow cast. Hmm.

Photograph mode has some hacks to improve color fidelity. These hacks hacks reduce the CMY levels in regions of fairly pure CMY (which is needed to prevent CMY from reaching saturation too quickly), and do some other things which appeared to reduce saturation (theoretically, they should have reduced saturation also; increasing the level of the least dominant ink will do so by definition). So I boosted the baseline saturation by 1.6 to compensate. Well, it appears that in practice that's going way too far. In fact, I got a really good print with saturation set to .625 -- the exact inverse of 1.6! So that suggests that no saturation boost at all is the right way to go. That's nice; it will speed things up too.

Anyway, with saturation at .625, yellow at .8, and overall gamma of .9 (that's also scaled from a printer and application specific baseline) I got some very nice prints. A brilliant sunset shot, unfortunately, is proving harder to fix.

What's really amazing me here is that reducing the saturation increases the perceived saturation of the color of the car. If I look at it very closely, though, the car isn't fully saturated at all; it's one of those perceptual quirks that makes it look like a brilliant, earth-shattering red. I don't have enough grounding in color theory to fully understand the details, but I think that what's going on is that cyan is the least dominant color in the output, so increasing the saturation decreases the cyan, which leaves only magenta and yellow. The end result is that the output looks slightly orange. Decreasing the gamma darkens the output, which also increases the perceived saturation.

What gimp-print really needs to do is to do everything in CMY space rather than RGB space, I think. That isn't going to magically solve anything, but it will at least make things consistent and hopefully easier to solve.

Of course, part of the story is that every printer is different. The 6 color Epson Stylus Photo EX is a more difficult printer to get light greens out of than a 4 color model; the result is that grass tends to look a bit brown. This isn't quite as offensive as one might think, because it's actually a fairly natural brown. When the saturation is less extreme, this is also less noticeable.

In any event, I think I'm probably going to rescale the saturation, since that's clearly wrong. The issue of yellow is very printer and lighting specific, and I think I'm going to leave it alone for now. Gamma is also a tough call, since every monitor's gamma is different.

I'm looking forward to khk's color management stuff for 4.1/4.2.

Well, I guess it's finally time to join Advogato...

It's amazing how fast things can happen under the right circumstances. About a year ago, I wanted a printer to let me print proofs to feed my photography hobby. So I bought an Epson Stylus Photo EX from a fellow photography buddy before I realized that there wasn't much of a driver available for it under Linux. The closest thing I could find to something useful for my purposes was the print plug-in that came with the Gimp -- it supported a lot of Epson printers, but not the 6-color ones. Just figuring out what driver would actually work at all was a bit of a challenge.

I did a lot of work, and released version 3.0 of the plugin (I started with Mike Sweet's 2.0 version), which supported the EX reasonably well. Certainly it did a lot more than 2.0, and it used the full print head at 1440x720 DPI.

Fast forward to now. The project, gimp-print, now has 19 registered developers. It's in beta for the 4.0 release. Not only is it the print plugin for the Gimp, it's a full-fledged Ghostscript and CUPS driver. Just for kicks, I went back and printed something out with my old 3.0 driver. Even a few months ago, the development line leading up to 4.0 did far better in 4-color mode than 3.0 did in 6-color mode, and I was quite impressed with the 6-color stuff back then. Over the intervening 2 months, it's only gotten better.

Having those 18 other folks around is the key to it. Professionally, I've focused on ultra high end servers and supercomputers, and low-level UNIX hacking, and performance. I've never done graphics, and I knew nothing at all about dithering. This has been a big learning experience. I've learned about dithering, and while I still don't consider myself an expert in it, I know who to talk to and who can write the code for it. Without Thomas Tonino to design dithering matrices, for example, the quality would still be back in the January days. Right now, 6-color mode on the EX really looks like a photograph, and 4-color mode looks like a grainy photograph. On Epson's top of the line printer, the 870, it's even better.

From a professional standpoint, I've learned that I have project management skills. If I can manage a group of 18 people who have absolutely no obligation to the project, I figure I can manage a complex project in any situation.

There's something really exciting about watching all of this develop.

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!