Build systems are the new black
So I got into a little bit of a twitter argument last night about distro packaging. Which actually wasn’t where I was trying to go (this time ) One of the problems with twitter is that 140 characters can be hard sometimes. So let’s see if more characters help.
There is a big push afoot from a lot of people towards omnibus packaging. It seems especially prevalent in the world of things written in Ruby I suspect because most even “current” Linux distros are or were shipping some Ruby 1.8.x build up until very recently. And people want to take advantage of the newer language features.
I have a lot I could write on this topic. But I’m not going to today.
The main point I was trying to make is that in going down this path, people are putting together increasingly large build chains that have complex dependencies and take a long time. And so they’re starting to do things like caching of sources, looking at short-circuited builds and a whole lot of other related things. Which, incidentally, are all things that all of the Linux distributions ran into, fought with and figured out (admittedly, each in their own way) years ago. So rather than just rediscover these things and do it yet another way, there’s a ton of knowledge that could be gained from people that have built distribution build systems to make things better for the omnibus world.
So reach out to people that have worked on things like the Debian build system, the OpenSuSE build system, plague, koji and a slew of others. There are some tricks and a lot of tools which are just as valuable as when they started being used. Things like caching build roots and how to fingerprint them for changes, what ccache can and can’t be good for, how to store things reasonably in a lookaside to not depend on upstream repositories being down while you build, incremental builds vs not, ccache, …
Just because you think that building system RPM or DEB packages doesn’t meet the needs of your users doesn’t mean that you have to throw away all of the work and experience that has gone into the toolchains and build systems present there.