New Configuration Management System

"A port to Alpha Linux is nearing completion, but is currently only useful as a client to a Tru64 server. We intend to being work on a port to IA-32 Linux shortly..."

Vesta looks interesting, but most of us won't be able to give it a test drive quite yet.

Make is the standard tool on Unix and the free software world. Can this Vesta be used with Make? What we need is a source code versioning management system. We should not trade CVS for something else at the cost of needing new build tools.

Vesta seems like it might be a nice control system if you are buildling a large project in a compiled language. Everything in it seems geared towards that idea.

However, it seems like it might be too 'smart' for a lot of smaller free-software project needs. For example, from what I gathered, it introspects into code; CVS is nice in that it is entirely line-oriented. It doesn't care what sort of files it is handling, except that it handles text files exceptionally nice.

Similarly, make is also 'dumb'; it doesn't care at all what it is handling; it's an 'updating' system at its heart (once you grasp this you can realize it can do some real fun things). It doesn't care if its dealing with source, binaries, docs, etc.

'Dumb' is good in that it provides a more flexible basis for more tools to be built on top. Sort of an end-to-end way of thinking.

Projects like Vesta will have their place in large projects where rigor is required. A good test case would be to see if Vesta would help the Mozilla project. If it can't help Mozilla, the behemoth of the Free Software community, it most likely is of little use the rest of the smaller projects.

Download from http://sourceforge.net/project/showfiles.php?group_id=34164&release_id=79499

There is a Linux x86 binary tarball (37MB). Source is 68MB.

Ooops, I got mixed up. That 68MB isn't source.

Has anyone found it?

Read the FAQ at http://www.vestasys.org/faq.html. It is self explanatory: kind of chicken and egg problem. On Sourceforge they provide the egg...

I don't make use of cvs or other revision control systems enough, especially since content management is a primary focus in my work.

I would be inclined to try subversion as a drop-in replacement for cvs... any other suggestions which would also be appropriately compared to Vesta?

Subversion certainly does look promising.

OK, so when are they going to put up comparisons that are easy to read and compare with the current best-of-breed systems like Aegis? Yes, Aegis takes some little time to learn, but it also helps you to do some things (eg regression testing) that you know you should be doing anyway. Like someone said about arch, it's an underpromoted alternative. Evaluate it and see.

I'm still looking through the docs on Vesta, but I think it solves a bigger problem than CVS, subversion and other config mgt systems.

CVS and subversion, for example, solve the problem of multiple people working on a software application's source code. AOLserver, PostgreSQL and other applications are each in their own CVS repositories. This works great and will certainly continue to do so.

The problem that Vesta may solve is one level above what subversion and CVS do. I'll give an example in my work right now: I'm using AOLserver, nsopenssl, nsldap nspostgres and PostgreSQL in a project. This means I have to pull each one from its (separate) repository or download each tar.gz distribution file. I then have to compile each and install each. Then I have to configure AOLserver with nspostgres to talk to PostgreSQL, and so on. Right now I have scripts I've written to recompile, reinstall and rerun everything, so that I can repeatedly build and rebuild the system over again as we add to it, to ensure that the changes don't break anything. Also, if the production system dies horribly, I want to be able to rebuild the *exact* production system as it was installed on the dead production host.

I think Vesta, while it sounds like it has the ability to manage versioned files like CVS and subversion, also goes one step further in that it can manage putting together the scenario above, and do it repeatedly.

I could certainly put all of the sources into our own repository and have it build from them directly using my locally written scripts, but it looks as if Vesta may make it much easier for me and others in my crew to be running with different versions of the core system, integrating their changes, and rebuilding the entire system from scratch repeatedly.

Plus, I work from home at times, and if Vesta can truly replicate and keep itself in sync between my work and home systems, that would be a big plus.

So, I'm going to play with Vesta for a while and see if I can use it to manage a staging server and then use it to migrate the staging system onto the production server at each upgrade point. This should be lots of fun!

/s.
http://scottg.net

The group used Vesta to manage builds with as much as 130 MB of source data, each producing 1.5 GB of derived data I just want to note that kdenonbeta is about 90MB. That 130MB doesn't seem too large to me. :)

My immediate reaction to hearing about arch was: I wouldn't trust my source to it. I see that `provably reliable' is on the list of improvements for it - not confidence inspiring. I'd like to see a credible article along the lines `why you can trust arch no matter what happens'. I trust from experience that I can recover from CVS snafus, and I trust the design priorities Subversion and Bitkeeper. Arch really gives the impression of `features first, reliability later'...

To be fair to Arch, the author isn't claiming it's robust enough to deploy yet, that's why 'make it fast and provably secure' is the first priority. As it says in the survey -

some goals for arch: Because of the mission critical nature of revision control, arch needs a serious test suite and it needs to be tested to shake out the obstacles to scaling to very large, very active projects.

As for it putting features before stability, if it weren't a fundamentally more featureful tool than Subversion, there would be no point in working on it. Subversion is great as the heir apparent to CVS, with basically incremental feature improvements, but I'm glad someone's working on fundamentally rethinking the way diffing is done to support much less form-fitting branching processes.

ftobin said:

However, it seems like it might be too 'smart' for a lot of smaller free-software project needs. For example, from what I gathered, it introspects into code; CVS is nice in that it is entirely line-oriented. It doesn't care what sort of files it is handling, except that it handles text files exceptionally nice.

I'm not sure what you mean by "introspects into code", could you elaborate? (If you mean what I think you do, I'd appreciate knowing what gave you that impression so I might try to correct it.)

Vesta does not do anything to examine the contents of your source files. It's dependency analysis is language and compiler independent, not a language-specific trick like makedepend.

In fact, Vesta is significantly less "smart" that CVS in this respect (and thus better :-). Unlike CVS, Vesta doesn't need to be told "this is a binary file, don't mess with it", because it never automatically modifies your files. It doesn't care what you kind of information you place under revision control, it's all just files and directories to Vesta. (Really, the repository is more like a filesystem than a tool that reads and copies files for you.)

Similarly, make is also 'dumb'; it doesn't care at all what it is handling; it's an 'updating' system at its heart (once you grasp this you can realize it can do some real fun things). It doesn't care if its dealing with source, binaries, docs, etc.

'Dumb' is good in that it provides a more flexible basis for more tools to be built on top.

Vesta's builder is just as general-purpose as Make (at least with regards to the kinds of processes which it can be used to automate). We use it to build the man pages for Vesta along with the executables.

Vesta is designed to be very flexible and general-purpose. There is nothing specific to software design about it. In fact, the major customer thus far has actually been a microprocessor teams using it to manage hardware designs. It's been used to do several other things, including running Perl scripts, constructing tar files, and using SWIG to generate interfaces for different scripting languages. Although I don't know if anyone has done these yet, I know that both managing web content and 3D rendering have been considered as processes that could benefit from using Vesta.

atai asked:

can it be used with Make?

Make is the standard tool on Unix and the free software world. Can this Vesta be used with Make? What we need is a source code versioning management system. We should not trade CVS for something else at the cost of needing new build tools.

The Vesta repository just stores files and directories. It doesn't care what you put in there. (As mentioned above, it has no problem with binary files because it never automatically modifies your content, unlike some other revision control systems.) So of course, you can use Make with Vesta.

However, you'd be missing out on some of Vesta's best features. Make, much as it is well known, has a host of problems (which I won't bother to detail here, as they're already described elsewhere). Vesta's builder has some, in comparison, really fantastic features, including completely automatic language-independent dependency detection and guaranteed repeatability of builds. There's a fairly short document detailing these and other significant features of Vesta that I recommend.

slef asked:

OK, so when are they going to put up comparisons that are easy to read and compare with the current best-of-breed systems like Aegis?

This, unfortunately, falls in the "If only we had infinite time" category. None of original researchers who worked on Vesta (and wrote the the comparison document) are still doing so professionally. Two of them remain involved, but it's no more than a part-time activity for them. Part of my work responsibilities include supporting and maintaining Vesta for my current employer, which is using Vesta. However, nobody is working full-time on Vesta at this point. I wish I had the time to get familiar enough with Aegis to add it to the comparison (not to mention jam, subversion, BitKeeper, Arch, ClearCase, Perforce...).

If anyone out there would be willing to write up a balanced pro/con section for one or more of the other tools out there, I'd be happy to work on getting it included in the comparison.

exa said:

How scalable :)

The group used Vesta to manage builds with as much as 130 MB of source data, each producing 1.5 GB of derived data I just want to note that kdenonbeta is about 90MB. That 130MB doesn't seem too large to me. :)

All I can say is that in 3+ years of use by a very demanding team, we've never had a scalability problem with it. I'm sure it must have limits, I just haven't seen them yet.

egnor noticed the out-of-date text in the comparison document stating that it only ran on Tru64. Thet's no longer the case (Alpha and x86 Linux are supported), and that page was fixed after someone else pointed that out.

The statement I made that Vesta seemed to introspect into code and was 'too smart' were based on snippets of the Vesta FAQ. I'll try to list some of the snippets here to explain what I apparently misunderstood.

• semantic conflicts may go undetected

  If CVS does not detect a conflict, it silently merges the changes made by multiple developers. As a result, some true conflicts may go undetected.

To me, this suggested that Vesta has language-specific structure knowledge that allows it to peer into the code and figure out conflicts, black-magic style.

• dependencies must be specified explicitly

  Dependencies of objects on the sources from which they were built must be listed in the Makefile. This is a completely manual process. Tools like makedepend(1) help, but makedepend suffers from several problems: it is slow, it works only for C/C++ sources, it does not detect all dependencies (e.g., dependencies on tools or other non-C/C++ files read during the build), and it must still be run by hand. Because it is slow, developers tend not to run makedepend as often as they should. As a result, the potential for producing an inconsistent build is increased.

This suggested to me that Vesta had black magic to look at dependencies of particular languages, and was simply 'more automated' and supported more languages than makedepend.

Of course, I probably mis-interprted these statements. I'm skeptical of new systems that promise to over-throw old, settled, flexible, although lacking systems. 'Dumbness' and separation of concern among applications is important to me since I believe that is what makes for good systems (including unix), and I zealously looked into finding problems with Vesta's design in that respect.

Since it's hard to grasp what new systems do without precedent, it's hard to accurately grasp what is going on. I know many people have a proble grasping the concept of CVS up-front (even simple as it is) without having used it, and I'm probably suffering from the same malady.

(Those are both from the comparison to other popular SCM systems, which is different from the FAQ.)

The comment on the "blind auto-merge" feature of CVS is really just saying "we think that's dangerous and a bad design choice". Vesta actually dodges this altogether, as it provides no built-in merging tools. Instead, it provides a filesystem (NFS) interface to all source versions, which makes it fairly easy to use standard tools like diff, patch, merge, etc. (This also makes other interesting things possible, like grep-ing across versions.)

Vesta's dependency detection is language independent, and it is a little magical. :-) Without going into excessive detail, it monitors what files are read during each tool invocation (compiling a source file, linking an executable, etc.). When it completes, it makes a record of the result of the tool invocation (i.e. the result files written) and includes the recorded dependency information (which files were read and a checksum of the contents of each of those files). If a subsequent build requests a tool invocation with the same command line and identical contents for all the files it depended on, the previous result is re-used. (This is much less error prone than Make's dependency mechanism, and also automatically records dependencies on things like the compiler executable itself.) For much more information on how this is implemented, consult chapters 6 and 7 of the book-length research report (and/or read the code :-).

I'm skeptical of new systems that promise to over-throw old, settled, flexible, although lacking systems. 'Dumbness' and separation of concern among applications is important to me since I believe that is what makes for good systems (including unix), and I zealously looked into finding problems with Vesta's design in that respect.

I appreciate your skepticism. It took about a year of working with Vesta for me to be thoroughly convinced. While I won't claim that it's the right choice for all SCM problems, I will say that it's a solid and very flexible system. The filesystem interface to the repository is a great example. It was designed that way precisely so that people can use all their favorite filesystem access/manipulation tools, rather than having to write new "Vesta-enabled" ones.

The researchers who wrote Vesta were good old-fashioned UNIX hackers who have the traditional fondness for composable, general tools. IMHO, once you get to know it, Vesta is a system any UNIX hacker could love.

The researchers who wrote Vesta were good old-fashioned UNIX hackers who have the traditional fondness for composable, general tools. IMHO, once you get to know it, Vesta is a system any UNIX hacker could love.

This is a key statement that definitely makes Vesta more appealing to me, and I'm much more willing to accept the cost of learning Vesta because of it.

This might be okay in an IT department, but Tools support for ant is becoming ubiquitous. Ant is a "likeable" build too unlike make. And source distributions to those whom you might not give access to your repository is a useful thing to do. Maintaining two build systems is unfeasible and I just utterly hate tying my build process to my SCM/VCS/etc system. I'd prefer an approach that let you extend tools like ant via custom tasks/etc to optionally work with your VCS/SCM/whatever and/or not if its just a snapshot. Learning a custom language for my CVS/SCM is....yucky.

-Andy