At last, the new macro stuff is nearly done, thanks to some work by Zack yesterday. We bootstrap and pass the tests in the testsuite, and are more precise about corner cases than before. Just -traditional stuff to go, and we should be able to apply it to CVS. If you use non-ISO stuff like the GNU ## extension to delete the previous token, or token pasting to get a non-token (remember, we're grown-up and token-based now) you'll get warnings telling you to clean up your act.
A lot of ugliness remains, but that will be easier to clean once we're happy we've got working code and binned the old text-based expander. Many areas are much cleaner, for example the three places (#assert, #unassert and #if/#elif) that need to parse assertions all use the same code now, rather than having their own slightly different version to handle the slight differences of syntax.
The token-based macro expansion process is quite simple in concept, but the reality is a bit messy and hard to understand from the source code. I'll try to clean it up and comment it once we're sure it's working, and have it in CVS.
After -traditional, the next stage is probably to get cpp re-integrated with the front ends, as a library and not a separate process. This will cut out a lot of overhead: an extra exec(), writing out the preprocessed file, the front end reading it in again, and re-tokenising.