Older blog entries for apm (starting at number 10)

Hey! I picked up the new 25th Anniversary of Dr. Dobbs journal today and was flipping through it while waiting for a compile (I need a faster computer!) One of the columns I always read is Michael Swaine's Programming Paradigms. Lo and behold he devotes a paragraph to Suneido! I had sent him an email about it, but I'm sure he gets a ton of email and I didn't really expect anything to come of it. He doesn't really give any opinion of it, positive or negative, but the fact that he mentioned it seems positive in itself. A mention of Suneido on the Dr. Dobbs website gave us a lot of traffic, so hopefully the magazine will be even better.

It would be very useful in Suneido to have a way to automatically release resources such as Windows "handles" and open files. Suneido's garbage collection will automatically release the memory but it doesn't know anything about resources. Currently, programs must explicitly release their resources when they're finished with them e.g. close files. In user interface code this is often done in response to the WM_DESTROY message. But in other places e.g. report formats, there is no obvious place to do this. It can be especially hard to ensure that resources are released in the event of exceptions (errors). It's really the same kind of problem that garbage collection solves for releasing memory.

Initially, I thought the way to do this would be to support "finalization" methods in Suneido. But after looking into it further, I think a better approach is to add "weak references" to Suneido, and to use them to release resources. My current feeling is that this is something that should be added to Suneido in the near future.

The garbage collector calls finalization methods after determining an object is no longer referenced, but prior to free'ing it. Finalization methods can release resources such as file handles that are used by the object. Java has finalization. It seems like a simple, obvious solution. However, there are a number of subtle problems with finalization. One problem is that the order of finalization is not predictable. Another problem is that it is possible for finalization methods to "resurrect" an object by storing a reference to it. This in turn "resurrects" any objects that this one references. But these other objects may already have been finalized! There are ways to work around these issues, but obviously it's not as simple as it might seem.

"Weak references" are another memory management facility. A weak reference stores a reference so a memory object in a way that doesn't stop the object from being free'd by garbage collection. At the end of garbage collection, any weak references that refer to objects that are now free'd are cleared. This might work as follows:

wr = WeakReference(value)
wr.Get() => value // value is still "live"
// some time later
wr.Get() => False // value has been free'd

Weak references are a more primitive mechanism; they are simpler to implement and do not have the same problems as finalization.

The key idea is that you can use weak references to implement a scheme to automatically release resources. The basic method is:

  • When you allocate a resource, you add it to a table.

  • The table contains an entry for each resource, including a weak reference to the object using the resource, and information about how to release the resource (e.g. the handle value and a "release" function)

  • Periodically (possibly by a background thread) the table is scanned for entries whose weak reference has been cleared. Using the information in the entry, the resource is released, and the entry is then removed from the table.

One enhancement is to have the garbage collector add cleared weak references to a queue. You can then eliminate the linear scan of the resource table by keying it on the weak reference.

A form of weak references can also be used to reference memory objects that can be free'd if memory space is required, for example, a cached copy of an external file. Java calls these "soft references". The garbage collector treats these references slightly differently in that it only free's the objects if memory space is running low.

For more information on weak references in Java see: Java finalization

30 Nov 2000 (updated 30 Nov 2000 at 20:32 UTC) »

I've been working on documentation for the last little while and I needed a break and some real coding. So I decided to make semicolons optional in the Suneido language. I've been toying with this idea for a while and over my coffee at Tim Hortons one morning, I worked out how to do it and it seemed pretty straightforward. In the end, it took me about half a day to get it 99% working, and another day and a half, and several complete rewrites, to get the last 1%. Typical. On the positive side, the current version is a lot cleaner than the initial version.

One thing I was afraid of was that the changes would ``break'' a bunch of the existing code in stdlib. So I wrote a quick function to syntax check all the records in stdlib.

QueryApply('stdlib')
    {|x| try x.text.Eval(); 
    catch (e) Print(x.name " - " e); }
Then I put this in a text file so I could run it even if I there were too many errors to get to the WorkSpace. (With the Print changed to output to a textfile.)

My first stab at the changes resulted in about 500 records with syntax errors. After fixing a few blatant issues, it was down to about 30. A few more fixes and voila, no errors. But that was on code that had semicolons on every statement. When I started to test examples without semicolons I ran into a bunch more problems that took quite a bit longer to fix.

I tried to follow a good refactoring approach, although, technically this wasn't refactoring because I was changing functionality. But I was preserving all the existing functionality. The basic plan was

1. Change the scanner to return a NEWLINE token instead of a WHITESPACE token for any run of whitespace that contained a linefeed or return. Then change the parser to ignore NEWLINE tokens.

This should not have changed anything - and it didn't. All my tests ran, and no syntax errors were introduced.

2. Add a variable to track nesting of (), [], and {} and ignore NEWLINE tokens if inside one of these. Also skip NEWLINE's after binary, and trinary operators.

Seems simple but this ended up taking quite a bit of fiddling to get right. At first, I was adjusting the nesting counter in various parsing methods. (Suneido uses a recursive descent parser so there is a method for each grammar construct.) But this got pretty ugly, so I ended up counting (), [], and {} in the method (match) that reads new tokens from the scanner. After this breakthrough (which, of course, seems obvious now) it went fairly smoothly.

When I released this version internally, we found two records in stdlib and a few more in other libraries, where there were no syntax errors, but the new interpretation was different from the old interpretation. For example:

return
    ... ;
used to be one statement, but now was two i.e. return ; ... ;

Another case was:

s = s
    .Replace(...)
    .Replace(...);
Which was now three statements instead of one. The solution is to put the operators at the end of the lines instead of at the beginning, which is our normal style guideline anyway.
s = s.
    Replace(...).
    Replace(...)
Overall, I'm pretty happy with this change. Personally, I'm so used to having to have semicolons in C and C++ that it's not really an issue for me. But I have noticed it can be a problem for beginners. And if you don't need them, why require them.

The next step on this path is to make braces optional and use indenting instead, like Python. One of these days when I need another fix of serious hacking...

Andrew McKinlay
Suneido Software

Just added blocks to Suneido. This is a Smalltalk idea. Ruby also has blocks although in a little different way from Smalltalk.

Basically, a block is a chunk of code within a function, that can be called like a function, but that operates within the context of the function call that created it (i.e. shares its local variables). But the cool part is that a block can outlive the function call that created it, and when it does so, it keeps the context (set of local variables) alive. For example:

make_counter = function (next)
    { return { next++; }; };
counter = make_counter(10);
Print(counter());
Print(counter());
Print(counter());
    =>  10
        11
        12
make_counter returns a block. The block returns next++.

At first, it seemed like a tough thing to add. One of the big issues is that function call contexts (Frame's) are kept on a stack because this is much faster than on the heap. But that won't work for blocks that outlive their function call. Smalltalk handles this by allocating contexts on the heap, but this is much slower than a stack. Then I remembered a discussion of this in ``The Design and Evolution of a High Performance System'' by David Unger. The idea was to use a stack for contexts, but to catch the cases where a block outlived its creator, and to move the context to the heap. This is what I did for Suneido. There were only two cases I had to handle - where the block was returned from the function, and where the block was assigned to the member of an object. As far as I can tell, those are the only ways for a value to survive a function invocation. It took about a day to add this, with a little refactoring of existing code, and of course, a test for it.

So now Suneido has blocks. Of course, nothing uses them yet, but I have some ideas.

Thanks to ajv I'm back to Journeyer and I posted my article. Hopefully someone finds it interesting.

12 Nov 2000 (updated 12 Nov 2000 at 18:18 UTC) »

Ouch, I'm hurt. I went to post an article and found that I've been down-graded from Journeyer to Apprentice so I can no longer post articles. It looks like one person certifying me as Apprentice did it.

I thought my last article was pretty well received and generated some good discussion.

Maybe someone is willing to take a look at the Suneido open source project that I'm lead on. Is 40,000 lines of C++ and 40,000 lines of Suneido code worth a Journeyer vote?

Oh well, I guess I'll go back to coding :-)

PS. Certifying yourself doesn't work :-(

Week 1 at Suneido.com

After Thursday's peak, the downloads dropped off to about 10 a day over the weekend, probably because the announcements all fell off the news pages. Of course it was a long weekend as well, but programmers aren't supposed to take weekends off, are they? One good thing about the reduced traffic was that I had less distractions and I actually got a "Getting Started" guide finished. And finally, after 8pm on Monday night, I uploaded the new version to the website. Tried to do a good job of testing everything, but it's always a little scary releasing a new version. Especially at this stage, when people are downloading it for the first time, you want it to work!

I'm hoping that the "Getting Started" guide will help people to get over the initial hurdle of a new system. There's probably more I should add to it, but it's a start. It also made me clean up a few odds and ends in Suneido because it was easier to fix the program than to document the awkward behavior! Being forced to work through building a simple application also renewed my perspective on Suneido. It's hard to see the big picture when you're buried deep in the code. Overall, I think it's looking pretty good. Still lots of rough edges and a few potholes, but definitely getting there.

I remembered NewsForge (www.newsforge.com) and posted an announcement there, and also got news items onto Dr. Dobbs (www.ddj.com) and Software Development (www.sdmagazine.com) so the traffic has picked up a little - 28 downloads today as of 4pm. That brings us to 184 downloads in the first week. I'm happy with that.

Whipped up a quick Suneido program this morning to send an update notice to everyone who downloaded prior to last night. Already had the SmtpClient we wrote to send the download replies so it was pretty easy. Almost forget to eliminate the duplicate emails, although there aren't too many of those yet. Easy fix, though, just had to add "project email" onto the query. Nice to be able to make use of our own tools, or as one of our visitors put it "eat your own dog food".

The forum seems to be getting a little more activity, which is good. But I've had some feedback that people don't like to use web based forums. Maybe a mailing list (or several) with web access to archived messages would be better. Hey, maybe we can write a Suneido program for that. :-)

Day 3 for www.suneido.com About 40 download emails waiting for me when I got in to work. Obviously, looking at each one and replying to them manually wasn't going to be an option much longer! Downloads were slower during the day, so we ended up with about 60 for the day, bringing the total to around 100, from 25 different countries. About 10% haven't entered a valid email address - a few of those are probably typos, but most are obvious refusals. I certainly understand people not wanting to get more junk email, and I appreciate all the people that have trusted us with their email address. We'll try our best not to abuse this trust!

By noon we had a Suneido program grabbing the emails from the POP server, sending the replies to our SMTP server, and entering the information in the database. I hope no one got multiple messages during the switch over. I want to set up a Library section on the website for sample apps (both ours and contributed) and we'll put this app on there in case anyone is interested.

More feedback trickling in - mostly by email - no one seems to want to use the forum. Of course, you have to register again to use the forum, which probably discourages some people. Our web guy is trying to come up with a way to combine the download and forum registrations, and use a cookie so you only have to register once. Of course, that risks the wrath of the anti-cookie contingent. Overall the feedback has been pretty positive, with some good constructive criticism. It should get more interesting as more people actually work with it.

The biggest issue so far is the lack of a tutorial and sample apps. Not that this surprises me; I knew we should have them. Just got too busy with everything else. It's at the top of my list right now. The problem is, I seem to be spending all my time watching the website and the downloads, responding to email, and fixing minor problems as they're discovered. The only time I've managed to find to work on the tutorial is lying on the couch with the laptop in the evening. With any luck I'll get something put together over the weekend. It helps to be a workaholic. (Maybe even mandatory!)

Day 2 for Suneido

30 more downloads bringing the total to around 50. A couple of people actually posted on our forum but it looks like that's going to be slow getting going. Got some emails from people who downloaded. No one has done much yet, but starting to get some useful feedback. I've been working on this so long that I'm don't really see it anymore. There's nothing like a bunch of fresh eyes to spot strengths and weaknesses. Of course, that's one of the powers of open source software.

Fixed a few minor problems with the documentation - some missing pages in the Database Query section. Updated the website version, I'll wait to update the download to see what else comes up. Started working on a Getting Started section to try to help new users.

Our ISP still hasn't got the stats working, so I still don't know how much traffic we're getting, or where people are coming from. I posted announcements on a few more newsgroups. Got an announcement on dotcomma.org It'll be a while till we get onto the search engines, although I'm not sure whether people use search engines to look for this kind of stuff. If anyone has any suggestions for places to announce/list Suneido I'd appreciate the help. Or if you know anyone who might be interested in the product, be sure to let them know about our website.

Day 1 of Suneido's public life. 20 down-loads. With no past experience to go by, it's hard to know if that's good or bad. But since the only announcements were on a few newsgroups, it seems okay. Those 20 down-loads came from 8 different countries - the global reach of the internet still amazes me. Got a few nice comments. Of course no one had actually tried it yet, or looked at the source code, so they're probably just being polite.

One of the down-loads was from David Ascher, a senior Python developer at ActiveState. It would be interesting to hear his comments. And, yes, I'm still personally looking at all the download registrations and replying to them by hand. The automated handling should be in place in the next day or two (written in Suneido, of course!).

I wonder what the next little while will bring? Unless we find some new places to post announcements, I suspect the down-loads will drop off. It would also be interesting to see the stats for the site but we can't seem to get them from our ISP yet.

Andrew McKinlay - www.suneido.com

1 older entry...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!