Older blog entries for MichaelCrawford (starting at number 43)

Back online after two days of vacation. I stayed home, but my wife doesn't like me to get online on Christmas Eve or Christmas. I turned off all my computers, even my network gateway (a Mac 8500 running Debian Woody PowerPC) that was starting to get a respectable uptime. The silence in the office was eerie. One of my long-term goals is to replace all my fans and disk drives with quieter ones.

Happy news: there was an email waiting in my inbox from a potential client I've been talking to for a couple months. He still wants me to work for him starting in January. Things have been pretty tight for a while and this new work should put me back on track. My wife was very relieved when I told her. It will be more embedded work!

My mom gave me a T-shirt that read Real Men Shop on December 24th. How true, or at least it used to be before I moved far away from everyone and have to ship the gifts I give.

There was a fierce storm last night (Christmas night), the wind was howling and the snow fell horizontally. I got up today to find bright sunshine and eight inches of fresh snow on the ground. The depth of the snow gave the snow plow man such trouble that he didn't get to our house until 5 pm, so I didn't get to go out to run the errands I planned to do today.

I find it really odd to live in a place where the removal of frozen water that falls from the sky is a significant economic activity.

Threads vs. State Machines

I can see the advantages of using state machines over threads, but I don't see how one can get operating system support for distributing a program across all the processors in an SMP machine if you're using a state machine that has only one thread of execution. Perhaps someone can clarify this.

I realize threads are implemented in an OS kernel as state machines. What I'm asking about is how a user program can do SMP via state machines without the use of threads.

Saying to run two state machines in different processes isn't the right answer. That's the same as using two threads and presents all the same difficulties.

I agree that threads are resorted to too quickly by the lazy who don't treat them with the respect they deserve. Threads present a lot of problems and it takes care and skill to deal with them properly.

I wrote a test tool at Apple when I was one of the QA engineers for MacTCP, Apple's original TCP/IP network stack. Macintoshes didn't have threads or even proper preemptive multitasking. One could run interrupt tasks and do asynchronous I/O.

I implemented it as a state machine and it worked very nicely. Each Mac could have 64 TCP streams or UDP "pseudo-streams" and all the Macs on a network could communicate in any combination to create a pretty fierce amount of I/O. Without state machines that would not have been possible.

State machines tend to be much more conservative of resources than threads usually are. On the BeOS, each thread has a fixed stack allocation of (I think) 256 kb, and that is often very wasteful.

Hacker Angst

Sometimes when I have some particularly difficult work to do, particular when I have something totally new to work on, I just can't face it. The thought of even so much as opening any of my source files fills me with dread. It stops me cold.

Usually what I do is go to lie down for a while. Sometimes I take a nap. It's the worst feeling.

When my wife sees me like this she tries to encourage me to get back on the horse. It's pretty easy for her to tell that I'm in this state because I get up abruptly from my desk only to lay on the couch or bed.

Usually the best thing to do is try to find something easier to do to get myself back in the rhythm. Any work at all as long as it's somehow productive.

I hit this wall tonight with something difficult I planned to do. I have a programming task that won't be a lot of code but it's not obvious to me at all how to do it. After taking my usual time out I decided I would spend some time writing test code first and doing some testing. I probably won't even get to the difficult work tonight. It only matters that I get it done in the next week or so, so I should be able to work up the momentum.

20 Dec 2002 (updated 21 Dec 2002 at 07:04 UTC) »

Updated: Embedded Systems Programming

I've been up all night working on a really whacky bug that cropped up yesterday. After backing out some recent changes I decided that I must have inadvertently introduced a bug in some working code while I was working on some unrelated code. My guess was the I typed something into a source file accidentally and it happened to compile but caused trouble.

So I decided that rather than track it down, I would restore a backup from before my latest work, make sure that code worked, and then merge in my latest changes.

But that didn't work.

I went to earlier and earlier backups, becoming increasingly dismayed and to be honest rather panicked.

At some point I tried out a known good binary that had been a test build I delivered to my client that I knew worked, that my client said worked for him too. And it had the same bug.

That's when I began to suspect I had an electrical bug.

After some examination I discovered that a ribbon cable wasn't plugged in quite right, its connection was at a slight angle. It looked like it should have been OK though. But I popped the cable off and plugged it back in again.

Bug fixed!

It is now 2 a.m. saturday morning and I'm now ready to start productive work for the day after spending all day trying to fix a hardware bug in my source code!

Live and learn, I guess.

Back to our regularly scheduled program...

Well, I'm pretty close to completing the embedded systems project I've been working on the last couple months. It's not quite done yet, and one really significant challenge remains, but it should be done in a week or two.

That means I'm going to need to get new work.

I have been in discussions with someone else for another embedded project that I would be quite stoked to work on. The guy has told me several times I am his top candidate to do the work, but it is not yet a sure thing. If I do the work it will start in January.

I was pleased just now to get a call from someone who saw my antry at MacTalent and was very interested to have me do a six month contract. But the rate he suggested was pretty dismal, and I'd have to live away from home for six months.

I told him what I would rather get for a rate, and he said he'd see what he could do (I think he is a headhunter). I also said that that rate would only apply if I could work out of my own office, to work away I would also need my extra living expenses covered.

I told him I'd maintained my own office for almost five years and had been very successful at it. It seemed like it hadn't occurred to them that someone could do the work at a distance. He actually seemed pretty open to it. So it's a possibility.

This embedded project I've been working on, I think it has been the most difficult programming project I have worked on in my entire life. I have had bigger jobs that took longer, but nothing that required the kind of effort and mental strain that this has. But now I know a lot more about how to do embedded work than I used to, and I feel much more comfortable now working with hardware.

Ironically, one reason I proposed it to my client was that I thought it would be really easy to do. I thought I could have a shippable product in no time at all. That hasn't been the way it turned it, but fortunately my client has maintained his enthusiasm for the product.

Hopefully I won't have to keep the secret of what my product is for much longer. The plan is for my client to demonstrate it at his booth at the upcoming MacWorld Expo in San Francisco.

Sometimes my wife expresses great distress at how tough things have been for us, and one way I try to reassure her is to tell her that things are tough for everyone. We are doing better than most - at least we are both working. I see posts from time to time here and elsewhere from people who are out of work, some who have been out of work for a year.

My wife has scored a contract with a second client for her web design business. She is quite stoked.

Her first clients are a folk-rock band, and at the release party for their recently completed CD, the band members publicly thanked Bonita for doing their site. This prompted one of the other women at the party to approach Bonita to do a site for her store downtown.

Word of mouth really is the most effective advertising. The challenge is in getting it to happen.

I'm proud that I got Bonita into using cascading stylesheets. Now she knows a lot more about CSS than I do.

I'm probably going to do a little web application programming to help Bonita out on this job, writing a small program to allow the store owner to easily list her inventory on her site without having to know how to code HTML. Very likely there is some open source application that does this.

Brother, Can You Spare a Clue?

I have made considerable progress at improving the performance of my embedded application. Using a better algorithm (what I believe now is considered the fastest algorithm) in C got me a factor of ten improvement. Re-coding that in assembler got me another factor of three improvement.

The problem is, it's still not fast enough to meet the client's requirements. I need to increase the performance by another factor of three. I believe I can do this by tweaking the assembly, but it's not at all obvious to me how.

That's why I submitted an "Ask Kuro5hin" on the topic of ARM Assembly Code Optimization?

I felt the discussion would be more useful to others, and likely to result in more ideas that would inspire me to solve my problem, if I encouraged people to post performance tips for any architecture.

I mention in the article that I don't have the budget to buy all the books that have been recommended to me. One reason for that is that I rather dramatically underestimated the time it would take me to deliver the proof of concept, and I had agreed to do that work for a fixed price (being under the impression it would be so easy!). But the happy news is that I asked my client if he would be willing to help me out with the cost of a couple of books that people recommended to help me speed up my assembly, and he agreed.

After a couple hours of calling around (and being disappointed that online bookstores won't ship on Saturdays) I got a bookstore a couple hours from my house to put on hold for me:

My friend Dave Lyons, who recommended Hacker's Delight to me, said that he was quite suprised by what it showed you can get a two's-complement execution unit to do.

Embedded Systems Programming

I worked through the night, night before last, to speed up my code, and hit what I believed at the time was a brick wall. The kind of brick wall that would make it impossible to get my product fast enough to bring to market.

But after I finished up for the day and spent a few hours just relaxing I thought of a possible solution. I don't know if it will work, but it could work, and maybe if I can get my solution working at all some further work will make it fast enough.

I was eager to implement my idea yesterday, but I hadn't slept and was exhausted. It was very hard to bring myself to go to bed. I managed to, but I woke up at 2:30 am today and got up to go to work. Even though I had gone to bed in the late afternoon, it's not like me to wake up so early and go straight to work. That's how badly I want to succeed.

This embedded project has been a Trial by Fire. But it still has the potential to pay off well, and the painful experience I'm gaining should make future embedded projects a lot easier.


Zach Frey very helpfully wrote to me about my last diary entry where I discussed the tension between my struggle to make a living and my desire to do things that I felt were more important like writing and political activism.

He referred me to a speech C.S. Lewis gave to some British University students at the outbreak of World War II, in 1939. The speech is called Learning in War-time.

A Google search for "Learning in War-time" doesn't find the complete text online but there are a number of quotes available that I have found helpful:

The war creates no absolutely new situation: it simply aggravates the permanent human situation so that we can no longer ignore it,..

Human life has always been lived on the edge of a precipice. Human culture has always had to exist under the shadow of something infinitely more important than itself. If men had postponed the search for knowledge and beauty until they were secure, the search would never have begun...

We are mistaken when we compare war with 'normal life'. Life has never been normal.

I'm going to go look for the full text of the speech in the library and a couple bookstores some time in the next few days.

Zach also wrote about the importance of what I do in the way of making a living, that this advances the cause of Freedom - working independently as a consultant, and making what contributions I can to Free Software.

My long-sufferring wife understands why what I do is so important to me, and she supports and encourages me as best she can, but the way my other activities sometimes threaten to bring us closer to the precipice often frighten her.

I wish I didn't cause her so much pain when I take off to do something that I'm not getting paid for. I wish the way I do make my living were much more secure, so that when I am working billable hours, there would be more certainty in our lives. Usually I make enough money from my work that I can afford to take take off now and then to write, but it's never certain at the time I do it that my consulting is going to work out well.

Very proud to have done my small part

So I was hanging out on Slashdot for quite a while during the discussion of Because only terrorists can use 802.11 because I felt it was an important opportunity to get the message out about working together to oppose the terrible course the U.S. has been taking since September 11.

But my wife was distressed because things are very difficult financially and I need to get a bunch of hard work done in order to get ourselves out of this jam.

The more upset she got that I should be working instead of raising people's consciousness on the Internet, the more upset I got that she was trying to prevent me from doing something that I felt was of signficantly more importance than spending the night hacking some code.

We had a long talk about this eventually. It cleared the air quite a bit.

My wife is a very apolitical person. She prefers not to read the news because it just gets her down. She doesn't feel that the individual has much power to make a real difference. And she is very frightened about how we are going to pay all these bills that are overdue.

I, on the other hand, read the news pretty obsessively and am very frightened about what's going to happen a year or two down the road if Bush, Cheney, Ashcroft & Company succeed in what they are up to.

I worry that saying stuff like this could get me imprisoned someday - or worse.

Many of you might feel that that's an absurd thing to say, but I say that it's only relatively recently in history that many people at all have been able to speak their minds without fear of reprisal, and many, many people even today live in places where speaking out can mean a sudden disappearance.

I feel the only thing that I can really do to prevent that from happening here is to continue to speak out.

After reading some of the comments posted in reaction to posting the essay at K5, I feel that it's even more important that I continue to speak out. A couple of people raised the point that the Left is rather lacking in people able to speak eloquently. Or even coherently.

I feel that my ability to reason with people through my writing is a gift that I should be using for the common good.

The main reason I am a Green and no longer a Democrat is that the Democratic Party has shown itself to be completely spineless as far as mounting a credible opposition.

If my writing is able to convince even one person to stop what they're doing, to speak out, and to take action, then I think it is very important that I write. I think writing "Is This the America I Love?" is one of the best things I have ever done. I feel I need to do more of it.

But then there's the problem of all those bills. I need to keep food on the table and a roof over our heads. If I don't keep producing as a programmer, I won't have to worry about bootheels at our door coming to arrest me, but about the bank coming to repossess our home.

But I told my wife that if I had to give up doing the things that I felt were the most important to me just to ensure our survival, that there was just no point to even continuing, I might as well give up.

Our life has been a constant struggle the last several years. But if the only thing I could do was to struggle just to survive, and not to pursue the finer things, then it would break my spirit.

To give up my writing, and my activism, would be to lose the struggle. It would be worse than to be hungry or homeless. It would be to have no reason to live anymore.

There has to be something better than just struggling to get by. There has to be a reason for what I do beyond mere survival.

For me, that reason is to write.

But I wish it weren't so hard on poor Bonita. I really do. It breaks my heart to see her so frightened when I take a night or sometimes a few days off to write. No one pays me to write. Writing doesn't put any food on our table and it doesn't pay the mortgage - but I think it makes the struggle to get food on the table worthwhile.

Thank you for your attention.

I have gotten my embedded product working quite a bit faster than it was. Recall that the demo I gave my client was way too slow and it took some convincing to get him to continue funding the product, but he did eventually decide to continue paying me for development.

Unfortunately, it is still too slow to be marketable. However, it has become suprisingly usable even for the performance that it does have. Before it wasn't usable at all - I could demonstrate that it worked correctly, and measure its miserable speed, but that's about it.

I can't claim that much credit for the speedup. It turned out that there was a more efficient algorithm for what I'm trying to do, it gets the same results but is much faster. However there was the challenge of getting it to actually work in my firmware, which took a couple of days. I was very relieved when I got the new algorithm functioning and found that a significant improvement had happened (a speedup of about 20 times from what I had).

But I need to speed it up by another factor of 10 before my client will consider marketing it. And at this point he's only committed to paying me to improve the speed, more work will be required to bring the product to market and I have to hit a minimum performance threshhold for that to happen.

I have always been a big fan of tweaking code for performance. I have yet to find an optimizing compiler that I cannot beat by writing better C. And I am always able to beat my best C performance by writing in assembler.

Everyone says that you should only try to improve performance by selecting a better algorithm, but what if you are already using the best known algorithm and it's not fast enough. That's when you need to tweak your code.

Also some algorithms are only more efficient when they are used on large data sets, because they are expensive to set up. If you have many independent and small data sets, what would ordinarily be considered the best algorithm may really be a poor choice. In that case an algorithm that scales poorly but can be coded tightly may be a better choice.

I'm afraid that because I considered performance not to be a real concern for the proof-of-concept demo, I didn't even try to look for better algorithms when I wrote my demo, because it simply didn't occur to me that the straightforward and obvious way could have had such disastrous results. I've done some research now and I'm pretty sure that the algorithm I have now is the best available.

I don't think it will be too hard to write it in assembly and there are a number of ways that I can see that this will help. I should be able to get some performance gain easily. Getting enough may be difficult but I still have several days left of the time I have funded. I could take more time if I needed it but it would make things unpleasant financially again.

Again I'm sorry I won't say what exactly it is I'm working on but hopefully it will only be about a week and a half until the product is announced.

What Kuro5hin members are saying about my essay Is This the America I Love?:

Alright, who let this intelligent, well written, reasonable person into the Left?
-- wiredog

Michael better be careful, or people are going to start thinking that Greens are capable of rational thought.
-- wiredog

very well written and articulated, profound and thought provoking. Plenty of information and examples backing your opinion.
-- theforlorneone

Thank you for your attention.

5 Dec 2002 (updated 5 Dec 2002 at 07:09 UTC) »

Updated: softkid, yes, implementing the performance-critical part of my product in assembly is one of the things I'm doing. But it's not completely obvious what the best assembly is. So one of my tasks this week is to experiment with different alternatives in assembly code, and understand the performance of each that I get.

One problem is that I haven't done assembly in a long time, although I used to do a lot. Another is that I haven't ever programmed the assembly for this instruction set architecture before. I've done well at learning it so far, but of course I don't know all the clever tricks yet. I used to write some really odd 68000 assembly code, stuff that no one could figure out just by reading it, but that was significantly faster than the best I could ever get in C.

There are some tricks to what I'm doing that turn out to speed it up quite a bit, that I was unaware of when I wrote the proof of concept demo. I had known that there were some well-known optimizations, but didn't look into them because I didn't realize they would be necessary even for a proof of concept. I had no idea that the straightforward, obvious way to implement this would be so slow. But I am aware of some of the optimizations now, and am researching others.

I'm working on a new implementation in C that should be a lot faster even that what I had in my first attempt at assembly code. When I get that working well, I will have another go at assembly and probably should be able to improve a little on the C.

Again, I'm sorry to be so oblique about what my product is. But it's something I think you will all be happy is available. It won't be expensive, and maybe some of you will want to get one. Maybe I can even ship it in time for Christmas.

Happy News!

My client has decided to fund continued development of the project I'm working on for him. As I previously reported he was testing my proof of concept demo before deciding whether to go further, but the performance was unsatisfactory, not fast enough to make it a marketable product..

I spent several days explaining to my client how I intended to speed it up, but some of those ways will be quite difficult and would take some time, and things are so tight financially that I could not afford to just implement those changes and prove it to him by showing him working code. He based his decision on his faith that my plans would work out.

Ordinarily I enjoy doing optimization. I think it is the best part of programming. I have some tricks for optimization that some people think are quite twisted, but have worked well in the past.

In this coming week I face one of the most significant challenges of my entire career. I have to make my code go so much faster than it does now that I really honestly don't know if I can succeed. There are some tricks I could use if I were running on a desktop PC but I am programming an embedded device with very little RAM or ROM.

This week will get funded, but whether funding for the product continues to bring it to market depends on the results I am able to achieve.

It would be fun if there weren't so much riding on it.

Sometimes I am able to forget about the peril we are in and just enjoy coding. The whole experience has been very hard on my poor wife, who asks how I can be in such good spirits during a time of such turmoil. I tell her that is is like programming with the Sword of Damocles hanging over my head, and my only hope for survival is to write good code. But I cannot write good code while I am frightened, so I do whatever is necessary to put it out of my mind.

34 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!