Older blog entries for uweo (starting at number 10)

The telekom installed the TAE for the DSL line on monday. Today QSC installed the DSL pipeline. It even worked ... 144kbit isn't bad.
The provider didn't get the hint on the IP address application (n IP address _plus_ firewall, so an additional /30 or a DSL pipe in bridge mode would have been perfect), but proxy arp is fine for this situation, so i went that way.

Changing the IP addresses of five machines was an interesting experience. If my customers told me that changing IP addresses is beyond them for some reason or couldn't be finished in the next few months ... in short, i really didn't believe them. Now it do.
Change IP addresses. Change main filter. Change DNS server address. Change multiple application and server configurations. Change multiple access lists (host.allow, tcpservers configuration). Tell the outside world about the change. Fix multiple access lists there. Fix a number of ssh configurations or authorized_keys since the reverse mapping is not working now. Detected an overly long /etc/hosts on one machine. It's quite late now.
I still have to move my brothers machine to the new address, but that can wait until he notices that something is broken.

Anyway, i'm quite happy about that. Seing that the telekom get's something right at the first time is delightful.

On the free software front not much has happened. I finished the basic design of the trouble ticket system. The only thing i left out was indexing - i'll first look at which kind of queries will be needed and then decide how to do that. Actually, this thing begins to interest me, and so the coding was quite easy (still unfinished).

The patch is now where it should have been: klogd_lose.patch.
Sorry.

I looked at a number of trouble ticket and bug tracking systems in the last two days. To put it mildly: they all lack.
I'm astonished that the majority of them, especially the newer ones, are web-based. Hey, people, get a clue:

• discussions with clients is best done through email.
• email has lower turnaround times.
• if done correctly email is more secure, especially if you don't use passwords. [use a one-time key for each trouble ticket, and let the customer decide with whom he wants to share this]. This obviously isn't that important in for open source bug tracking.

Well, maybe it's just me, but i don't want to use a web browser to update the state of a ticket after i resolved the thing together with the customer. I just want to set the state to resolved in the last mail i send.
Apart from that email is just more easy - to use and to program processors for.

And all i've seen seem to be a bit unflexible. I'm not going to introduce different task-systems for company hotline (non-software customers), software customer support and developers internal stuff. My cow-orkers would kill me before i finish the sentence, i think.
In other words: the basic design has to be very, very simple, but extensible.

There is nothing wrong with having a web interface (aside from the simple fact that i will not use it if i don't need to use it), but being forced to use one? Heck, no.
And you can't easiely put something else about a web interface, but you can create some kind of web interface for almost everything without too much work.

I also went through my personal todo list:

1. BBS software: decided to not work on it anymore. Not even bug fixing. It's not something i want to continue working on. I wouldn't design it like this, and the basic design is from 1988 or so (and the UI is even older). Period.
2. gateway: will rewrite it. The basic design is still ok, but i' m a better coder today, and have learned a few lessons. Beside that: I now have some libraries with i didn't have in 1993, and they _will_ make the thing more beautiful. But the change is quite fundamental, and anyway, the best way to get rid of bloat and historical workaround in code is to rewrite it.
   Schedule: don't know.
3. utftpd: i'll rewrite it. The configuration file language is fine, except there shouldn't be a configuration file language at all. Yeah, it's highly flexible. And this is not needed.
   I'll keep the one client, one TFTP server process design for now, but will cleanup the server if i happen to find something which wouldn't work in a single server design. I'm still unsure whether multicast TFTP is useful. Mind you, utftpd has been written to backup my router configuration files.
   Schedule: This year.
4. udpserver: decided to finish it. I need something like this to start the TFTP server.
   Schedule: Soon.
5. syslog server: decided to finsish it. I do not really need it, (except that linux sysklogd is horrible), but a clean and reliable implementation of a syslog daemon which doesn't fill the disk is worthwhile. [takes and translates network input, sends it to multilog]
   Schedule: Soon. What needs to be done is to write a useful configuration tool.
   Originally i didn't ever want to use it one any other machine then the log server at my workplace, but now i think this software might be useful for others, too.
   • speaking of linux sysklogd: That hack _still_ loses if the syslogd is restarted. I sent a fix for that to the authors / maintainers in 1996 and 1998. See this patch. Shall i be amused?
6. lrzsz v1: i'm tempted to remove all traced of internationaliation from lrzsz. gettext _still_ is at version 0.10.35, and still throws out that annoying warning that messages must not contain \r (which is wrong, btw). I would also get rid of a number of bugs in Makefile.in.in, of which the wrong usage or not-usage of DESTDIR is the most annoying. And building the package on a machine without GNU make will then be possible.
   No, i'm not volunteering to maintain gettext.
   Schedule: Still undecided.
   • in case you wonder: my personal build environment disencourages warnings. It's quite annoying to get them mailed twice a day.

7. lrzsz v2: Will possibly finish writing X/Ymodem sender and receiver tools this year. I'm not happy about the sender state machine now, it works, but it looks ugly. This needs more thinking.
   The Zmodem state machine will get even more ugly. I need to re-read a few descriptions of the protocol first.
8. watchdog: the watchdog software the company uses now could benefit from a cleanup. It now runs for more than 6 months, and all the problems with the design should have shown up by now. Actually quite a number did show up, and the ad-hoc changes not only made the code messy, but desynchronized code and documentation.
   Schedule: This year.
   Mostly easy stuff, but the scheduler ... at the moment the interval of 300 seconds is fixed. I should get rid of this, for cleanness sake. Oh, and i need to convince the boss to release it under GPL. Should be easy.
9. conflib:
   support discontinued. I don't use it anymore.
10. ftpcopy:
    i will implement some kind of "move" functionality -

           ftpcopy ftp://re.mo.te/ ./re.mo.te/
    
           (cd re.mo.te ; find . -type f -print0) | xargs -0
    
    
               ftpdelete ftp://re.mo.te/ 
    

    Not using -print0 is also possible. Files with \n inside the file name are then just not deleted. (i suspect that the are impossible to download anyway).

Netscape eats an astonishing amount of memory now. Oh, well. Maybe i should start to post with some other tool.

• What a week. Bug hunting continued. I'm used to read sources with difficult identation, but ... let's say it politely: one of my cow-orkers has a strange, very strange, way to indent his perl code. And a habit to let variable names and comments get out of sync with the code. Together with hurry, haste and brokenness (sp?) it may be the most difficult code i've read in the last 20 months.
  And database triggers are, well, somewhat counterintuitive.

  1. update table where key = x;
  2. if (ok)
  3.   delete from table where key = x;
  

  Bug? No, delete-trigger. No single word about it in the code, of course. Ugh.

• I didn't try to do much coding outside work. I'm feeling somewhat tired ... burned out? May be.
  
  1. finally updated the getopt clones documentation.
  2. updated ftpcopy to use the lastest strhash.
  3. added a missing \n in the ftpcopy logging output. It's one of those things which could have been easiely catched by a test suite.
  4. ^z or any other interrupting signal was handled as timeout.

• <rant>
  Why can't linux distributors, debian and *bsd maintainers send the patches they apply to the packages back to the author? Is it really so much fun to apply and fix the same patch to 5 revisions of the software? Or is maintaining paid or honored by the number of patches applied?
  </rant>
  

• LC_ALL=fi_FI logger test
  LC_ALL=C logger test2
  tail /var/log/messages
  Please ... someone please spend a cluon on whoever did that.

aaronl: Is it just bad mood or are you trying to discredit yourself totally? Technically you are right, but your behaviour ...

itp: I think you'll be surprised by the number of people who neither use or like GNOME (or KDE). Not everybody needs the value added by those toolkits.

mjs: You forgot to take some things into account, too. A better procedure may be to:

1. reboot.
2. make sure all libraries you can reasonably expect to be loaded _are_ loaded and used in a typical way.
3. free >>record
4. run the program / toolkit
5. free ...
   The difference between the last and this output usually is somewhat higher than the "real" memory usage of the program / toolkit and includes fragmentation and caching inside other services.
6. run the program / toolkit again.
7. free ...
   If the difference is about the same than step 2 was well done (it usually isn't).
8. repeat the last two steps until the result get's stable.
9. repeat the whole process a few times (including reboot, of course). Omit step 2 at least once.

This way you catch memory fragmentation (X server, window manager, kernel) and caching (x font server, x server, DNS caches, whatever).
Interpretation of the results is another story. Reducing them to a single number is useless or a lie in many cases.

Finished the xmodem sending. It works. It actually is even a little bit faster than lsx from the lrzsz suite. Why? I didn't think about performance ...
Error handling is quite simple: count the number of errors per block. If it reaches a fixed number then abort. Good enough? At least not worse than lsx.
Thought about doing a fallback from 1024 to 128 just a little bit too late. Well, no other xmodem implementation i'm aware of can do it, too, but that is no excuse. I feel a bit stupid. But on the other hand: it's a prototype, and if there is a reason to prototype something then it's to show stupidities early, right?

• No, i do not believe in rapid prototyping for low to medium sized software. Actually i do not believe in rapid prototyping at all. Prototypes are useful, especially in major projects, where one want's to get things into a somehow useful state very early. But rapid prototyping?
  People should think, especially about the major design decisions. User interfaces, the most prominent victims of rapid prototyping, are major design decisions. Rapid thinking is a contradiction in terms.
  I also think that rapid prototyping often leads to omitting testing of the modules. The desire to put things together early and see whether it fits seems to be very strong, stronger than that of testing for correctness.
  </rant>

Speaking about xmodem: Now, some hours later, i still don't understand why the fallback isn't implemented in any xmodem sending software i know. There are still devices left which don't accept more than 128 bytes per block, and a fallback mechanism would get rid of that interoperability problem. There has to be a drawback?

Anyway, it's time to stop dealing with computers for today.

Todays lessons:

1. options have to be implemented carefully.
   I phoned one hour to do support for a project i didn't work on for more than one year. I still use that software, but i'm not feeling well about how things look out.
   
   • too much old cruft to work around stupidities in other software, which mostly have been fixed a long time ago.
   • too many configuration variables. I didn't count them, but the number may be near 100. If i'd add the number of possible command line switches and configuration options in secondary configuration files then the number might be near 200.
     Needless to say: i don't know them all. And i'm not sure that i know what those switches do i still remember.
   • too many hidden dependencies.
2. bloat is evil
   Speaking of old software: i did the same mistake at least twice in the old BBS times, in the BBS software and the message gateway. Inexperience? Featuritis (the disease of adding features which aren't really needed)? Possibly both.
   It's astonishing hard to bring the code back into shape. To remove a feature someone might depend upon is not an easy decision.
   Bloat also tends to make the source nearly unmaintainable.
3. too much is too much
   i've never stopped supporting on of my projects before, but the time will come that i will have to. The BBS software, the gateway, lrzsz, utftpd, ftpcopy, plus a number of libraries, plus running a BBS, plus a new project, a news server, plus some minor things.
   We officially started working on the BBS around 1992-05-27 (did some tools before that time). Developing and running it has been fun. There are still a few BBS running this software, and quite a lot of users. But it isn't fun anymore. It was teamwork, but now i'm doing it alone. And it seems people just want to use and talk about development, but don't help. Oh yes, they all know enough to decide what language has to be used because any other is just broken.
   But anyway, it's not an easy decision: If i want to continue running the BBS i'll have to either continue doing at least some software maintainance or have to switch software to the only alternative solution for that BBS network (which more urgently needs even more development). I would have to give up both, and then i also could stopping to work on the gateway software. Oh well, to work on this one still is fun, but i don't feel this will get any priority soon.
4. Speaking about TFTP: Let's suppose inetd sees 5 or 10 TFTP requests coming in every second. It forks a tftpd process, and then has to wait until that one has read the packet, forked again and exited (the freshly forked process continues on another port). Then inetd forks the next tftpd. Nothing really problematic? I thought so, too.

   There actually is a problem. tftpd does a getpwnam(). The machine i did some tests on today had a few thousand users in /etc/passwd, and the high numbers, like nobody, come at the end. The getpwnam took quite a bit of time, about 0.1 seconds. The whole select / recvfrom(PEEK) / fork / host.allow / exec / recvfrom / getpwnam / fork / exit() cycle took almost 0.2 seconds.
   This got interesting when a whole room full of equipment needing TFTP access was booted: 15 machines requested images or configuration files in the same second. And quite a number of them seem to have a timeout of one or two seconds (which is stupid). That was not all, some machines took longer to boot. About 50 machines requested their configuration within 10 seconds.
   Temporary "solution": set[ug]id(hard-coded-number), IP filter instead of hosts.allow.

   long-time solution: inetd "wait" mode has to die. inetd should read the packet, create a pipe and feed the packet to it's child through the pipe (it also can set some environment variables contining IP addresses, port numbers and hosts names).

   btw: utftpd supports a "user.group" notation, meaning it does an extra getgrpnam if this is used. No wonder that it was even slower and got more problems. It also supports numbers, but they weren't used.

   btw: it wasn't too funny to debug that problem on a machine in japan. Doing useful work with 200 ms turnaround time can't be called recovery.

In addition i played around with xmodem today. I'm trying to find some kind of design for a X/Y/Zmodem library, which does the state machine stuff internally. I want the library to just to I/O, and return to the caller as soon as this is done. The caller then does whatever it want's too, finally calls select or poll, and returns into the library as soon as there is something happening on one of the file descriptors.
I'd bet an euro that the state machine for zmodem will be very "interesting", so i played with xmodem.
One question remains: why?. The answer may be pretty simple: "because it can be done" (is this really a good answer?).

Things got worse. Really worse. Three problems more and not even one solved. I left for the weekend after only ten hours, took Tronn with me and brought him to the railway station, which really wasn't a bright idea: He couldn't stop talking about work.
I then went to a lake, to swim a round. This time something was OK: i expected and found the water to be cold, but not too cold. Fine: i'm feeling well now.

I'll have to decide whether ftpcopy shall learn to delete files it copied from remote. I dislike to do this since ftp is fundamentally insecure, but this is the most often requested feature. Which is quite strange.

lrzsz hurts (?) again: in the last months i got a bunch of reports that it doesn't seem to work over telnet connections anymore ( it used to work for exactly the same people it doesn't work anymore for). I'm quite sure that people just need to tell their telnet clients to properly disable certain misbehaviours, but maybe i'm wrong.
I also got reasonable requests for more useful logging and status display. The problem here is that i don't want to break things / change behaviour before i do the rewrite (which may never happen. I wanted to start that for more than one year now, and i don't see that i'll get the time soon). I think i should better keep it stable ...
This needs more thinking.

Well, i suppose the answer is that i need the time for the other thing i'm working on.
And that i need to remember that ftpcopy is meant to mirror something and not meant to be the last word in overfeatured FTP clients. What about another client with the ability to delete if a file has a certain size and modification time? Some shell script then could decide to delete files after ftpcopy ran ... and the script might be even smarter than ftpcopy ever could be. I guess that's more like UNIX.

This diary might even turn out to be useful. Interesting idea, indeed.
Thanks, Raph.

Two horrible days at work. I found a problem 4 days ago, and i thought it would be a major one. But i found a different thing 36 hours ago, and the first one now looks very small in comparision. Two quite simple mistakes, but the software can't recover from that automatically ...

I'm tired - two long days, and the last night i only slept for about 4 hours, thinking about code for a few additional hours, which didn't make me feel good (especially as it was that code). Let's guess: at the end of the day i'm even more tired, and have done a large step towards a coke addiction.

On the bright side: the code in question is not my code. I just happen to not be on vacation. Bad luck, i guess. Or is that piece of software really that bad that it needs hand-holding and band-aiding at any time, not only during the developers vacations? [i'll possibly think different as soon as i'm feeling better again, but i must admit that a bit of flaming makes me feel better now]

In summary: this might cost me more than a week of time. It will delay the update/rewrite of my software for about two weeks. I'm behind schedule anyway (schedule? Forget about it, there is not time-table with the slightest connection to reality for that project. Take version 1, using developers 1,2,3, protocol X, languages A,B,C. make version 2, doing everything right this time, using developers 1,2,3,4,5 plus a few people with additional ideas and opinions, protocol Y and languages A,B,D. Change Y to Z long months after the project should have been finished. Oh yes, 1,2,3 are quite busy doing other stuff. 4 and 5 are quite new and at least 4 of the 5 aren't good teamworkers. Second system syndrome alert. Death march alert.
And i don't even get the time i need to work on that!

It's time to ask for my interim report, i think.

I'm really tired.

Yesterday i somehow managed to get about an hour to work on the string hashing library. Surprisingly enough one hour was enough to do a few space optimizations and package the whole stuff.
I sent Fefe a notice about it, he might be interested to include it into his libdjb project.

Today, on my way to work: well, i now have the dynamic string hashing library i wanted to write for about a year. Fine, but what about a fixed size record hashing library? The string length makes for about 8 bytes of overhead per record now.
Let's see whether there is a year between idea and realization this time.

Later. Fefe answered, he is interested, and is also interested in the getopt clone. This means i have to clean it up ... and update the documentation, which might be a good thing.

A completely unrelated note: advogato lacks a spell checker. Ok, that's not fair: improper spelling is my fault.