Older blog entries for Zaitcev (starting at number 153)

In kernel 2.4.18 or so, if two Token Ring boards are used in a SMP computer, a lock-up happens. IBM asked me to look at a patch by Denis J. Barlow which was said to fix the lockup. The patch was too big and touched too much code, so I figured what it did to locking and wrote a small patchlet which fixed it. Today I received a mail from Arnaldo (the maintainer of weird networking stacks) which said that someone else did exactly the same thing for 2.5 already, so he is taking the patch and sending it to Marcelo. In a year or so we'll know if it actually worked...

I am a little sad to see all the rest of Denis' patch to fall by wayside, but he is stubbornly refusing to split it into manageable chunks.

Another moral of the story is: Avoid exotic locks, such as spin_lock_bh(). The "my" patch simply replaces spin_lock_bh with spin_lock_irqsave. Why did the author use spin_lock_bh in the first place? I place the blame squarely on premature optimization.

Somewhere in August, Robert Love wrote an article in Linux Journal about Linux locks. It was an honest article, which listed all lock varieties and even tried to explain how to use them. When I read it, I thought it was a wonderful article; the only omission I noticed was that he did not mention that semaphore down() cannot be used in an interrupt. But guess what, about a month ago, Grant Edwards, a comp.os.linux.development.system regular, made a posting where he told he was going to use exotic locking. He wrote that he went through RML's article, and so, everything was going to be ok, because he'll use those correctly. I had to step in to direct him to sane locking. Of course, his code would be correct - in that particular version of the kernel. One step left, one step right, and it bitrots. Just like Token Ring did.

I extend the KISS phylosophy way beyond spin_lock_bh, into the realm of reader-writer locks. They can improve performance of contented locks a lot, but they are rife with pitfalls. In this regard Bill Irwin grieves me especially, because he advocates using reader-writer locks by default while being sharp and respected. He probably corrupted minds of hundreds of newbie driver writers. He does not understand that they are not ready for the sophisticated stuff.

Newbies need something simple instead, which may not be the most scalable, but working robustly, like a set of these "Zaitcev's Commandments":

• If you are in a process context (any syscall) and want to lock other process out, use a semaphore. You can take a semaphore and sleep (copy_from_user or kmalloc(x,GFP_KERNEL)).
• Otherwise (== data can be touched in an interrupt), use spin_lock_irqsave and spin_unlock_irqrestore.
• Avoid holding spinlock for more than 5 lines of code and across any function call (except accessors like readb).

That would be it. Just keep it simple. Any of these rules can be broken under some circomstances, but the problem is that newbies cannot judge adequately. This is why they are "Commandments", to prevent them from confusing themselves.

Today, Advogato article system was DoSed by our local DCE and Microsoft worshipper. It was pretty funny, because it reminded me about my buddy Sergey, who went to work for Microsoft and got assigned to support DCE. His comments about the code were quite entertaining, if a little harsh.

Couple of days ago I sent a patchlet for USB hub code to gregkh. The fix was about 10 lines big and it would not deserve mentioning but for its amusing work and social context.

It was known for a long time that the debouncing loop was broken from the conception, but nobody did anything to fix it. An incompetent programmer somewhere eventually came up with a patch, which does not fix the algorithm, but just short-circuits the loop instead, letting it to proceed. This fixed it for people who were frustrated by the original (broken) loop implementation. The patch began to float around, and one day SuSE began to ship it. Eventually, a gentleman from Khroatia came to the linux-usb-devel and wrote "This patch fixes my problem, SuSE ships it, so be so kind and include it in the mainstream kernel at last."

This was sooo wrong on so many levels, that I felt compelled to fix the darn thing. Also, I got some negative lessons, which may be obvious to some, but perhaps my elitism and snobism blinded me.

• Some time ago, if a person could formulate a patch with correct argument order for diff, I would assume he or she read the code, at least. It is NOT SO.
• (Corrolary: there is a swarm of garbage patches floating around).
• Lots of people mastered the skill of downloading and applying patches without a foggiest understanding of the code.
• Even big and respectable vendors do not read crap which they ship with their kernels.

Regarding the last point, I'm not saying that SuSE is particularly bad, or Red Hat is good in this respect. We always review patches we ship. This should be sufficient, right? Well, in case of the debounce loop, the garbage patch floating on the 'Net was so small, that it provided no context. It was pretty hard to tell from three lines up and down what it actually did. I caught it because I "own" the USB subsystem in Red Hat, so I am somewhat familiar with the code. We also have owners of other major subsystems, such as scsi (Doug), IDE (Alan & Arjan), filesystems (SCT & Al), processes/signals (DaveM/Alan), scheduler (Mingo), networking (DaveM). These people zealously guard their territories from infiltration of rogue code. But what if someone posts a patch for IEEE-1394 which fixes a problem on someone's box? Or same for ISDN? Thankfuly, we have a gatekeeper (Arjan), who is good at saying "no", but this is a personality. He may quit one day.

The answer is, of course, push everything through the community, and resist additional patches as much as possible. The pressure to include is strong, however. I remember a public outcry when we shipped 81 patch RPM for 5.2 or something like that. Despite Arjan's valiant efforts, we now ship about 300, and everyone appears resigned to it. But that's not the limit. United Linux pushes 800! Worse, Linus preaches vendor kernels in interviews. I dunno where this is going to end, I really don't.

Gloomy today, am I.

P.S. Do not download and apply crap patches without looking at the code they are touching.

I had to touch s390 again after a long break. Guess what I found. Marcelo 2.4.20 does not compile due to a syntax error in the chandev. And nobody posted anything. The stupid mainframe simply has no community. Nobody ever tests whatever is in the open. IBM produces "drops", which vendors incorporate in big chunks. The result is the proprietary software development model which happens to produce GPLed code.

Today I managed to do a little sparc work, applied Eric Brower's patch to support nodes with "interrupts" property, but without "intr" property. Also did other small things around serial ports, etc.

The incredible shrinking Red Hat Linux

To my amazement, I just found that RH8.0 takes about 27% less space than RH7.3 (860MB vs. 1200MB) with essentially identical installation on my laptop. Of course, I have only myself to blame for even paying attention to this. I partitioned the disk in the way which allows me to test betas painlessly and I thought that 1300MB system partitions would be plenty for the life time of the laptop. The 7.3 release got me thinking about repartitioning and moan about bloat at mailing lists.

UML, what UML?

I found that we shipped a broken User-Mode kernel in RH8.0. The bug was pretty easy to hit, and known to the UML community. Jeff Dike told me what to fix, so the patch is in the CVS already, tested to work, and I'm running patched kernels already.

But this got me thinking. Nobody, not a single customer posted an issue to issue-tracker, or a bug to bugzilla. This can mean only one thing: nobody is using UML. Why is that?

If we consider rumors, the main reason is that the jail mode is slow, and "regular" mode provides no security. Thus, UML cannot be used by ISPs to hand out shells. Without the proper security, UML is nothing but a debugging tool for kernel hackers, and those just ignore what we ship and download all latest anyway. This is all speculation, of course. Perhaps we just have a solution in search of a problem here.

Back from the Red Hat World-wide Engineering Meeting. It was fun. Here's a couple of observations.

Our core development departments apparently follow a staffing model "hire superstars, and they'll work it out", which of course is helped by the overall state of U.S. economy. The rest of the company gets to collect all the good folks who had enough of it with proprietary software from other places, in particular from DEC's campuses.

The collorary to this is that it's pretty hard to place even solid performers who I know personally, the competition is so stiff. I seriously fail to understand how I was hired in the first place. Also, it is very pleasant to work with people who, at minimum, do not need elementary things explained, and typically are better than that.

I found more pilots in the development department, although most struggle to stay current in respect to number of landings, BFR, and medical. Tough life...

Matt Jacob presented me with a SPARCserver-1000 (Scorpion), so now I have no excuse but to support sun4d. Uh-oh...

Great many thanks for Matt!

I am off to Red Hat Worldwide Engineering meeting.

Seems like I found a yet another deadlock in the 2.4.x quota code, which is not unexpected. Generally speaking, filesystem operations may poke quota as a side effect, but quota itself has to call filesystem to update the quota file. The result is an invitation for deadlocks. The one I am talking about involved 3 (THREE) processes, forming a circle. Who could have thought that such things occur in real life outside of CS classes with their dining phylosophers :)

Quota was pretty buggy in all UNIX-es which I encountered, and Linux is strangely reliable in this area. Perhaps this lures users into thinking that quota is something that can actually be used.

BTW, quota in Linux bears the dubious distinction of the only piece of core code lifted wholesale from BSD by some Russian guy. I am wondering how BSD people manage their deadlock city.

hacker sez...

Which brings me to my second rant.. car manufacturers. For the same reasons.. in the US, there is no state that allows you to exceed 65mph that I'm aware of, and yet domestic cars are sold with the ability to go 120mph plus. Sure, you can jam your foot on the pedal and go as fast as you want.. but you're breaking the law. I wonder how many unnecessary injuries and deaths could be prevented if cars simply could not exceed the maximum speed limit, unless modified third-party? [...]

... and puts his very big foot into his mouth. The guy is quite ignorant.

For one thing, Montana allows any speed (during daytime), providing it is "reasonable and prudent", and Arizona has 75mph speed limit.

Second, special tracks exist, where people pay money for several laps of driving fast (such as Thunderhill in Illinois and Laguna Seca in California).

Third, cars are speed limited. For instance, Dodge Neon is limited to 118mph, which is the rating of its stock Goodyear Eagle tires, coincidently.

Bite me, stinking Nazi.

About five years ago, a guy was driving a back street at 25mph (according to police investigation), when a 3 year old boy sat on a skateboard and rolled from inclined driveway across the street in Newark, CA. Plop, plop. So, let's install PCM limiting cars to 10mph. That sure will cut on unnecessary death, right?

Goverment can do a lot of paternalistic shit to reduce "unnecessary" injury and death. In the process, the life becomes a living hell.

My assesement of Rob Levin's work differs from Martin Pool's. In an entry, mbp said that lilo did not show the code. I think lilo did, actually. He produced the OPN network, which is his "code", or something which you can examine and decide if it's worth anything.

I figure some people like it, so if they want to donate money to lilo for it, that's fine. If lilo wants to ask money, that's fine too. Personally, I see no value in OPN whatever, and I have no trouble tuning off discussions about it. I do not see how this can irritate anyone. We saw worse diaries at advogato than lilo's.

Now, the idea to swap handles for no reason is irritating. If someone are irritated by that, they should refrain from certifying rlevin. The raph's code takes care of the rest, like magic.