There's nothing like burning half a day on something stupid.

Last night I tried to test some changes to a program I hadn't re-built in a while, and when I ran it I got the following unhelpful message:

no such file or directory: /tmp/Verify_Cache

I scratched my head for a bit, then worked my way around to the idea that it could have to do with shared libraries, so I ran ldd on it and got:

/usr/bin/ldd: /tmp/Verify_Cache: No such file or directory

Some time later, a co-worker suggested using the LD_DEBUG environment variable. I tried that, and got the same error message as when I tried to run it the first time.

Much head scratching later, I started to look at the contents of the binaries with various other inspection tools. That's when I noticed that while "readelf --program-headers" on working binaries showed:

[Requesting program interpreter: /lib/ld-linux.so.2]

On my broken one it showed:

[Requesting program interpreter: /usr/lib/libc.so.1]

Which doesn't exist on any of my reasonably modern Linux systems. I made that a symlink to /lib/ld-linux.so.2 and my broken binary suddenly worked.

It turns out that the build instructions had gotten a little stale, and this binary had "-static" on its link line but also had shared objects on its link line. Garbage in garbage out, I guess. Still, it seems like there should be a big "attention jackass: you told me to do something stupid so I did" warning in this case.

P.S. I'm wearing my "It must be user error" t-shirt today. I obviously picked the right one, I just didn't know it was referring to me.