Yesterday I took the time to look into an annoyance, that threaded me for some weeks now: When I tried to rsync a local Mercurial copy of an SVN repository (managed by hgsubversion), it failed suddenly with 'No space left on device'.
No, what? I'm monitoring my drives quite loosely, but a look at actual free disc space confirmed: less than 70 % used. So what?
Turned out the sheer amount of small files used especially for Mercurial meta-data had eaten up all available inodes, what had gone unnoticed up to that point. The repository is locally stored inside a 10G partition, but it's copied to another place only 2,5G big. While both are ext3 file systems, not only the fs size significantly affected the total number of inodes, but the double inode size of 256 Byte for the smaller partition (newer installation) compared to 128 on the older, bigger partition did a great deal on shortening available inodes as well.
The diagnosis was followed by an - initially rather desperate - search to find and understand possible solutions. I wouldn't buy some early results, but it become obvious be few more hours: There's no such thing like file system tuning for inodes. Not that technically it can't be done, more likely there has been not much demand to write code for bringing such a feature i.e. into tune2fs. You're forced to rebuild your fs with more sensible settings for the fs management internals to better fit it to your usage.
Good thing, I had plenty of room in my LVM managed remote system, so build another 2,5G part and experimenting with different bytes/inode ratios ('-i' switch for mkfs.ext3) was a matter of minutes. After finding a sensible value I copied the whole fs over with 'cp -a', and after working with the copy on the new partition a bit I copied the whole thing back with dd_rescue (favored over dd for a nicer progress indicator), every time with the copied source as fallback option.
All in all a great and quite painless lesson on fs internals, that have been rather subtle to me so far, despite doing GNU/Linux system setups for 10+ years now.