Older blog entries for etbe (starting at number 994)

2 Aug 2012 »

Sam Harris on Lying

The Neuroscientist and atheism advocate Sam Harris has written a short blog post about a journalist named Jonah Lehrer who destroyed his career through false quotes and lies about them [1]. The main point of the article seems to be to promote his new eBook about Lying. The book is available for free until the end of the week (not sure if that is Friday, Saturday, or Sunday and in what time zone – get it quick if you want it).

The book is very short, 58 pages with a single column of large font text. If written a densely as a typical research paper it would probably be about 12 pages. But it has some good points to make. He makes a good moral case against most forms of lying, even answering questions such as “do I look fat“.

It seems that anyone who was to follow his advice would be unusually honest even by Aspie standards.

[1] http://www.samharris.org/blog/item/the-fall-of-jonah-lehrer

Lies and Online Dating Separating Fact From Fiction: An Examination of Deceptive Self-Presentation in...
Links January 2012 Cops in Tennessee routinely steal cash from citizens [1]. They...
Desks Lindsay Holmwood has written about the benefits of a standing...

Syndicated 2012-08-02 11:52:23 from etbe - Russell Cokeretbe - Russell Coker

1 Aug 2012 »

Hetzner now Offers SSD

Hetzner is offering new servers with SSD, good news for people who want to run ZFS (for ZIL and/or L2ARC). See the EX server configuration list for more information [1]. Unfortunately they don’t specify what brand of SSD, this is a concern for me as some of the reports about SSD haven’t been that positive, getting whichever SSD is cheapest isn’t appealing. A cheap SSD might be OK for L2ARC (read cache), but for ZIL (write cache) reliability is fairly important. If anyone has access to a Hetzner server with SSD then please paste the relevant output of lsscsi into a comment.

The next issue is that they only officially offer it on the new “EX 8S” server. SSD will be of most interest to people who also want lots of RAM (the zfsonlinux.org code has given me kernel panics when running with a mere 4G of RAM – even when I did the recommended tuning to reduce ARC size). Also people who want more capable storage options will tend to want more RAM if only for disk caching.

But I’m sure that there are plenty of people who would be happy to have SSD on a smaller and cheaper server. The biggest SSD offering of 240G is bigger than a lot of servers. I run a Hetzner server that has only 183G of disk space in use (and another 200G of backups). If the backups were on another site then the server in question could have just a RAID-1 of SSD for all it’s storage. In this case it wouldn’t be worth doing as the server doesn’t have much disk IO load, but it would be nice to have the option – the exact same server plus some more IO load would make SSD the ideal choice.

The biggest problem is that the EX 8S server is really expensive. Hard drives which are included in the base price for cheaper options are now expensive additions. A server with 2*3TB disks and 2*240G SSD is E167 per month! That’s more expensive than three smaller servers that have 2*3TB disks! The good news for someone who wants SSD is that the Hetzner server “auction” has some better deals [2]. As is always the case with auction sites the exact offers will change by the moment, but currently they offer a server with 2*120G SSD and 24G of RAM for E88 per month and a server with 2*120G SSD, 2*1.5T HDD, and 24G of RAM for E118. E88 is a great deal if your storage fits in 240G and E118 could be pretty good if you only have 1.5T of data that needs ZFS features.

The main SSD offering is still a good option for some cases. A project that I did a couple of years ago would probably have worked really well on a E167/month server with 2*3TB and 2*240G SSD. It was designed around multiple database servers sharding the load which was largely writes, so SSD would have allowed a significant reduction in the number of servers.

They also don’t offer SSD on their “storage servers” which is a significant omission. I presume that they will fix that soon enough. 13 disks and 2 SSD will often be more useful than 15 disks. That’s assuming the SSD doesn’t suck of course.

The reason this is newsworthy is that most hosted server offerings have very poor disk IO and no good options for expanding it. For servers that you host yourself it’s not too difficult to buy extra trays of disks or even a single rack-mount server that has any number of internal disks in the range 2 to 24 and any choice as to how you populate them. But with rented servers it’s typically 2 disks with no options to add SSD or other performance enhancements and no possibility of connecting a SAN. As an aside it would still be nice if someone ran a data center that supported NetApp devices and gave the option of connecting an arbitrary number of servers to a NetApp Filer (or a redundant pair of Filers). If anyone knows of a hosting company that provides options for good disk IO which are better than just providing SSD or cheaper than E167 per month then please provide the URL in a comment.

Servers vs Phones Hetzner have recently updated their offerings to include servers with...
Hetzner Failover Konfiguration The Wiki documenting how to configure IP failover for Hetzner...
Dedicated vs Virtual Servers A common question about hosting is whether to use a...

Syndicated 2012-08-01 09:55:18 from etbe - Russell Cokeretbe - Russell Coker

1 Aug 2012 »

Cheap SATA Disks in a Dell PowerEdge T410

A non-profit organisation I support has just bought a Dell PowerEdge T410 server to be mainly used as a file server. We need a reasonable amount of space and really good reliability features because the system may have periods without being actively monitored, it also has to be relatively cheap.

Dell servers are quite cheap, but disks are not cheap at all when Dell sells them. I get the impression that disks and RAM are major profit centers for Dell and that the profit margins on the basic servers are quite small. So naturally we decided to buy some SATA disks from a local store, one advantage of this is that Dell sells nothing bigger than 2TB while 3TB disks are available cheaply everywhere else.

So we bought 4 cheap 3TB (2.7TiB) SATA disks, connected them to the server, and found that only 2TiB was accessible. The Dell Tech Center says that some of the RAID controllers don’t support anything larger than 2TiB [1]. Obviously we have one of the older models. There’s lots of SATA sockets on the motherboard that could be used, however there is one problem.

The above picture is the side view of the T410, it was taken with a Samsung Galaxy S so the quality is a little poor (click for the original picture). The server is quite neat, not many cables for a system with 6 disks, not the 12 separate cables you would get in a typical white-box system.

The above picture shows the disk enclosure. You can see that each disk has a single connector for power and data, also the disks aren’t separate, multiple disks have the same power wires and the data cables are paired.

Above you can see the SAS controller, it has two large connectors that can each handle the data cables for 4 disks, nothing like the standard cables.

It’s easy to buy SATA data cables and connect them, but there are no spare power cables in the box. The connector that supplies power to all the disks appears to be something proprietary to Dell which goes straight to the double-connectors on each disk that supply power and data. This setup makes cabling very neat but also provides no good option for cabling regular disks. I’m sure I could make my own cables and if I hunted around the net enough I could probably buy some matching power cables, but it would be a hassle and the result wouldn’t be neat.

So the question was then whether to go to more effort, expense, and possibly risk the warranty to get the full 3TB or to just use the SAS controller and get 2TiB (2.2TB) per disk. One factor we considered is the fact that the higher sector numbers typically give much slower access times due to being on shorter tracks (see my ZCAV results page for test results from past disks [2]). We decided that 2.2TB (2TiB) out of 3TB (2.7TiB) was adequate capacity and that losing some of the slow parts of the disk wasn’t a big deal.

I’ve now setup a RAID-Z2 array on the disks, and ZFS reports 3.78TiB available capacity, which isn’t a lot considering that we have 4*3TB disks in the array. But the old server had only 200G of storage, so it’s a good improvement in capacity, performance, and RAID-Z2 should beat the hell out of RAID-6 for reliability.

Dell PowerEdge T105 Today I received a Dell PowerEDGE T105 for use by...
ZFS vs BTRFS on Cheap Dell Servers I previously wrote about my first experiences with BTRFS [1]....
How I Partition Disks Having had a number of hard drives fail over the...

Syndicated 2012-07-31 23:55:56 from etbe - Russell Cokeretbe - Russell Coker

31 Jul 2012 »

ZFS on Debian/Wheezy

As storage capacities increase the probability of data corruption increases as does the amount of time required for a fsck on a traditional filesystem. Also the capacity of disks is increasing a lot faster than the contiguous IO speed which means that the RAID rebuild time is increasing, for example my first hard disk was 70M and had a transfer rate of 500K/s which meant that the entire contents could be read in a mere 140 seconds! The last time I did a test on a more recent disk a 1TB SATA disk gave contiguous transfer rates ranging from 112MB/s to 52MB/s which meant that reading the entire contents took 3 hours and 10 minutes, and that problem is worse with newer bigger disks. The long rebuild times make greater redundancy more desirable.

BTRFS vs ZFS

Both BTRFS and ZFS checksum all data to cover the case where a disk returns corrupt data, they don’t need a fsck program, and the combination of checksums and built-in RAID means that they should have less risk of data loss due to a second failure during rebuild. ZFS supports RAID-Z which is essentially a RAID-5 with checksums on all blocks to handle the case of corrupt data as well as RAID-Z2 which is a similar equivalent to RAID-6. RAID-Z is quite important if you don’t want to have half your disk space taken up by redundancy or if you want to have your data survive the loss or more than one disk, so until BTRFS has an equivalent feature ZFS offers significant benefits. Also BTRFS is still rather new which is a concern for software that is critical to data integrity.

I am about to install a system to be a file server and Xen server which probably isn’t going to be upgraded a lot over the next few years. It will have 4 disks so ZFS with RAID-Z offers a significant benefit over BTRFS for capacity and RAID-Z2 offers a significant benefit for redundancy. As it won’t be upgraded a lot I’ll start with Debian/Wheezy even though it isn’t released yet because the system will be in use without much change well after Squeeze security updates end.

ZFS on Wheezy

Getting ZFS to basically work isn’t particularly hard, the ZFSonLinux.org site has the code and reasonable instructions for doing it [1]. The zfsonlinux code doesn’t compile out of the box on Wheezy although it works well on Squeeze. I found it easier to get a the latest Ubuntu working with ZFS and then I rebuilt the Ubuntu packages for Debian/Wheezy and they worked. This wasn’t particularly difficult but it’s a pity that the zfsonlinux site didn’t support recent kernels.

Root on ZFS

The complication with root on ZFS is that the ZFS FAQ recommends using whole disks for best performance so you can avoid alignment problems on 4K sector disks (which is an issue for any disk large enough that you want to use it with ZFS) [2]. This means you have to either use /boot on ZFS (which seems a little too experimental for me) or have a separate boot device.

Currently I have one server running with 4*3TB disks in a RAID-Z array and a single smaller disk for the root filesystem. Having a fifth disk attached by duct-tape to a system that is only designed for four disks isn’t ideal, but when you have an OS image that is backed up (and not so important) and a data store that’s business critical (but not needed every day) then a failure on the root device can be fixed the next day without serious problems. But I want to fix this and avoid creating more systems like it.

There is some good documentation on using Ubuntu with root on ZFS [3]. I considered using Ubuntu LTS for the server in question, but as I prefer Debian and I can recompile Ubuntu packages for Debian it seems that Debian is the best choice for me. I compiled those packages for Wheezy, did the install and DKMS build, and got ZFS basically working without much effort.

The problem then became getting ZFS to work for the root filesystem. The Ubuntu packages didn’t work with the Debian initramfs for some reason and modules failed to load. This wasn’t necessarily a show-stopper as I can modify such things myself, but it’s another painful thing to manage and another way that the system can potentially break on upgrade.

The next issue is the unusual way that ZFS mounts filesystems. Instead of having block devices to mount and entries in /etc/fstab the ZFS system does things for you. So if you want a ZFS volume to be mounted as root you configure the mountpoint via the “zfs set mountpoint” command. This of course means that it doesn’t get mounted if you boot with a different root filesystem and adds some needless pain to the process. When I encountered this I decided that root on ZFS isn’t a good option. So for this new server I’ll install it with an Ext4 filesystem on a RAID-1 device for root and /boot and use ZFS for everything else.

Correct Alignment

After setting up the system with a 4 disk RAID-1 (or mirror for the pedants who insist that true RAID-1 has only two disks) for root and boot I then created partitions for ZFS. According to fdisk output the partitions /dev/sda2, /dev/sdb2 etc had their first sector address as a multiple of 2048 which I presume addresses the alignment requirement for a disk that has 4K sectors.

Installing ZFS

deb http://www.coker.com.au wheezy zfs

I created the above APT repository (only AMD64) for ZFS packages based on Darik Horn’s Ubuntu packages (thanks for the good work Darik). Installing zfs-dkms, spl-dkms, and zfsutils gave a working ZFS system. I could probably have used Darik’s binary packages but I think it’s best to rebuild Ubuntu packages to use on Debian.

The server in question hasn’t gone live in production yet (it turns out that we don’t have agreement on what the server will do). But so far it seems to be working OK.

Discovering OS Bugs and Using Snapshots I’m running Debian/Unstable on an EeePC 701, I’ve got an...
Starting with BTRFS Based on my investigation of RAID reliability [1] I have...
ZFS vs BTRFS on Cheap Dell Servers I previously wrote about my first experiences with BTRFS [1]....

Syndicated 2012-07-31 04:03:59 from etbe - Russell Cokeretbe - Russell Coker

31 Jul 2012 »

The Retina Display

Last night I played with an Apple Macbook Pro with the new Retina Display (Wikipedia link). Wikipedia cites some controversy about whether the display actually has higher resolution than the human eye can perceive. When wearing glasses my vision is considerably better than average (I have average vision without glasses) and while kneeling in front of the Macbook I couldn’t easily distinguish pixels. So Apple’s marketing claims seem technically correct to me.

When I tested the Macbook Pro I found that the quality of the text display was very high, even now the 1680*1050 display on my Thinkpad T61 looks completely crap when compared to the 2880*1800 display on the Macbook. The Macbook was really great for text and for a JPEG that was installed on the system for testing. But unfortunately pictures on web sites didn’t look particularly good. Pictures on my blog looked quite poor and pictures returned by a Google search for “art” didn’t look that great either. I wonder if Safari (the Apple web browser) isn’t properly optimised for the display or if there is something that we should do when preparing pictures for web sites to make them look better on Safari.

The retina display has a 71% greater DPI which means 2.93* the total number of pixels of my Thinkpad. The overall quality of the experience for me (apart from web pictures) seems more like a factor of 2.93 when compared with my Thinkpad than a factor of 1.71. This has to be one of the most desirable products I’ve seen from a company that’s opposed to freedom for it’s users. I’m not about to buy one though, $2,300 is a lot of money for a system that can’t be upgraded, repaired, or recycled, and doesn’t even have an Ethernet port. I’m sure that if I bought one I would discover that it some of the hardware features don’t work properly with Linux.

The new Apple design trend of making it impossible to repair anything works reasonably well for phones and tablets which are cheap enough that they are hardly worth repairing when they have been used for a while. Lots of people can afford to spend about $600 on something that may be discarded after a year or two, but very few people can afford to spend more than $2,000 on such a disposable product.

Why is Apple the only company producing systems with such displays? If someone produced regular PCs that have the expected features (including an Ethernet port) with such a display at a lower price then I’m sure that there would be a great demand.

Modern Laptops Suck One of the reasons why I’m moving from a laptop...
A First Digital Camera I’ve just been asked for advice on buying a digital...
RPC and SE Linux One ongoing problem with TCP networking is the combination of...

Syndicated 2012-07-31 03:27:31 from etbe - Russell Cokeretbe - Russell Coker

30 Jul 2012 »

Links July 2012

The New York Times has an interesting article about “hacker hostels” [1]. I had an idea for similar things after watching a Japanese movie about Tokiwa-sō – a shared apartment for Manga artists which among others inspired the creator of Astro Boy [2].

The TED blog has an interesting interview with William Noel about open access to art and historical data [3]. Most of his talk concerns an Archimedes codex which has been recovered and published on the Internet. He advocates publishing all manner of art and historical data under a Creative Commons license.

The education system is often criticised for trying too hard to make children feel successful and not teaching them skills needed to be successful, it seems that the US military fails the same way in it’s war games [4].

Webroot.com published an interesting article last year about the first BIOS rootkit in the wild [5]. I really wish that they would design motherboards with a switch to enable BIOS writing which would default to “off”. I recently did a poll at a LUG meeting and found that only half the audience had updated the BIOS on most systems they owned, if the most technical people generally don’t need a dangerous feature then it should probably be disabled by default.

Matthew Wright wrote an interesting article about the costs of upgrading the electricity grid in Australia vs the costs of upgrading air-conditioners [6]. It seems that it would be a lot cheaper for the government to buy everyone a new air-conditioner than to upgrade the grid.

Owl City has a post of 10 Myths About Introverts [7]. That could probably be titled 10 Myths About Aspies and still be correct.

Susan Cain gave the most popular talk of TED 2012 about “The Power of Introverts”, here is an interesting interview about the talk and Introversion [8].

Links April 2012 Karen Tse gave an interesting TED talk about how to...
Links July 2011 The Reid Report has an article about the marriage pledge...
Links March 2012 Washington’s Blog has an informative summary of recent articles about...

Syndicated 2012-07-30 12:54:53 from etbe - Russell Cokeretbe - Russell Coker

13 Jul 2012 »

Some Proprietary Platform Issues

Android vs iPad

I’m currently in discussions with a client about a potential future project which involves a tablet computer talking to some electronic equipment. The options are an Android tablet and an iPad. One advantage of Android is that it runs on devices of all shapes and sizes, so we can choose a device that fits the need rather than designing everything around the iPad.

But the real problem with iPad is Apple. To run an app on an iPad you need to submit it to Apple, hopefully get it accepted into the App Market, then install it. This process causes some delay, a minor fee, and has the potential to derail the project if Apple doesn’t accept the app on the first try. With Android there is no need to even deal with Google, the app can be installed directly without the Google Play store.

I may end up working with an iPad (which admittedly is really nice hardware), but it seems most likely that the project in question will run on Android only.

Windows vs Linux and Apple OS/X

One of my clients recently paid a web development company to redevelop his web site. I turned out that the web developers in question only knew how to develop for Windows and my client didn’t discover this until too late. Now a site that’s currently using a small fraction of the resources on a $80 per month Linode instance will run on a Windows virtual server costing $300 per month (which includes SQL server license).

The Windows virtual server will probably be managed (because my client uses only Apple and Linux systems and doesn’t employ anyone with Windows skills) which adds an extra $100 per month. If the server isn’t managed then they will have to hire someone to apply patches and that won’t necessarily be cheaper.

So using Windows is going to cost my client an extra $400 per month when compared to the possibility of running a Linux system on the existing virtual server. Even if my client had someone with Windows skills to run the server it would still be an extra $300 per month. If the NBN was available then my client could run a Windows server in their office, but it’s not yet available in their area.

Even for a company that employs people with more Windows skills than Linux skills there are still economic factors in favor of Linux due to smaller hardware requirements and the lack of license fees for all the core software (OS, database server, web server, etc).

Summary

These anecdotes aren’t unusual, it’s the sort of thing that happens all the time. Sometimes the result is good (EG avoiding the iPad), sometimes it isn’t (being stuck with a proprietary web service).

I think I’ll have to suggest to my clients that every contract have a “no proprietary software” clause. Contracts can be amended if there is a reason, but it seems best to make a preemptive strike against companies that sneak proprietary software in and cause significant unexpected expense and difficulty.

The Lenovo U1 Hybrid – an example of how Proprietary OSs Suck Lenovo have announced their innovative new U1 “Hybrid” laptop [1]....
My Prediction for the iPhone I have previously written about how I refused an offer...
Liberty and Mobile Phones I own two mobile phones at the moment, I use...

Syndicated 2012-07-13 11:35:11 from etbe - Russell Cokeretbe - Russell Coker

9 Jul 2012 »

Breaking SATA Connectors

I’ve just broken my second SATA connector. This isn’t a lot considering the number of hard drives I’ve worked with, but it’s still really annoying as I generally don’t break things.

The problem is that unplugging a SATA cable requires pushing a little clip, this isn’t overly difficult but it unfortunately doesn’t fit well with habits formed from previous hardware. The power cables used for hard drives based on the ST-506 interface which was copied for the IDE interface was large and had a fairly tight fit. Removing such a cable requires a significant amount of force – which is about the same as the amount of force required to break a SATA connector.

When I first started using PCs a reasonably configured AT system cost over $5,000 (maybe something like $10,000 in today’s money). With that sort of price hardly anyone had a set of test PCs. When hardware prices dropped such that hard drives of reasonable size became reasonably affordable on the second-hand market I bought more disks and used some for extra storage and some for testing software. As there was nothing like VMWare for testing OS images the way to test a new OS was to plug in a different hard drive and boot it. So I got a lot of practice at removing IDE power cables with as much force as was necessary.

Now I own a pile of test PCs, SATA disks less than 100G are free, I use Xen for a lot of my testing, and generally I have much less need to swap hard drives around. In most situations in which I would swap hard drives in the 90′s I will now swap PCs and I have piles of PCs ready for this purpose. So I haven’t had enough practice with SATA disks to develop habits for safely removing them.

So far this lack of habit development has resulted in damaging two disks due to changing drives while not concentrating enough. Fortunately duct-tape works well for holding a SATA connector in place when the plastic that attaches to the clip is broken.

Strange SATA Disk Performance Below is a GNUPlot graph of ZCAV output from a...
Vibration and Strange SATA Performance Almost two years ago I blogged about a strange performance...
Dell PowerEdge T105 Today I received a Dell PowerEDGE T105 for use by...

Syndicated 2012-07-09 10:03:18 from etbe - Russell Cokeretbe - Russell Coker

9 Jul 2012 »

Postfwd and Local Only Email

Over a year ago when I was considering my first Android phone purchase I setup a test account on my mail server so that I could test email clients on phones and tablets. I used a short password because I didn’t want to type a lot on small screens and because typing a password into a random system owned by someone else isn’t particularly secure anyway. Then I forgot about the account until I noticed that my mail server was sending out spam.

Next time I setup such a test account I’ll put rules similar to the following in my Postfwd [1] configuration to stop Postfix from sending such messages. That will prevent the test account from receiving mail from outside or sending mail out of the server. The former is optional (getting a few thousand spam messages in an unused test account is no big deal) but the latter is needed to prevent getting my server blacklisted.

id=R_test_recipient ; recipient==test@coker.com.au ; sender!~.*@coker.com.au ; action=REJECT
id=R_test_sender ; sender==test@coker.com.au ; recipient!~.*@coker.com.au ; action=REJECT

[1] http://postfwd.org/

email disclaimers Andre Pang blogs about the annoyance of email disclaimers. For...
Email Passwords I was doing some routine sysadmin work for a client...
Some Postfix Scripts for dealing with Outbound Spamming I’ve just written some small scripts to help me manage...

Syndicated 2012-07-09 09:40:20 from etbe - Russell Cokeretbe - Russell Coker

30 Jun 2012 »

Links June 2012

This Youtube video is an amusing satire of EULAs and copyright law as applied to uploading consciousness [1].

Washington’s Blog has an insightful article about the way that the lack of trust in the US is killing their economy [2]. It seems that as long as the 1% are allowed to get away with breaking the law the US economy won’t recover. It’s something we should all keep in mind at election time, let’s not be like the US.

AirBnB is an interesting service to allow people to rent a room or an apartment, a quick scan indicates that it’s a lot cheaper than hotels and offers many good locations [3]. It’s probably of most interest to the more social people though which is a down side for me.

Queensland’s highest court has ruled that “vilification of homosexuals is also vilification of bisexuals” because “an essential aspect of bisexuality is a sexual feeling of a person of the same sex, that is, homosexuality” [4]. Anyone who didn’t find that totally obvious could simply consult any dictionary or encyclopedia to find out. But the Australian legal system needed a 46 page ruling. We really need some sanity in the courts.

Father Gregory Boyle founded an organisation named Homeboy Industries with the purpose of providing jobs for people with criminal records [5]. It’s amazing the way he is helping people turn their lives around and it’s apparently a lot cheaper than sending them to jail.

Links February 2012 Sociological Images has an interesting article about the attempts to...
Links March 2012 Washington’s Blog has an informative summary of recent articles about...
Links April 2012 Karen Tse gave an interesting TED talk about how to...

Syndicated 2012-06-30 13:54:32 from etbe - Russell Cokeretbe - Russell Coker

985 older entries...