Older blog entries for etbe (starting at number 992)

Cheap SATA Disks in a Dell PowerEdge T410

A non-profit organisation I support has just bought a Dell PowerEdge T410 server to be mainly used as a file server. We need a reasonable amount of space and really good reliability features because the system may have periods without being actively monitored, it also has to be relatively cheap.

Dell servers are quite cheap, but disks are not cheap at all when Dell sells them. I get the impression that disks and RAM are major profit centers for Dell and that the profit margins on the basic servers are quite small. So naturally we decided to buy some SATA disks from a local store, one advantage of this is that Dell sells nothing bigger than 2TB while 3TB disks are available cheaply everywhere else.

So we bought 4 cheap 3TB (2.7TiB) SATA disks, connected them to the server, and found that only 2TiB was accessible. The Dell Tech Center says that some of the RAID controllers don’t support anything larger than 2TiB [1]. Obviously we have one of the older models. There’s lots of SATA sockets on the motherboard that could be used, however there is one problem.

View of the open side of a PowerEdge T410

The above picture is the side view of the T410, it was taken with a Samsung Galaxy S so the quality is a little poor (click for the original picture). The server is quite neat, not many cables for a system with 6 disks, not the 12 separate cables you would get in a typical white-box system.

Disks in a PowerEdge T410

The above picture shows the disk enclosure. You can see that each disk has a single connector for power and data, also the disks aren’t separate, multiple disks have the same power wires and the data cables are paired.

SAS controller in a PowerEdge T410

Above you can see the SAS controller, it has two large connectors that can each handle the data cables for 4 disks, nothing like the standard cables.

It’s easy to buy SATA data cables and connect them, but there are no spare power cables in the box. The connector that supplies power to all the disks appears to be something proprietary to Dell which goes straight to the double-connectors on each disk that supply power and data. This setup makes cabling very neat but also provides no good option for cabling regular disks. I’m sure I could make my own cables and if I hunted around the net enough I could probably buy some matching power cables, but it would be a hassle and the result wouldn’t be neat.

So the question was then whether to go to more effort, expense, and possibly risk the warranty to get the full 3TB or to just use the SAS controller and get 2TiB (2.2TB) per disk. One factor we considered is the fact that the higher sector numbers typically give much slower access times due to being on shorter tracks (see my ZCAV results page for test results from past disks [2]). We decided that 2.2TB (2TiB) out of 3TB (2.7TiB) was adequate capacity and that losing some of the slow parts of the disk wasn’t a big deal.

I’ve now setup a RAID-Z2 array on the disks, and ZFS reports 3.78TiB available capacity, which isn’t a lot considering that we have 4*3TB disks in the array. But the old server had only 200G of storage, so it’s a good improvement in capacity, performance, and RAID-Z2 should beat the hell out of RAID-6 for reliability.

Related posts:

  1. Dell PowerEdge T105 Today I received a Dell PowerEDGE T105 for use by...
  2. ZFS vs BTRFS on Cheap Dell Servers I previously wrote about my first experiences with BTRFS [1]....
  3. How I Partition Disks Having had a number of hard drives fail over the...

Syndicated 2012-07-31 23:55:56 from etbe - Russell Cokeretbe - Russell Coker

ZFS on Debian/Wheezy

As storage capacities increase the probability of data corruption increases as does the amount of time required for a fsck on a traditional filesystem. Also the capacity of disks is increasing a lot faster than the contiguous IO speed which means that the RAID rebuild time is increasing, for example my first hard disk was 70M and had a transfer rate of 500K/s which meant that the entire contents could be read in a mere 140 seconds! The last time I did a test on a more recent disk a 1TB SATA disk gave contiguous transfer rates ranging from 112MB/s to 52MB/s which meant that reading the entire contents took 3 hours and 10 minutes, and that problem is worse with newer bigger disks. The long rebuild times make greater redundancy more desirable.

BTRFS vs ZFS

Both BTRFS and ZFS checksum all data to cover the case where a disk returns corrupt data, they don’t need a fsck program, and the combination of checksums and built-in RAID means that they should have less risk of data loss due to a second failure during rebuild. ZFS supports RAID-Z which is essentially a RAID-5 with checksums on all blocks to handle the case of corrupt data as well as RAID-Z2 which is a similar equivalent to RAID-6. RAID-Z is quite important if you don’t want to have half your disk space taken up by redundancy or if you want to have your data survive the loss or more than one disk, so until BTRFS has an equivalent feature ZFS offers significant benefits. Also BTRFS is still rather new which is a concern for software that is critical to data integrity.

I am about to install a system to be a file server and Xen server which probably isn’t going to be upgraded a lot over the next few years. It will have 4 disks so ZFS with RAID-Z offers a significant benefit over BTRFS for capacity and RAID-Z2 offers a significant benefit for redundancy. As it won’t be upgraded a lot I’ll start with Debian/Wheezy even though it isn’t released yet because the system will be in use without much change well after Squeeze security updates end.

ZFS on Wheezy

Getting ZFS to basically work isn’t particularly hard, the ZFSonLinux.org site has the code and reasonable instructions for doing it [1]. The zfsonlinux code doesn’t compile out of the box on Wheezy although it works well on Squeeze. I found it easier to get a the latest Ubuntu working with ZFS and then I rebuilt the Ubuntu packages for Debian/Wheezy and they worked. This wasn’t particularly difficult but it’s a pity that the zfsonlinux site didn’t support recent kernels.

Root on ZFS

The complication with root on ZFS is that the ZFS FAQ recommends using whole disks for best performance so you can avoid alignment problems on 4K sector disks (which is an issue for any disk large enough that you want to use it with ZFS) [2]. This means you have to either use /boot on ZFS (which seems a little too experimental for me) or have a separate boot device.

Currently I have one server running with 4*3TB disks in a RAID-Z array and a single smaller disk for the root filesystem. Having a fifth disk attached by duct-tape to a system that is only designed for four disks isn’t ideal, but when you have an OS image that is backed up (and not so important) and a data store that’s business critical (but not needed every day) then a failure on the root device can be fixed the next day without serious problems. But I want to fix this and avoid creating more systems like it.

There is some good documentation on using Ubuntu with root on ZFS [3]. I considered using Ubuntu LTS for the server in question, but as I prefer Debian and I can recompile Ubuntu packages for Debian it seems that Debian is the best choice for me. I compiled those packages for Wheezy, did the install and DKMS build, and got ZFS basically working without much effort.

The problem then became getting ZFS to work for the root filesystem. The Ubuntu packages didn’t work with the Debian initramfs for some reason and modules failed to load. This wasn’t necessarily a show-stopper as I can modify such things myself, but it’s another painful thing to manage and another way that the system can potentially break on upgrade.

The next issue is the unusual way that ZFS mounts filesystems. Instead of having block devices to mount and entries in /etc/fstab the ZFS system does things for you. So if you want a ZFS volume to be mounted as root you configure the mountpoint via the “zfs set mountpoint” command. This of course means that it doesn’t get mounted if you boot with a different root filesystem and adds some needless pain to the process. When I encountered this I decided that root on ZFS isn’t a good option. So for this new server I’ll install it with an Ext4 filesystem on a RAID-1 device for root and /boot and use ZFS for everything else.

Correct Alignment

After setting up the system with a 4 disk RAID-1 (or mirror for the pedants who insist that true RAID-1 has only two disks) for root and boot I then created partitions for ZFS. According to fdisk output the partitions /dev/sda2, /dev/sdb2 etc had their first sector address as a multiple of 2048 which I presume addresses the alignment requirement for a disk that has 4K sectors.

Installing ZFS

deb http://www.coker.com.au wheezy zfs

I created the above APT repository (only AMD64) for ZFS packages based on Darik Horn’s Ubuntu packages (thanks for the good work Darik). Installing zfs-dkms, spl-dkms, and zfsutils gave a working ZFS system. I could probably have used Darik’s binary packages but I think it’s best to rebuild Ubuntu packages to use on Debian.

The server in question hasn’t gone live in production yet (it turns out that we don’t have agreement on what the server will do). But so far it seems to be working OK.

Related posts:

  1. Discovering OS Bugs and Using Snapshots I’m running Debian/Unstable on an EeePC 701, I’ve got an...
  2. Starting with BTRFS Based on my investigation of RAID reliability [1] I have...
  3. ZFS vs BTRFS on Cheap Dell Servers I previously wrote about my first experiences with BTRFS [1]....

Syndicated 2012-07-31 04:03:59 from etbe - Russell Cokeretbe - Russell Coker

The Retina Display

Last night I played with an Apple Macbook Pro with the new Retina Display (Wikipedia link). Wikipedia cites some controversy about whether the display actually has higher resolution than the human eye can perceive. When wearing glasses my vision is considerably better than average (I have average vision without glasses) and while kneeling in front of the Macbook I couldn’t easily distinguish pixels. So Apple’s marketing claims seem technically correct to me.

When I tested the Macbook Pro I found that the quality of the text display was very high, even now the 1680*1050 display on my Thinkpad T61 looks completely crap when compared to the 2880*1800 display on the Macbook. The Macbook was really great for text and for a JPEG that was installed on the system for testing. But unfortunately pictures on web sites didn’t look particularly good. Pictures on my blog looked quite poor and pictures returned by a Google search for “art” didn’t look that great either. I wonder if Safari (the Apple web browser) isn’t properly optimised for the display or if there is something that we should do when preparing pictures for web sites to make them look better on Safari.

The retina display has a 71% greater DPI which means 2.93* the total number of pixels of my Thinkpad. The overall quality of the experience for me (apart from web pictures) seems more like a factor of 2.93 when compared with my Thinkpad than a factor of 1.71. This has to be one of the most desirable products I’ve seen from a company that’s opposed to freedom for it’s users. I’m not about to buy one though, $2,300 is a lot of money for a system that can’t be upgraded, repaired, or recycled, and doesn’t even have an Ethernet port. I’m sure that if I bought one I would discover that it some of the hardware features don’t work properly with Linux.

The new Apple design trend of making it impossible to repair anything works reasonably well for phones and tablets which are cheap enough that they are hardly worth repairing when they have been used for a while. Lots of people can afford to spend about $600 on something that may be discarded after a year or two, but very few people can afford to spend more than $2,000 on such a disposable product.

Why is Apple the only company producing systems with such displays? If someone produced regular PCs that have the expected features (including an Ethernet port) with such a display at a lower price then I’m sure that there would be a great demand.

Related posts:

  1. Modern Laptops Suck One of the reasons why I’m moving from a laptop...
  2. A First Digital Camera I’ve just been asked for advice on buying a digital...
  3. RPC and SE Linux One ongoing problem with TCP networking is the combination of...

Syndicated 2012-07-31 03:27:31 from etbe - Russell Cokeretbe - Russell Coker

Links July 2012

The New York Times has an interesting article about “hacker hostels” [1]. I had an idea for similar things after watching a Japanese movie about Tokiwa-sō – a shared apartment for Manga artists which among others inspired the creator of Astro Boy [2].

The TED blog has an interesting interview with William Noel about open access to art and historical data [3]. Most of his talk concerns an Archimedes codex which has been recovered and published on the Internet. He advocates publishing all manner of art and historical data under a Creative Commons license.

The education system is often criticised for trying too hard to make children feel successful and not teaching them skills needed to be successful, it seems that the US military fails the same way in it’s war games [4].

Webroot.com published an interesting article last year about the first BIOS rootkit in the wild [5]. I really wish that they would design motherboards with a switch to enable BIOS writing which would default to “off”. I recently did a poll at a LUG meeting and found that only half the audience had updated the BIOS on most systems they owned, if the most technical people generally don’t need a dangerous feature then it should probably be disabled by default.

Matthew Wright wrote an interesting article about the costs of upgrading the electricity grid in Australia vs the costs of upgrading air-conditioners [6]. It seems that it would be a lot cheaper for the government to buy everyone a new air-conditioner than to upgrade the grid.

Owl City has a post of 10 Myths About Introverts [7]. That could probably be titled 10 Myths About Aspies and still be correct.

Susan Cain gave the most popular talk of TED 2012 about “The Power of Introverts”, here is an interesting interview about the talk and Introversion [8].

Related posts:

  1. Links April 2012 Karen Tse gave an interesting TED talk about how to...
  2. Links July 2011 The Reid Report has an article about the marriage pledge...
  3. Links March 2012 Washington’s Blog has an informative summary of recent articles about...

Syndicated 2012-07-30 12:54:53 from etbe - Russell Cokeretbe - Russell Coker

Some Proprietary Platform Issues

Android vs iPad

I’m currently in discussions with a client about a potential future project which involves a tablet computer talking to some electronic equipment. The options are an Android tablet and an iPad. One advantage of Android is that it runs on devices of all shapes and sizes, so we can choose a device that fits the need rather than designing everything around the iPad.

But the real problem with iPad is Apple. To run an app on an iPad you need to submit it to Apple, hopefully get it accepted into the App Market, then install it. This process causes some delay, a minor fee, and has the potential to derail the project if Apple doesn’t accept the app on the first try. With Android there is no need to even deal with Google, the app can be installed directly without the Google Play store.

I may end up working with an iPad (which admittedly is really nice hardware), but it seems most likely that the project in question will run on Android only.

Windows vs Linux and Apple OS/X

One of my clients recently paid a web development company to redevelop his web site. I turned out that the web developers in question only knew how to develop for Windows and my client didn’t discover this until too late. Now a site that’s currently using a small fraction of the resources on a $80 per month Linode instance will run on a Windows virtual server costing $300 per month (which includes SQL server license).

The Windows virtual server will probably be managed (because my client uses only Apple and Linux systems and doesn’t employ anyone with Windows skills) which adds an extra $100 per month. If the server isn’t managed then they will have to hire someone to apply patches and that won’t necessarily be cheaper.

So using Windows is going to cost my client an extra $400 per month when compared to the possibility of running a Linux system on the existing virtual server. Even if my client had someone with Windows skills to run the server it would still be an extra $300 per month. If the NBN was available then my client could run a Windows server in their office, but it’s not yet available in their area.

Even for a company that employs people with more Windows skills than Linux skills there are still economic factors in favor of Linux due to smaller hardware requirements and the lack of license fees for all the core software (OS, database server, web server, etc).

Summary

These anecdotes aren’t unusual, it’s the sort of thing that happens all the time. Sometimes the result is good (EG avoiding the iPad), sometimes it isn’t (being stuck with a proprietary web service).

I think I’ll have to suggest to my clients that every contract have a “no proprietary software” clause. Contracts can be amended if there is a reason, but it seems best to make a preemptive strike against companies that sneak proprietary software in and cause significant unexpected expense and difficulty.

Related posts:

  1. The Lenovo U1 Hybrid – an example of how Proprietary OSs Suck Lenovo have announced their innovative new U1 “Hybrid” laptop [1]....
  2. My Prediction for the iPhone I have previously written about how I refused an offer...
  3. Liberty and Mobile Phones I own two mobile phones at the moment, I use...

Syndicated 2012-07-13 11:35:11 from etbe - Russell Cokeretbe - Russell Coker

Breaking SATA Connectors

I’ve just broken my second SATA connector. This isn’t a lot considering the number of hard drives I’ve worked with, but it’s still really annoying as I generally don’t break things.

The problem is that unplugging a SATA cable requires pushing a little clip, this isn’t overly difficult but it unfortunately doesn’t fit well with habits formed from previous hardware. The power cables used for hard drives based on the ST-506 interface which was copied for the IDE interface was large and had a fairly tight fit. Removing such a cable requires a significant amount of force – which is about the same as the amount of force required to break a SATA connector.

When I first started using PCs a reasonably configured AT system cost over $5,000 (maybe something like $10,000 in today’s money). With that sort of price hardly anyone had a set of test PCs. When hardware prices dropped such that hard drives of reasonable size became reasonably affordable on the second-hand market I bought more disks and used some for extra storage and some for testing software. As there was nothing like VMWare for testing OS images the way to test a new OS was to plug in a different hard drive and boot it. So I got a lot of practice at removing IDE power cables with as much force as was necessary.

Now I own a pile of test PCs, SATA disks less than 100G are free, I use Xen for a lot of my testing, and generally I have much less need to swap hard drives around. In most situations in which I would swap hard drives in the 90′s I will now swap PCs and I have piles of PCs ready for this purpose. So I haven’t had enough practice with SATA disks to develop habits for safely removing them.

So far this lack of habit development has resulted in damaging two disks due to changing drives while not concentrating enough. Fortunately duct-tape works well for holding a SATA connector in place when the plastic that attaches to the clip is broken.

Related posts:

  1. Strange SATA Disk Performance Below is a GNUPlot graph of ZCAV output from a...
  2. Vibration and Strange SATA Performance Almost two years ago I blogged about a strange performance...
  3. Dell PowerEdge T105 Today I received a Dell PowerEDGE T105 for use by...

Syndicated 2012-07-09 10:03:18 from etbe - Russell Cokeretbe - Russell Coker

Postfwd and Local Only Email

Over a year ago when I was considering my first Android phone purchase I setup a test account on my mail server so that I could test email clients on phones and tablets. I used a short password because I didn’t want to type a lot on small screens and because typing a password into a random system owned by someone else isn’t particularly secure anyway. Then I forgot about the account until I noticed that my mail server was sending out spam.

Next time I setup such a test account I’ll put rules similar to the following in my Postfwd [1] configuration to stop Postfix from sending such messages. That will prevent the test account from receiving mail from outside or sending mail out of the server. The former is optional (getting a few thousand spam messages in an unused test account is no big deal) but the latter is needed to prevent getting my server blacklisted.

id=R_test_recipient ; recipient==test@coker.com.au ; sender!~.*@coker.com.au ; action=REJECT
id=R_test_sender ; sender==test@coker.com.au ; recipient!~.*@coker.com.au ; action=REJECT

Related posts:

  1. email disclaimers Andre Pang blogs about the annoyance of email disclaimers. For...
  2. Email Passwords I was doing some routine sysadmin work for a client...
  3. Some Postfix Scripts for dealing with Outbound Spamming I’ve just written some small scripts to help me manage...

Syndicated 2012-07-09 09:40:20 from etbe - Russell Cokeretbe - Russell Coker

Links June 2012

This Youtube video is an amusing satire of EULAs and copyright law as applied to uploading consciousness [1].

Washington’s Blog has an insightful article about the way that the lack of trust in the US is killing their economy [2]. It seems that as long as the 1% are allowed to get away with breaking the law the US economy won’t recover. It’s something we should all keep in mind at election time, let’s not be like the US.

AirBnB is an interesting service to allow people to rent a room or an apartment, a quick scan indicates that it’s a lot cheaper than hotels and offers many good locations [3]. It’s probably of most interest to the more social people though which is a down side for me.

Queensland’s highest court has ruled that “vilification of homosexuals is also vilification of bisexuals” because “an essential aspect of bisexuality is a sexual feeling of a person of the same sex, that is, homosexuality” [4]. Anyone who didn’t find that totally obvious could simply consult any dictionary or encyclopedia to find out. But the Australian legal system needed a 46 page ruling. We really need some sanity in the courts.

Father Gregory Boyle founded an organisation named Homeboy Industries with the purpose of providing jobs for people with criminal records [5]. It’s amazing the way he is helping people turn their lives around and it’s apparently a lot cheaper than sending them to jail.

Related posts:

  1. Links February 2012 Sociological Images has an interesting article about the attempts to...
  2. Links March 2012 Washington’s Blog has an informative summary of recent articles about...
  3. Links April 2012 Karen Tse gave an interesting TED talk about how to...

Syndicated 2012-06-30 13:54:32 from etbe - Russell Cokeretbe - Russell Coker

Targeted Advertising

Don Marti has written another blog post about targeted advertising [1]. His main point is that when a company uses the most targeted adverts (such as Google advertising) everyone knows that they are paying a small number of cents per click and nothing for the people who don’t click. This compares to TV adverts which cost a lot of money and for which most viewers either leave the room or use fast-forward. Therefore using Google adverts doesn’t send a signal about the amount of money invested in the products. Don also cited an example of a company sponsoring an OK Go film clip, that was a great idea, it shows that the company can do expensive things which are also a bit creative and fans will thank them (watch all the OK Go videos on Youtube, they are great).

The next question is how else companies can advertise? One thing I’d really like to see is sponsorship of authors. Pick an author and pay them a salary with paid editorial services for releasing a book a year for free in HTML and ebook formats. Having a fixed salary is a significant benefit when it comes time to apply for a mortgage or plan a holiday and being able to freely distribute books would be a significant benefit for an author who hasn’t got a large fan base.

In the computer industry it seems that there’s a lot of potential for sponsoring people who produce free things. That ranges from free software and designs for free hardware to blog posts and documentation. Five years ago Sun had a blogging contest and my friend Dave Hall won a server that was worth $21K [2]. It would be nice if some other companies started doing similar things and if Sun did a repeat so some other people I like could get some free kit.

Related posts:

  1. What is Appropriate Advertising? Colin Charles writes about a woman who is selling advertising...
  2. Advertising Free Software Projects Today I just noticed the following advert on one of...
  3. Friends and Adverts For some time I have been running Google Adsense adverts...

Syndicated 2012-06-24 14:50:36 from etbe - Russell Cokeretbe - Russell Coker

New SE Linux Policy for Wheezy

I’ve just uploaded a new SE Linux policy for Debian/Wheezy. It now works correctly with systemd and Chromium, two significant features that I wanted for Wheezy. Now it turns out that we have until the end of the month for Wheezy updates, so I may get another version of the policy uploaded before then. If so it will only be for relatively minor changes, I think that most SE Linux users would be reasonably happy with policy the way it is. Anything that doesn’t work now can probably be solved by local configuration changes.

execmem

The current version of KDE in Debian is 4.8.4, it seems that large parts of the KDE environment depend on execmem access, this includes kwin and plasma-desktop. Basically there is no possibility of having a KDE desktop environment without those programs and therefore KDE depends on execmem access.

Debugging this is difficult as the important programs SEGV when denied execmem access and the KDE crash handler really gets in the way of debugging it – running /usr/bin/plasma-desktop results in the process forking a child and detaching from the gdb session.

The most clear example of an execmem issue in KDE is from the program /usr/lib/kde4/libexec/kwin_opengl_test which gives the following error:
LLVM ERROR: Allocation failed when allocating new memory in the JIT
Can’t allocate RWX Memory: Permission denied

To make this work you run the command “setsebool -P allow_execmem 1” which gives many domains the ability to create writable-executable memory regions.

I raised this issue for discussion on the SE Linux mailing list and Hinnerk van Bruinehsen wrote an informative message in response summarising the situation [1]. It seems that it’s possible to compile some of the programs in question to not use the JIT and therefore not require such access and there is a build option in Gentoo to allow it. But it’s impractically difficult for me to fork KDE in Debian so the only option is to recommend that people enable the allow_execmem boolean for Debian desktop systems running SE Linux.

Related posts:

  1. /run and SE Linux Policy Currently Debian/Unstable is going through a transition to using /run...
  2. An Update on DKIM Signing and SE Linux Policy In my previous post about DKIM [1] I forgot to...
  3. New SE Linux Policy for Squeeze I have just uploaded refpolicy version 0.2.20100524-1 to Unstable. This...

Syndicated 2012-06-21 14:12:14 from etbe - Russell Cokeretbe - Russell Coker

983 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!