Older blog entries for etbe (starting at number 972)

5 May 2012 »

Liberty and Mobile Phones

I own two mobile phones at the moment, I use a Samsung Galaxy S running Cyanogenmod [1] (Android 2.3.7) for most things, and I have a Sony Ericsson Xperia X10 running Android 2.1 that I use for taking photos, some occasional Wifi web browsing, and using some applications.

Comparing Android Hardware

The hardware for the Xperia X10 is better than that of the Galaxy S in many ways. It has a slightly higher resolution (480*854 vs 480*800), a significantly better camera (8.1MP with a “flash” vs 5MP without), and a status LED which I find really handy (although some people don’t care about it).

The only benefit of the Galaxy S hardware is that it has 16G of internal storage (of which about 2G can be used for applications) and 512M of RAM while the Xperia X10 has 1G of internal storage and 384M of RAM. These are significant issues, I have had applications run out of RAM on the Xperia X10 and I have been forced to uninstall applications to make space.

Overall I consider the Xperia X10 to be a significantly better piece of hardware as I am willing to trade off some RAM and internal storage to get a better resolution screen and a better camera. The problem is that Sony Ericsson have locked down their phones as much as possible and they don’t even give users the option of making a useful backup – they inspired my post about 5 principles of backups [2].

The fact that the Galaxy S allows installing CyanogenMod which then gives me the liberty to do whatever I want with my phone is a massive feature. It outweighs the hardware benefits of the Sony Ericsson phones over Samsung phones prior to the Galaxy Nexus and Galaxy Note.

For an individual user the ability to control their own hardware is a feature. Such an ability wouldn’t be much use if there wasn’t a community of software developers, so if you buy an Android phone that isn’t supported by CyanogenMod or another free Android distribution then whether it is locked probably won’t matter to you. But for any popular Android phone that’s sold on the mass market it seems that if it’s not locked then it will get a binary distribution of Android in a reasonable amount of time.

Comparing with Apple

It seems that Apple is the benchmark for non-free computing at the moment. The iPhone is locked down and Apple takes steps to re-lock phones that can be rooted – as opposed to the Android vendors who ship phones and then don’t bother to update the firmware for any reason. The Apple app market is more expensive and difficult to enter and if an app isn’t in the market then you have to pay if you want to install it on a small number of development/test phones. This compares to Android where the Google market is cheaper and easier to enter and anyone can distribute an app outside the market and have people use it.

But for an individual this doesn’t necessarily cause any problems. I have friends and clients who use iPhones and are very happy with them. In terms of software development it’s a real benefit to have a large number of systems running the same software. As Apple seems to have higher margins and larger volume than any other phone vendor as well as shipping only one phone at any time (compared to every other phone vendor which seems to ship at least 3 different products for different use cases) they are in a much better economic position to get the software development right. As far as I can tell the hardware and software of the iPhone is of very high quality. The iPad (which has a similar market position) is also a quality product. The fact that the Apple app market is more difficult to enter (both in terms of Apple liking the application and the cost of entry) also has it’s advantages, I get the impression that the general quality of iPhone apps is quite high as opposed to Android where there are a lot of low quality apps and many more fraudulent apps than there should be.

The lack of choice in Apple hardware (one phone and one tablet) is a disadvantage for the user. There is no option for a phone with a slide-out keyboard, a large screen (for the elderly and people with fat fingers), or any of the other features that some Android phones have. The lack of a range of sizes for the iPad is also a disadvantage. But it seems that Apple has produced hardware that is good enough for most users so there aren’t many complaints about a lack of choice.

It seems to me that the biggest disadvantage of the closed Apple ecosystem is for society in general. Anyone who wants to write a mobile app to do something which might be considered controversial would probably think twice about whether to develop for the iPhone/iPad as Apple could remove the app at a whim which would waste all the software development work that was invested in writing the app. Google seem to have much less interest in removing apps from their store and if they do remove an app then with some inconvenience it can be distributed on the web without involving them – so the work won’t be wasted.

How Much Freedom Should a Vendor Provide?

The Apple approach of locking everything down is clearly working for them at the moment. The Samsung approach of taking the Google prescribed code and allowing users to replace it is good for the users and works well. The Sony Ericsson approach of taking the Google code, adding some proprietary code, and then locking the phone down is bad for the users and I think it will be bad for Sony Ericsson. People are more likely to tell others about negative experiences and negative reviews are more likely to be noticed than positive reviews. So while many people are reasonably happy with Sony Ericsson products (until they find themselves unable to restore from a backup) it’s still not a good situation for Sony Ericsson marketing.

It seems that there are benefits to hardware vendors for being really open and for locking their users in properly. But being somewhat open isn’t a good choice, particularly for a vendor that ships poor quality proprietary apps such as the Sony Ericsson ones.

In terms of application distribution Google isn’t as nice as they appear. The Skyhook case revealed that Google will do whatever it takes to prevent apps that compete with Google apps from being installed by default [3]. Google is also trying to make money from DRM sales via Youtube which it denies to rooted phones [4]. Again it seems to me that the best options here are being more open than Google is and being as closed as Apple. Google might gain some useful benefits from applying DRM (even though everyone with technical knowledge knows that it doesn’t work) but the Skyhook shenanigans have got to be costing Google more than it’s worth.

How to make Android devices more Free

The F-droid market is an alternative to the Google App market which only has free software [5]. On it’s web site there are links to download the source for the applications, including the source and binaries for old versions. In the Google App market if an upgrade breaks your system then you just lose, with F-droid you can revert to the old version.

A self-hosted OwnCloud installation for a private or semi-private cloud [6] can be used as an alternative to the Google Music store as well as for hosting any other data that you want to store online.

The Open Street Map for Android (Osmand) project provides an alternative to the Google Map service [7]. Osmand allows you to download all the vector data for the regions you will ever visit so it can run without Internet access. But it doesn’t have the ability to search for businesses and the search for an address functionality is clunky and doesn’t accept plain text, which among other things precludes pasting data that’s copied from email or SMS. While Osmand provides some important features that Google Maps will probably never provide, it doesn’t provide some of the most used features of Google Maps so uninstalling Google Maps isn’t a good option at the moment.

The K9mail project provides a nice IMAP client for Android [8]. Use K9 with a mail server that you run and you won’t need to use Gmail.

There are alternatives to all the Google applications. It seems that apart from the lack of commercial data and search ability in Osmand an Android device that is used for many serious purposes wouldn’t lack much if it had no Google apps.

Google seems to be going too far in controlling Android. Escaping from their control and helping others to do the same seems to be good for society and good for the users who don’t need apps which are only available in proprietary form.

Related posts:

  1. Dual SIM Phones vs Amaysim vs Contract for Mobile Phones Currently Dick Smith is offering two dual-SIM mobile phones for...
  2. My Ideal Mobile Phone Based on my experience testing the IBM Seer software on...
  3. Old Mobile Phones as Toys In the past I have had parents ask for advice...

Syndicated 2012-05-05 14:31:09 from etbe - Russell Cokeretbe - Russell Coker

3 May 2012 »

Acoustiblok/Thermablok

Acoustiblok is an interesting product for blocking sound, it works by dissipating sound energy through friction within the sound barrier materiel [1]. They sell it in varieties that are designed for use within walls and for use as fences. As it isn’t solid it won’t reflect sound so it can be used to line the walls to stop sound being reflected back at you. It’s design is based on NASA research.

The web site claims that a 3mm sheet of Acoustiblok gives a greater noise reduction than 12 inches (30.7cm) of poured concrete. I am a little dubious about that claim as I’ve read a report of someone using three layers of Acoustiblok to make a quiet room for recording music (and to be used as a play-room for an Autistic child). I find it difficult to imagine someone needing a meter of concrete to stop any sort of noise that they might encounter in a residential area so the fact that someone needed three layers of Acoustiblok is an indication that it might not be quite as good as they claim (although there is the possibility that Acoustiblok was badly installed). I wonder whether the claims about concrete concern particular frequencies. The technical specifications and product comparisons page [2] shows that Acoustiblok is least effective at 130Hz where it only reduces noise by 12dB and that it’s effectiveness increases to 38dB at 5KHz. So perhaps a concrete wall to stop low frequencies and Acoustiblok to stop high frequencies would be the best solution.

The Australian distributor for Acoustiblok is based in Brisbane [3].

The same company also sells Thermablok [4] which is the first aerogel based insulation that I’ve seen being advertised for commercial sale. I guess that it must be rather expensive as they are mostly advertising it for use as thin strips to cover stud faces (steel studs conduct heat well and can cause a lot of heat loss). A note in their FAQ says that it’s available in rolls for insulating entire walls or floors. The FAQ also indicates that they sell samples suitable for science classes. They are also apparently looking for retailers, it would be nice if someone wanted to sell this in Australia.

Related posts:

  1. Noise Canceling Headphones My patience with the noise of airlines has run out,...
  2. Testing Noise Canceling Headphones This evening I tested some Noise Canceling Headphones (as described...
  3. I Bought the Bose QC-15 I bought the Bose QC15 noise canceling headphones for my...

Syndicated 2012-05-03 05:44:23 from etbe - Russell Cokeretbe - Russell Coker

30 Apr 2012 »

The Royal Caribbean Official Android app

I’ve just played with the official Android app [1] for the Royal Carribean cruise line [2]. The cruise line is apparently great (I’ve never been on one of their ships but the reviews are good) but the Android app isn’t.

Net Access

The most obvious and significant problem with this app is that it’s entirely useless without net access. All data of note comes from the Internet which means that the program is useless in any location where Internet access is unavailable (or unreasonably expensive). They wrote an app about cruising that can’t be used on a cruise ship! Did they even think about what they were doing?

The correct thing to do when writing such an application is to have all basic data about all ships included in the app. This means that when they change the deck plan of a ship they need to release a new version of the app and have people download it. Having done a lot of software development I understand that forcing software updates (even updates to included data files) involves some effort and expense. But when they spend $20,000,000 to update a ship (which is about the minimum that is spent for a major ship in dry-dock according to TV documentaries I’ve watched) it seems quite reasonable to budget $10,000 to release new software. Also one benefit of updating the software is that it can promote the changes, after spending tens of millions of dollars improving a ship they probably want to promote that to customers and pushing a new app update with adverts for the improved ship seems like a good way of doing that.

There is some data that can’t reasonably be included in the app due to size constraints with photos of ships being the most notable example. The solution to this is to provide an option for the user to cache the data that interests them. For example if I was meeting some people to discuss the possibility of a group cruise on a RCL ship then I could download all the pictures of that ship on my home Wifi network and then have them all available with no delay or 3G costs.

Also the app seems to hang if net access is temporarily interrupted. As phones are expected to have unreliable net access this is also a significant flaw.

Maps

The maps of the ships are comprised of a series of pictures which each show one deck. In addition to being downloaded (not cached or included in the app) they aren’t scalable (they should be SVG or at least allow zooming the high resolution pictures) and they don’t allow a 3D view. The paper maps used to promote cruises (including RCI cruises) and which are given to all passengers on Princess cruises (I’m not sure about other lines) show a side cutaway view of the ship which is handy for working out which things are near where you are. It seems that an ideal cruise ship mapping program would have some sort of 3D component, maybe X3D.

My experience is that a two night cruise isn’t long enough to become familiar with one of the smaller cruise ships. Using a map is essential and a smart phone is a good way of managing such a map as typical 2D paper maps just aren’t good enough for such a large and complex structure.

Photos

One of the significant things that is wrong with the app is a lack of care in displaying the photos of ships. They display three pictures of the Allure of the Seas (one of the two newest, biggest, and most luxurious ships in their fleet), but one of those three photos is actually of the Oasis of the Seas. The fact that the two ships are almost identical is no excuse, there is a principle at stake! Also only having three pictures is pretty poor, there is no way that less than 50 pictures could do justice to such a big ship!

A Google search for the words cruise and photos turns up many sites with pictures of cruise ships and it’s not particularly difficult to find pictures of any particular ship. Photographs by customers are often of high quality as some of the better DSLR cameras are in the same price range as some of the cheaper tickets for cruise ships. Probably the best thing that RCI could do is to run a contest and allow their customers to enter photos and vote towards the winning entries. That would get them photos that aren’t as sterile as the official photos and which include the things that are of most interest to customers.

Finally in terms of caching, pictures are the most easily cached source of data and as phones get higher resolution they keep getting bigger. The storage space for a modern phone is equivalent to the entire 3G download quota for about a year on an affordable Australian 3G plan. When dealing with photos downloaded from the net the default should be to cache everything.

Navigation

Navigating a smart-phone app is a lot more difficult than navigating the same data on a desktop system (which would be in a web browser). Users can compensate for some deficiencies with web site organisation by using a large monitor and having several web browser windows with multiple tabs. But with a phone it should be possible to switch between things quickly.

The main menu has a “View Our Ships” option (which allows viewing deck plans and pictures) and a “View Our Staterooms” option which offers a list of ships and then describes the state rooms available for each one. This means that you can’t see all the information about a ship in one place and even worse you can’t easily compare ships. As it seems likely that people will want to use this app for selecting a cruise it should be possible to select a few ships that are of interest and then quickly flip between them. For example the Rhapsody of the Seas and the Voyager of the Seas are cruising in my part of the world so it would be nice if I could tell the app to compare those ships and then allow me to view a page about one ship and then flick to the equivalent page about the other ship.

Another notable problem is that the ships are listed in alphabetical order. The sensible thing to do is to list them by class going from biggest to smallest.

Lessons to be Learned

These problems aren’t specific to the RCI app, many other Android apps have the same flaws. For example the Google Play market app doesn’t cache the icons of the installed apps so every time I want to see a list of installed apps it goes slow and wastes some of my bandwidth. Doing something wrong in the same way as Google isn’t necessarily a great mistake, although using the Google Play market on a cruise ship is probably very uncommon.

Probably the biggest problem is a lack of testing. They should have sent the developers on a cruise as a live test. Every cruise ship has a sales desk for booking future cruises so it wouldn’t be difficult to have a dozen Android phones at the sales desk to see how real customers who really want to book a cruise find it. I presume that even if net access was available then such a test would fail dismally. If a 3D display of a ship combined with all the data management capabilities of a modern smart phone (which is a lot more powerful than the desktop systems I used prior to 2000) can’t at least be a useful supplement to a stack of paper brochures then it’s probably a failure.

I think that the RCI app is an example of how to make an Android app which doesn’t fall into the more common failings (such as being a quick and dirty port from iOS) but yet still isn’t useful to customers. I recommend that people who develop apps which have an objective of imparting information to users try it out as an example of what not to do. Try a few basic tasks like comparing the three biggest classes of RCI ships in terms of features, after failing to do that with the app you can then use Wikipedia to get the result. But don’t use the Wikipedia client apps, use a tabbed browser such as Opera Mini.

Related posts:

  1. An Introduction to Android I gave a brief introductory talk about Android at this...
  2. Choosing an Android Phone My phone contract ends in a few months, so I’m...
  3. Galaxy S vs Xperia X10 and Android Network Access Galaxy S Review I’ve just been given an indefinite loan...

Syndicated 2012-04-30 05:40:12 from etbe - Russell Cokeretbe - Russell Coker

28 Apr 2012 »

Nando’s Voucher Interpretation

Every year my parents buy a book of vouchers for various businesses in Victoria. It’s one of those deals where businesses (mostly restaurants) pay for advertising space to have their tear-off vouchers in the book (which typically allow a discount of between $5 and $30) and the customers buy the book for something like $40 (I’m not really sure as I don’t pay). Every year I take my pick of the vouchers that don’t suit my parents, the Nando’s chain of chicken and chips restaurants that specialises in Peri-Peri spicy sauce [1] is one that doesn’t suit my parents (they prefer the traditional English-Australian food).

The Nando’s vouchers say “Enjoy one complimentary 1/4 flame-grilled peri-peri chicken item when another 1/4 flame-grilled peri-peri chicken item is purchased” with no explanation of exactly what an “item” is. Every Nando’s store that I’ve been to in the past has interpreted “item” as chicken and chips, usually they include the drink that comes with the “quarter chicken meal” in the “item” that is free. I can do without a second soda as it’s really cheap from the supermarket and I’m not going to drink two at one meal anyway so I’m not bothered when someone interprets the voucher as not involving a free drink. But the lack of chips is annoying.

At the Nando’s store on Swanston St between La Trobe St and Little Lonsdale St they interpret “item” as being just the 1/4 chicken. I think that most people would regard this as an unusual interpretation. If the intent was to only offer 1/4 chicken then the voucher could have stated that a free 1/4 chicken was offered and removed all doubt. The fact that the voucher says a free “1/4 flame-grilled peri-peri chicken item” instead of offering a free “Quarter Chicken” (which is the description for chicken on it’s own on the Nandos menus) seems to be a reasonable indication that more than just the free chicken is offered.

I won’t be attending the Nandos store on Swanston St again and recommend that others avoid it too. Failing to offer the full value on the voucher is annoying, it decreases the value for money (which is a problem given how expensive Nandos is), and it makes me wonder what other cost-saving measures might be used at that store. I’ve got a stack of vouchers (many of which will expire before being used) and the Melbourne CBD has many places to eat.

No related posts.

Syndicated 2012-04-28 09:01:14 from etbe - Russell Cokeretbe - Russell Coker

27 Apr 2012 »

BTRFS and ZFS as Layering Violations

LWN has an interesting article comparing recent developments in the Linux world to the “Unix Wars” that essentially killed every proprietary Unix system [1]. The article is really interesting and I recommend reading it, it’s probably only available to subscribers at the moment but should be generally available in a week or so (I used my Debian access sponsored by HP to read it).

A comment on that article cites my previous post about the reliability of RAID [2] and then goes on to disagree with my conclusion that using the filesystem for everything is the right thing to do.

The Benefits of Layers

I don’t believe as strongly in the BTRFS/ZFS design as the commentator probably thinks. The current way my servers (and a huge number of other Linux systems) work of having RAID to form a reliable array of disks from a set of cheap disks for the purpose of reliability and often capacity or performance is a good thing. I have storage on top of the RAID array and can fix the RAID without bothering about the filesystem(s) – and have done so in the past. I can also test the RAID array without involving any filesystem specific code. Then I have LVM running on top of the RAID array in exactly the same way that it runs on top of a single hard drive or SSD in the case of a laptop or netbook. So Linux on a laptop is much the same as Linux on a server in terms of storage once we get past the issue of whether a single disk or a RAID array is used for the LVM PV, among other things this means that the same code paths are used and I’m less likely to encounter a bug when I install a new system.

LVM provides multiple LVs which can be used for filesystems, swap, or anything else that uses storage. So if a filesystem gets badly corrupted I can umount it, create an LVM snapshot, and then take appropriate measures to try and fix it – without interfering with other filesystems.

When using layered storage I can easily add or change layers when it’s appropriate. For example I have encryption on only some LVs on my laptop and netbook systems (there is no point encrypting the filesystem used for .iso files of Linux distributions) and on some servers I use RAID-0 for cached data.

When using a filesystem like BTRFS or ZFS which includes subvolumes (similar in result to LVM in some cases) and internal RAID you can’t separate the layers. So if something gets corrupted then you have to deal with all the complexity of BTRFS or ZFS instead of just fixing the one layer that has a problem.

Update: One thing I forgot to mention when I first published this is the benefits of layering for some uncommon cases such as network devices. I can run an Ext4 filesystem over a RAID-1 array which has one device on NBD on another system. That’s a bit unusual but it is apparently working well for some people. The internal RAID on ZFS and BTRFS doesn’t support such things and using software RAID underneath ZFS or BTRFS loses some features.

When using DRBD you might have two servers with local RAID arrays, DRBD on top of that, and then an Ext4 filesystem. As any form of RAID other than internal RAID loses reliability features for ZFS and BTRFS that means that no matter how you might implement those filesystems with DRBD it seems that you will lose somehow. It seems that neither BTRFS nor ZFS supports a disconnected RAID mode (like a Linux software RAID with a bitmap so it can resync only the parts that didn’t change) so it’s not possible to use BTRFS or ZFS RAID-1 with an NBD device.

The only viable way of combining ZFS data integrity features with DRBD replication seems to be using a zvol for DRBD and then running Ext4 on top of that.

The Benefits of Integration

When RAID and the filesystem are separate things (with some added abstraction from LVM) it’s difficult to optimise the filesystem for RAID performance at the best of times and impossible in many cases. When the filesystem manages RAID it can optimise it’s operation to match the details of the RAID layout. I believe that in some situations ZFS will use mirroring instead of RAID-Z for small writes to reduce the load and that ZFS will combine writes into a single RAID-Z stripe (or set of contiguous RAID-Z stripes) to improve write performance.

It would be possible to have a RAID driver that includes checksums for all blocks, it could then read from another device when a checksum fails and give some of the reliability features that ZFS and BTRFS offer. Then to provide all the reliability benefits of ZFS you would at least need a filesystem that stores multiple copies of the data which would of course need checksums (because the filesystem could be used on a less reliable block device) and therefore you would end up with two checksums on the same data. Note that if you want to have a RAID array with checksums on all blocks then ZFS has a volume management feature (which is well described by Mark Round) [3]. Such a zvol could be used for a block device in a virtual machine and in an ideal world it would be possible to use one as swap space. But the zvol is apparently managed with all the regular ZFS mechanisms so it’s not a direct list of blocks on disk and thus can’t be extracted if there is a problem with ZFS.

Snapshots are an essential feature by today’s standards. The ability to create lots of snapshots with low overhead is a significant feature of filesystems like BTRFS and ZFS. Now it is possible to run BTRFS or ZFS on top of a volume manager like LVM which does snapshots to cover the case of the filesystem getting corrupted. But again that would end up with two sets of overhead.

The way that ZFS supports snapshots which inherit encryption keys is also interesting.

Conclusion

It’s technically possible to implement some of the ZFS features as separate layers, such as a software RAID implementation that put checksums on all blocks. But it appears that there isn’t much interest in developing such things. So while people would use it (and people are using ZFS ZVols as block devices for other filesystems as described in a comment on Mark Round’s blog) it’s probably not going to be implemented.

Therefore we have a choice of all the complexity and features of BTRFS or ZFS, or the current RAID+LVM+Ext4 option. While the complexity of BTRFS and ZFS is a concern for me (particularly as BTRFS is new and ZFS is really complex and not well supported on Linux) it seems that there is no other option for certain types of large storage at the moment.

ZFS on Linux isn’t a great option for me, but for some of my clients it seems to be the only option. ZFS on Solaris would be a better option in some ways, but that’s not possible when you have important Linux software that needs fast access to the storage.

Related posts:

  1. Starting with BTRFS Based on my investigation of RAID reliability [1] I have...
  2. ZFS vs BTRFS on Cheap Dell Servers I previously wrote about my first experiences with BTRFS [1]....
  3. Reliability of RAID ZDNet has an insightful article by Robin Harris predicting the...

Syndicated 2012-04-27 08:40:05 from etbe - Russell Cokeretbe - Russell Coker

27 Apr 2012 »

BTRFS and ZFS as Layering Violations

LWN has an interesting article comparing recent developments in the Linux world to the “Unix Wars” that essentially killed every proprietary Unix system [1]. The article is really interesting and I recommend reading it, it’s probably only available to subscribers at the moment but should be generally available in a week or so (I used my Debian access sponsored by HP to read it).

A comment on that article cites my previous post about the reliability of RAID [2] and then goes on to disagree with my conclusion that using the filesystem for everything is the right thing to do.

The Benefits of Layers

I don’t believe as strongly in the BTRFS/ZFS design as the commentator probably thinks. The current way my servers (and a huge number of other Linux systems) work of having RAID to form a reliable array of disks from a set of cheap disks for the purpose of reliability and often capacity or performance is a good thing. I have storage on top of the RAID array and can fix the RAID without bothering about the filesystem(s) – and have done so in the past. I can also test the RAID array without involving any filesystem specific code. Then I have LVM running on top of the RAID array in exactly the same way that it runs on top of a single hard drive or SSD in the case of a laptop or netbook. So Linux on a laptop is much the same as Linux on a server in terms of storage once we get past the issue of whether a single disk or a RAID array is used for the LVM PV, among other things this means that the same code paths are used and I’m less likely to encounter a bug when I install a new system.

LVM provides multiple LVs which can be used for filesystems, swap, or anything else that uses storage. So if a filesystem gets badly corrupted I can umount it, create an LVM snapshot, and then take appropriate measures to try and fix it – without interfering with other filesystems.

When using layered storage I can easily add or change layers when it’s appropriate. For example I have encryption on only some LVs on my laptop and netbook systems (there is no point encrypting the filesystem used for .iso files of Linux distributions) and on some servers I use RAID-0 for cached data.

When using a filesystem like BTRFS or ZFS which includes subvolumes (similar in result to LVM in some cases) and internal RAID you can’t separate the layers. So if something gets corrupted then you have to deal with all the complexity of BTRFS or ZFS instead of just fixing the one layer that has a problem.

The Benefits of Integration

When RAID and the filesystem are separate things (with some added abstraction from LVM) it’s difficult to optimise the filesystem for RAID performance at the best of times and impossible in many cases. When the filesystem manages RAID it can optimise it’s operation to match the details of the RAID layout. I believe that in some situations ZFS will use mirroring instead of RAID-Z for small writes to reduce the load and that ZFS will combine writes into a single RAID-Z stripe (or set of contiguous RAID-Z stripes) to improve write performance.

It would be possible to have a RAID driver that includes checksums for all blocks, it could then read from another device when a checksum fails and give some of the reliability features that ZFS and BTRFS offer. Then to provide all the reliability benefits of ZFS you would at least need a filesystem that stores multiple copies of the data which would of course need checksums (because the filesystem could be used on a less reliable block device) and therefore you would end up with two checksums on the same data. Note that if you want to have a RAID array with checksums on all blocks then ZFS has a volume management feature (which is well described by Mark Round) [3]. Such a zvol could be used for a block device in a virtual machine and in an ideal world it would be possible to use one as swap space. But the zvol is apparently managed with all the regular ZFS mechanisms so it’s not a direct list of blocks on disk and thus can’t be extracted if there is a problem with ZFS.

Snapshots are an essential feature by today’s standards. The ability to create lots of snapshots with low overhead is a significant feature of filesystems like BTRFS and ZFS. Now it is possible to run BTRFS or ZFS on top of a volume manager like LVM which does snapshots to cover the case of the filesystem getting corrupted. But again that would end up with two sets of overhead.

The way that ZFS supports snapshots which inherit encryption keys is also interesting.

Conclusion

It’s technically possible to implement some of the ZFS features as separate layers, such as a software RAID implementation that put checksums on all blocks. But it appears that there isn’t much interest in developing such things. So while people would use it (and people are using ZFS ZVols as block devices for other filesystems as described in a comment on Mark Round’s blog) it’s probably not going to be implemented.

Therefore we have a choice of all the complexity and features of BTRFS or ZFS, or the current RAID+LVM+Ext4 option. While the complexity of BTRFS and ZFS is a concern for me (particularly as BTRFS is new and ZFS is really complex and not well supported on Linux) it seems that there is no other option for certain types of large storage at the moment.

ZFS on Linux isn’t a great option for me, but for some of my clients it seems to be the only option. ZFS on Solaris would be a better option in some ways, but that’s not possible when you have important Linux software that needs fast access to the storage.

Related posts:

  1. Starting with BTRFS Based on my investigation of RAID reliability [1] I have...
  2. ZFS vs BTRFS on Cheap Dell Servers I previously wrote about my first experiences with BTRFS [1]....
  3. Some RAID Issues I just read an interesting paper titled An Analysis of...

Syndicated 2012-04-27 07:10:05 from etbe - Russell Cokeretbe - Russell Coker

23 Apr 2012 »

Links April 2012

Karen Tse gave an interesting TED talk about how to stop police torture as an investigative tool [1]. Mostly it’s about training and empowering public defenders.

Phil Plait gave an interesting TED talk about how to defend the Earth from asteroids [2].

Julian Baggini wrote an interesting article for the Financial Times about the persecution of Atheists in the US [3].

Charlie Todd of Improv Everywhere gave an amusing TED talk about Improv events that he has run [4]. He is most famous for organising people to wear blue shirts and khaki pants in Best Buy, but he’s done lots of other funny things.

Paul Zak gave an interesting TED talk about trust, morality, and oxytocin [5]. One of the many interesting fact that he shared is that oxytocin levels can significantly increase when using social networking sites. So people who use Facebook etc are likely to be more trustworthy as well as more trusting.

The Occupy the Judge Rotenberg Center movement aims to stop the torture of Autistic children in the US [6], Anonymous is involved in that too.

Paul Lewis gave an insightful TED talk about the use of crowdsourced data in news reporting [7]. A lot of the analysis of “citizen journalism” is based on comparing bloggers with full-time paid journalists, but Paul describes how professional full-time journalistic work can be greatly assisted by random people filming and photographing things that seem noteworthy. Make sure your next phone has the best possible camera – phone cameras will never be great but the quality of the camera you have with you is what matters.

Sam Harris published an interesting interview with Tim Prowse who is a Baptist minister who faked belief for two years after becoming an atheist [8A]. He also references The Clergy Project – a support group for atheists who are current or former members of the clergy [8B].

Cracked has an insightful article about 6 things that rich people need to stop saying [9]. How do the 1% not understand these things?

Barack Obama and Nichelle Nichols (who played Lt Uhura in the original Star Treck) give the Vulcan Salute in the White-House [10].

Gabriel Arana wrote an insightful article about his experiences with the ex-gay movement [11]. The “therapist” who hurt him so much is still doing the same to other victims.

S#!T Ignorant People Say To Autistics is an interesting youtube video about ignorant and annoying people [12]. Strangely I’ve received little of that myself, I wonder whether women on the Autism Spectrum get a lot more of that than men.

At GManCaseFile an ex-FBI agent has written an informative post about how the TSA is failing [13].

The Nieder Family has an interesting article about how patents are being used to prevent the creation of assisted communication (AAC) devices for children [14]. Apparently the company that has the patents wants all AAC devices to be really expensive and profitable for them. This is yet another example of patents doing harm not good.

Renew Economy has an informative article by Giles Parkinson about the affect that solar power generation will have on power prices [15]. In short as solar systems produce power when it’s most needed (during the day and at the hottest time of the day for warm climates) it will dramatically reduce the auction price for wholesale power. That will hurt the business of the power companies and also allow lower prices on the retail market.

Related posts:

  1. Links March 2012 Washington’s Blog has an informative summary of recent articles about...
  2. Links February 2012 Sociological Images has an interesting article about the attempts to...
  3. Links April 2010 Sam Harris gave an interesting TED talk about whether there...

Syndicated 2012-04-23 14:59:31 from etbe - Russell Cokeretbe - Russell Coker

22 Apr 2012 »

Neighborhood Watch

While writing my previous post I heard a huge noise at the front of my house. I found one man being restrained in a seated position on the ground at my front door, the man who was holding him down was accusing him of theft and asking me to call the police, and a woman was hanging around and crying.

When calling the police I discovered that Optus (the Telco that provides the virtual service which Virgin Mobile uses) doesn’t accept 112 as an emergency number! This combined with the fact that CyanogenMod 7 on my phone doesn’t accept 000 as an emergency number meant that I had to unlock my phone before calling the police. Unlocking your phone late at night when there’s a situation that needs police attention isn’t as easy as you would hope. As an aside there are usually no penalties for testing the emergency service on your phone, people who install PABX systems and other significant telephony devices test emergency services calls as a matter of routine, so testing emergency calls from your phone is a really good idea. If anyone knows how to configure CyanogenMod 7 to support 000 as an emergency call then please let me know!

Anyway the man who was held down claimed that a friend of his had given him a bag containing tools that he had lugged from some place not particularly near my house. The man who was holding him down said that he witnessed the other man stealing the tools from his neighbor – not far from my house. The woman was apparently the girlfriend of the man who was accused of burglary.

The end result was that the police arrested the man who was accused of burglary and his girlfriend. He didn’t have any obvious injuries and the police said that the man who detained him did them a favor, so it seems unlikely that there will be any assault charges filed. Presumably the man who detained the burglar is explaining it all at the police station now, I hope the police gave him a chance to put on pants and shoes first.

The man who made the burglary accusation said that his house was robbed last night which is why he was more observant than usual tonight.

This makes me glad of my policy of rejecting every job offer which involves moving to the US. In Australia hand guns are really hard to get so there’s no way that a house burglary will involve a gun and there’s also no way that someone who wants to help the police will have a gun. So while it was unpleasant to have this happen at my front door it didn’t involve any risk to me. It could have ended up with someone other than me getting a beating but the probability of serious injury or death for them was quite low. As everyone knew that no-one had a gun and no-one wanted to be charged with assault it made sense for everyone to avoid excessive force. From what I saw no excessive force was used.

The police arrived fairly quickly and EVERYONE was glad to see them. All up it took a bit more than 30 minutes from the first noise to the police departing after arresting both suspects and filling out a bunch of paperwork. I was impressed by that!

Related posts:

  1. CyanogenMod and the Galaxy S Thanks to some advice from Philipp Kern I have now...

Syndicated 2012-04-22 16:00:28 from etbe - Russell Cokeretbe - Russell Coker

22 Apr 2012 »

Autism as an Excuse

A Polish geek going by the handle of mmemuar has recently written a blog post claiming that people use Autism as an excuse for bad behavior [1]. He gets enough things wrong in one short post to make it worth debunking it. It seems that Google’s translation of Polish isn’t as good as some other languages, but unless Google mistranslated about a dozen sections such that they had the exact opposite meaning then mmemuar’s post has a lot of wrong ideas.

Does Anyone use Autism as an Excuse?

I’ve read a lot of blog posts written by people on the Autism Spectrum, read many forum discussions, and talked to more than a few people in person. So far I haven’t yet encountered any evidence of people using an Autism Spectrum Disorder (ASD) diagnosis as an excuse. There are probably about 70,000,000 people who meet the diagnostic criteria (most of whom have not been diagnosed due to not having access to anyone who is qualified to do an assessment). The number of people who have been diagnosed is large enough that I couldn’t claim that none of them have ever used it as an excuse. The number of self-diagnosed people is also large enough that there has to be some people who wouldn’t get diagnosed if professionally assessed. But I don’t see any evidence that using an ASD as an excuse is at all common.

I imagine that some people would take someone merely mentioning the fact that they have an ASD diagnosis in a public place (EG a mailing list or a blog post) as some sort of an excuse. One problem with such an interpretation is that for every way in which people on the Autism Spectrum annoy other people it’s the ones who aren’t diagnosed (or who reject a diagnosis) that will do it the most. Being diagnosed with an ASD is correlated with annoying other people less. Another problem is that keeping quiet about such things when they get raised for discussion so often takes a psychological toll.

When someone is diagnosed as an adult a fairly common reaction is to study Psychology and Sociology (usually through web sites such as Sociological Images – which I highly recommend [2]) and try to get a better understanding of other people. Any time you assume that everyone else thinks like you then you will get things wrong, when someone gets an ASD diagnosis they will probably take more care to avoid making such mistakes.

Is Autism Obvious?

Mmemuar says “if you were full of autistic, which is very easy to overheat the brain from excess signals at the input, it would be very obvious to all“. There are some people who are utterly incapable of acting like an NT. But the majority of people on the Autism Spectrum have some ability to pass as NT, it just takes a lot of effort. So if someone is spending all their effort to walk, talk, and make eye contact in a way that most people expect then they will have little spare effort for other things. This can result in them having little patience for other people. The solution to this is to not require people to look average.

Even apart from Autism there are people who fidget, don’t make eye contact in the way you expect, and do other things a little differently. Being tolerant of such things won’t hurt you and will generally make things easier for everyone.

TheAnMish gave a good Youtube presentation about the way that she has to act “normal” [3]. Note that while her video represents her own personal experiences (which differ slightly from those of other Aspies – particularly male Aspies) they are regarded as representative enough for her video to be shown by Tony Attwood (a world renowned expert on Asperger Syndrome) at a conference about women on the Autism Spectrum. As an aside I disagree with her use of the word “normal” without scare-quotes.

Learning about Psychology through Sci-Fi

In the end of the blog post and in some of the comments there is discussion about learning to understand people through reading sci-fi books. The first problem with this is that fiction books generally have a range of characters that is determined by the author’s understanding of people, if you read multiple books by an author then you will usually notice the same character types. The second problem is that characters in fiction books are simplified to fit into a reasonable sized book, in real life people have lots of really boring reasons for doing what they do, in fiction only stuff that is interesting ends up in print.

But the biggest problem is that fiction books just aren’t a good way of learning about people. Whatever lessons might be in a fiction book will probably be missed by a reader who is concentrating on the plot. You can learn things about people by reading an analysis of literature by a Psychologist or a Sociologist, but apart from that you probably won’t be able to learn things if you don’t already know them.

As an aside, my experience of reading sci-fi books suggests that some of the popular sci-fi authors have such a poor understanding of people that it impairs their ability to write believable fiction about human characters. If I was going to try and learn about people by reading fiction I’d choose something that’s been popular for a long time. If a book has been popular for more than 100 years and sold well in different times, cultures, and languages then it probably has something to say about the human condition.

Catching up on Youth

Mmemuar says “absolutely nothing prevents you to the age of twenty he began to catch up on some ‘of youth’“. Actually one significant thing is that the human brain develops in particular stages, the older you get the more difficult it becomes to learn things. So even if it was just a matter of learning things someone who was behind at age 20 would have some significant difficulty in catching up. But some things just can’t be learned, for example someone who has extreme discomfort in making eye contact can’t just learn to be happy with it.

Also there are some things which people would have learned if it was possible. For example people who have Prosopagnosia (an inability to recognise faces) suffer extreme bullying in school, if they could just learn to recognise people then they surely would do so, so if they complete school without learning then it’s probably going to be impossible for them. Prosopagnosia is one of many conditions which can contribute to social difficulties and therefore contribute to an ASD diagnosis.

Can Aspies get Married?

Some people think that people on the Autism Spectrum can’t get married. In fact this belief is so widely held that some people who seem very obviously Autistic are convinced that they are NT simply because they are married! The fact that there are more than few books offering advice to people who have married someone who is on the Autism Spectrum is clear proof that such claims are bogus.

The Relevance to Geek Communities

People who meet the diagnostic criteria for Asperger Syndrome will almost certainly be Geeks due to the “Restricted, repetitive patterns of behavior, interests, or activities” section of the diagnostic criteria (the proposed revision for the DSM-V is the best reference I know for this [4]) as the modern use of the term Geek applies to anyone who has an extreme interest in something. The more Geeky a community is the greater the incidence of people who could be diagnosed with an ASD.

Spreading ideas such as those of mmemuar will lead to people not being assessed for an ASD. I would have been assessed earlier if it wasn’t for hearing an influential member of the Linux community say some things which were similar in concept (although not as deliberate). Having people not be assessed is bad for the individuals in question and bad for the community.

Related posts:

  1. Autism Awareness and the Free Software Community It’s Autism Awareness Month April is Autism Awareness month, there...
  2. Autism vs Asperger Syndrome Diagnostic Changes for Autism Spectrum Disorders Currently Asperger Syndrome (AS)...
  3. Communication Shutdown and Autism The AEIOU Foundation The AEIOU Foundation [1] is a support...

Syndicated 2012-04-22 15:03:46 from etbe - Russell Cokeretbe - Russell Coker

17 Apr 2012 »

The Most Important things for running a Reliable Internet Service

One of my clients is currently investigating new hosting arrangements. It’s a bit of a complex process because there are lots of architectural issues relating to things such as the storage and backup of some terabytes of data and some serious computation on the data. Among other options we are considering cheap servers in the EX range from Hetzner [1] which provide 3TB of RAID-1 storage per server along with reasonable CPU power and RAM and Amazon EC2 [2]. Hetzner and Amazon aren’t the only companies providing services that can be used to solve my client’s problems, but they both provide good value for what they provide and we have prior experience with them.

To add an extra complication my client did some web research on hosting companies and found that Hetzner wasn’t even in the list of reliable hosting companies (whichever list that was). This is in some ways not particularly surprising, Hetzner offers servers without a full management interface (you can’t see a serial console or a KVM, you merely get access to reset it) and the best value servers (the only servers to consider for many terabytes of data) have SATA disks which presumably have a lower MTBF than SAS disks.

But I don’t think that this is a real problem. Even when hardware that’s designed for the desktop is run in a server room the reliability tends to be reasonable. My experience is that a desktop PC with two hard drives in a RAID-1 array will give a level of reliability in practice that compares very well to an expensive server with ECC RAM, redundant fans, redundant PSUs, etc.

My experience is that the most critical factor for server reliability is management. A server that is designed to be reliable can give very poor uptime if poorly maintained or if there is no rapid way of discovering and fixing problems. But a system that is designed to be cheap can give quite good uptime if well maintained, if problems can be repidly discovered and fixed.

A Brief Overview of Managing Servers

There are text books about how to manage servers, so obviously I can’t cover the topic in detail in a blog post. But here are some quick points. Note that I’m not claiming that this list includes everything, please add comments about anything particularly noteworthy that you think I’ve missed.

  1. For a server to be well managed it needs to be kept up to date. It’s probably a good idea for management to have this on the list of things to do. A plan to check for necessary updates and apply them at fixed times (at least once a week) would be a good thing. My experience is that usually managers don’t have anything to do with this and sysadmins either apply patches or not at their own whim.
  2. It is really ideal for people to know how all the software works. For every piece of software that’s running it should either have come from a source that provides some degree of support (EG a Linux distribution) or be maintained by someone who knows it well. When you install custom software from people who become unavailable then it puts the reliability of the entire system at risk – if anything breaks then you won’t be able to get it fixed quickly.
  3. It should be possible to rapidly discover problems, having a client phone you to tell you that your web site is offline is a bad thing. Ideally you will have software like Nagios monitoring the network and reporting problems via a SMS gateway service such as ClickaTell.com. I am not sure that Nagios is the best network monitoring system or that ClickaTell is the best SMS gateway, but they have both worked well in my experience. If you think that there are better options for either of those then please write a comment.
  4. It should be possible to rapidly fix problems. That means that a sysadmin must be available 24*7 to respond to SMS and you must have a backup sysadmin for when the main person takes a holiday, or ideally two backup sysadmins so that if one is on holiday and another has an emergency then problems can still be fixed. Another thing to consider is that an increasing number of hotels, resorts, and cruise ships are providing net access. So you could decrease your need for backup sysadmins if you give a holiday bonus to a sysadmin who uses a hotel, resort, or cruise ship that has good net access. ;)
  5. If it seems likely that there may be some staff changes then it’s a really good idea to hire a potential replacement on a casual basis so that they can learn how things work. There have been a few occasions when I started a sysadmin contract after the old sysadmin ceased being on speaking terms with the company owner. This made it difficult for me to learn what’s going on.
  6. If your network is in any way complex (IE it’s something that needs some skill to manage) then it will probably be impossible to hire someone who has experience in all the areas of technology at a salary you are prepared to pay. So you should assume that whoever you hire will do some learning on the job. This isn’t necessarily a problem but is something that needs to be considered. If you use some unusual hardware or software and want it to run reliably then you should have a spare system for testing so that the types of mistake which are typically made in the learning process are not made on your production network.

Conclusion

If you have a business which depends on running servers on the Internet and you don’t do all the things in the above list then the reliability of a service like Hetzner probably isn’t going to be an issue at all.

Related posts:

  1. Servers vs Phones Hetzner have recently updated their offerings to include servers with...
  2. Why Internet Access in Australia Sucks In a comment on my post about (relatively) Cheap Net...
  3. The National Cost of Slow Internet Access Australia has slow Internet access when compared to other first-world...

Syndicated 2012-04-17 12:55:11 from etbe - Russell Cokeretbe - Russell Coker

963 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!