benad is currently certified at Apprentice level.

Name: Benoit Nadeau
Member since: 2002-04-03 17:23:09
Last Login: 2014-04-22 17:48:00

FOAF RDF Share This

Homepage: http://benad.me/

Notes:

I am Benoit Nadeau, jr. eng. in Software Engineering,
living and working in Montreal, Canada.

Projects

Recent blog entries by benad

Syndication: RSS 2.0

Flattening Circular Buffers

A few weeks ago I discovered TPCircularBuffer, a circular buffer implementation for Darwin operating system implementations, including Mac OS X and iOS. Now, I’ve implemented circular buffers before, so I though there wasn’t much need for yet another circular buffer implementation (let alone one specific to iOS), until I noticed something very interesting in the code.

A trick TPCircularBuffer uses is to map two adjacent memory blocks to the same buffer. The buffer holds the actual data, and the virtual memory manager ensures that both maps contain the exact same data, since effectively both virtual memory blocks remaps to the same memory. This makes things a lot easier than my naive implementations: Rather than dealing with convoluted pointer arithmetics each time the producer or consumer reads or writes a sequence of values that cross the end of the buffer, a simple linear read or write works. In fact, the pointers from that doubly-mapped memory can be safely given to any normal function that accepts a pointer, removing the need to make memory copies before each use of the buffer by an external function.

In fact, this optimization is so common that a previous version of the Wikipedia page for circular buffers had some sample code using common POSIX functions. There’s even a 10-year-old VRB - Virtual Ring Buffer library for Linux systems. As for Windows, I’ve yet to seen some good sample code, but you can do the equivalent with CreateFileMapping and MapViewOfFile.

Both Wikipedia’s and VRB’s implementations can be misleading, and not very portable though. On Darwin, and I suspect BSD and many other systems, the mapped memory must be fully aligned to the size of a memory page (”allocation granularity” in Windows terms). On POSIX, that means using the value of sysconf(_SC_PAGESIZE). Since most of the times the page size is a power of 2, that could explain the otherwise strange buffer->count_bytes = 1UL << order from Wikipedia’s sample code.

By the way, I’d like to reiterate how poor the built-in Mac OS X documentation is for POSIX and UNIX-like functions. Though it does warn pretty well about page size alignment and the risks involved with MAP_FIXED of mmap, the rest of the documentation fails to mention how to set permissions of the memory map. Thankfully, the latest Linux man pages for the same functions are far better documented.

Syndicated 2016-05-23 18:01:52 from Benad's Blog

The Static Blog

A quick note to mention that I added The Static Blog to my main web site, discussing the relocation of the blog you’re reading right now to this site.

Syndicated 2016-04-27 01:11:57 from Benad's Blog

Final Blog Move?

This is a short post to mention that my blog, originally hosted on Squarespace as blog.benad.me has now moved alongside my main web site under benad.me/blog.

All links to each post and the RSS feed should now automatically redirect to the new location. This may have created some duplicates in your feed reader.

There is still some clean up to do to make the older posts look better, and I need to post a longer article to explain the rationale behind this, but for now everything should be working fine. The blog.benad.me domain will remain active as an option if something were to happen, but for the time being my blog should remain where it is.

Syndicated 2016-03-29 19:51:33 from Benad's Blog

Amazon Cloud Drive Backend for Duplicity

To follow up on my previous post, I wrote acdbackend, to add Amazon Cloud Drive support to duplicity, by wrapping around the acd_cli tool.

Actually, adding online storage services to duplicity is pleasantly easy to implement. If you have some command-line tool for your service that supports download and upload using stdin/stdout, file listing, and deleting files, then in an hour you can have it working in duplicity.

Syndicated 2016-03-03 02:16:19 from Benad's Blog

DIY Backup

While in the past I did recommend CrashPlan as an online backup solution, I stopped using it in December. At first I used it because their multi-year, unlimited plans had reasonable prices, and was the only online backup (back in 2011) that had client-side encryption support. But over the years I ran into multiple major issues. In 2013, they started excluding backing up iOS backups done in iTunes in a transparent background update, even though they would publicly say otherwise. In fact, that hidden file exclusion list was pushed from their "enterprise" version, which has now moved to a much nicer version 5 while they abandoned their home users to version 4. In November, their Linux client started requiring Java 1.7, so my older client running in 1.6 kept downloading updates and failing to install it until the hard drive was full. Their pricing just kept increasing over time, making it difficult for me to keep reniewing.

I moved to iDrive, which is half the cost of CrashPlan and works pretty well, though I'm still a bit worried that I have to trust client-side encryption to some closed-source software. Also, if you back up anything beyond 1 TB their pricing becomes punitive.

The main issue that I have with all those backup services is that your backups become locked-in into online storage plans that are more expensive than competing generic cloud storage providers, and your valuable backups are held hostage if they increase their pricing. Even tarsnap, with its open-source client for the paranoid, locks you in an expensive storage plan, since the client requires some closed-source server software that only they host. I miss the days of older backup software like MobileMe Backup, where the backup software was somewhat separate from the actual storage solution.

Arq Backup looks more like this traditional backup software I was looking for. It can back up to a handful of cloud storage providers, with different pricing models, and many with free initial storage plans if you have small backups. The software is $40 per machine, and then you're free to pick any support cloud storage. The software isn't open-source, but the recovery software is open-source and documented, so you can vouch for its encryption to some extent.

But what if you're on Linux, or insist on an open-source solution (especially for the encryption part)? If you simply want to back up some files once, with no history, you can combine encfs, in "reverse" mode to have an encrypted view of your existing files, with Rclone. Note that with this approach extended file information may be lost in the transfer. If you want a more thourough versioned backup solution, Duplicity should work fine. It encrypts the files with GPG, and does file-level binary deltas to make backup files as small as possible. If duplicity doesn't support your cloud storage directly, you can store the backups to disk and sync them with Rclone. To make using Duplicity easier, you can also use the wrapper tool duply.

As for what cloud storage provider to use, it depends on your needs. If you can fit your backups in less than about 15 GB, you can use the free version of Google Drive. If you want a flexible pricing and good performance at the lowest cost, Google Nearline looks like a great deal at 0.01$ per GB per month. If you already have Office 365, then you already have 1 TB of OneDrive, though downloads can be a bit slow. The Unlimited plan of Amazon Cloud Drive has good transfer speeds and is worry-free, though Duplicity doesn't support it.

Syndicated 2016-02-25 03:56:28 from Benad's Blog

122 older entries...

 

benad certified others as follows:

  • benad certified benad as Journeyer
  • benad certified llasram as Journeyer
  • benad certified shlomif as Journeyer

Others have certified benad as follows:

  • benad certified benad as Journeyer
  • llasram certified benad as Journeyer
  • pasky certified benad as Apprentice

[ Certification disabled because you're not logged in. ]

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

X
Share this page