Recent blog entries

9 Feb 2010 bagder   » (Master)

My Rockbox talk at Fosdem 2010

As I’ve mentioned several times before in this blog, I did a talk about Rockbox and reverse engineering at Fosdem 2010 Feburary 6-7 in Brussels, and since there was no “pre-arranged” video recording of the talks in the embedded devroom, Peter D’Hoye stepped up and recorded the whole thing using his Nokia n900 phone.

I decided to not make the slides for this talk available separately, as they were more or less the same as the ones I used for my FSCONS 2009 talk, so you can go watch them instead if this video isn’t enough!

Rockbox-talk-Fosdem2010

To view it, I suggest you use VLC or similar and tell it to stream directly from one of these URLs, the file is a 1.1GB one with 848×480 resolution running for 51 minutes. Annoyingly, none of the usual free online video services allow this long ones.

http://www.qnapclub.be/rockbox/fosdem2010_rockbox.mp4

http://download.rockbox.org/movies/fosdem2010_rockbox.mp4

Syndicated 2010-02-09 21:54:52 from daniel.haxx.se

9 Feb 2010 bradfitz   » (Master)

Realtime LiveJournal -> Buzz

If you've seen all my "test" posts over the past few days, you probably knew I was up to something.

Indeed....

If you add your LiveJournal to your Google Profile (and your LJ links to your Google Profile and is crawled), and then you "Connect" your LJ to your Google Buzz account by adding it as a connected site, all your LJ posts flow into Buzz in 1-2 seconds, using PubSubHubbub.

And hopefully LiveJournal will support the Salmon Protocol so comments left in Buzz will flow back to LiveJournal, and vice-versa. But that's the future.

Much love to Brett Slatkin, the Reader team, Brian Stoler and the Buzz team, Sebastian Kanthak, John Panzer, and others who made all this work. Huge team effort, but in the end I think open, decentralized protocols will win and are the future.

Syndicated 2010-02-09 20:24:14 from Brad Fitzpatrick

9 Feb 2010 marnanel   » (Journeyer)

Language icons, etc

There is a tool I look after called ProjectJournal which is used to publish the Metacity Journal posts.  I am considering releasing it, because some people have expressed an interest.

One of the things it does is list all the people who have contributed translations to a project recently.  At the moment, it looks like this (apologies for the old example, but it was the first I found):
  • On branches/gnome-2-22: en_GB by pwithnall, es by jorgegonz
  • On trunk: es by jorgegonz
Things I should perhaps fix about this include
  • writing the translator's full name rather than their username
  • maybe making the translator's name a link to their home page if I had it
  • writing the full name of the language rather than the ISO 639 code
  • alternatively, displaying an icon next to the language-- but it's not obvious what icons to use.  Some people use flags to represent languages, but of course this is rather broken because there's no 1-1 mapping between flags and languages.  Wikipedia uses icons which incorporate the ISO 639 code, which makes it seem hardly worthwhile using icons at all.  I feel this is a bit of a minefield and probably one we should stay out of.
What do you think?

Syndicated 2010-02-09 20:45:50 (Updated 2010-02-09 20:51:14) from Monument

9 Feb 2010 ralsina   » (Master)

Marave 0.4 is out!

Version 0.4 of Marave, a distraction-free fullscreen editor is out at http://marave.googlecode.com

This version includes several bugs fixed and features implemented since 0.4:

  • It works better with WindowMaker (dialogs appeared behind main window)
  • Works better with some Qt styles (combo boxes were unusable)
  • Added support for SVG backgrounds
  • Code cleanups

Marave is free softare released under the GPL, and should work in all major desktop platforms.

I would love feedback on this release, as well as ideas for Marave's future, so if you want to help, please join the mailing list:

http://groups.google.com/group/marave-discuss

Of course, if you like Marave, feel free to give me money


Syndicated 2010-02-09 16:04:44 from Lateral Opinion

9 Feb 2010 dan   » (Master)

The Programmer's Guide to Financial Book-keeping, Part I

Once upon a time I knew enough about bookkeeping to implement a a rudimentary accounting system for the consulting business I was running at the time. Then I got a real job, and after that I forgot most of it. Recently I've had to relearn it all, and as the accountancy/bookkeeping web pages that I've found on the Internet are decidedly mixed (an honourable mention here for the Gnucash manual, which is actually quite good), this time I'm writing it down.

The intended audience for this is chiefly me and people like me: computer programmer types who have to make their systems talk to accounts departments and accountants. If you are looking for more information on bookkeeping or accountancy from a professional perspective, it is less likely to be useful.

It should not be necessary - though it probably is - to state that I hold no professional qualifications and have had no training in the field, and if you want proper advice you'll have to pay for it from someone entitled to give it. This information is offered as-is, and no warranties as to its correctness, usefulness or completeness are offered.

Feedback welcome - see the page footer for details.

<h2> Definition </h2>

Let us define bookkeeping as: the collection and processing of financial records for an entity, with the object that interested parties can learn (1) as of a specified time, how much money (and other valuable stuff) it owns, against how much it owes to other entities; (2) over a specified period of time, how much has come in and how much has gone out. Bookkeeping deals not just with money but with all kinds of valuable stuff: cash, shares, financial instruments, land, saleable equipment, stock in trade, etc etc - in the rest of this post I'll be lumping it all together as "value". <h2> End results </h2>

In the UK, the end processes of bookkeeping/accountancy for a company or other trading entity are usually produced annually -

  • the Balance Sheet - a document of type (1), which lists the assets (stuff we've got) and liabilities (stuff we owe) broken down by category, at the end of the trading year. We start with assets, listed in order from most liquid (e.g. money at hand or in the bank) to least liquid (things we own that would be complicated to sell), then we subtract liabilities (usually ordered from short-term to long-term), then the bottom line is what we're worth. This is often referred to as the Accounting Equation: <center>Assets - Liabilities = Equity</center> although other people will say that Equity is really what the company owes its owners (e.g. the shareholders) so the equity will appear as a liability account and the equation is "Assets = Liabilities". Mathematically it makes no difference.

  • the Profit and Loss account, or P&L - also known in the US as the Income Statement. This is a document of type (2) which lists what's come in and what's gone out over the course of the year.

We probably also want quarterly reports for VAT (that's "Sales Tax" in other countries), and ad-hoc reports for credit control (we need to know who owes us money so we can chase them) and management accounting. <h2> Derivation </h2>

Obviously, if your trading entity is you and you alone and there's no regulatory requirement on you to show anyone else the figures, you can choose any categories you like. But for most of us, there are accepted rules about the breakdown that people want to see, what you're allowed to assign to which categories, and what you'd actually want to assign to which categories (which might be a question with different answers depending on whether you're trying e.g. to maximise profit for the investors or minimise it for the taxman). This kind of decision is what you have an accountant for: keeping the numbers is what you have a bookkeeper for. So, look on the difference between those two roles as a policy/mechanism distinction (and a big difference in hourly rate: don't pay an accountant to do a bookkeeper's job) <h2> Accounts and transactions </h2>

So, with the aid of an accountant we can establish how we need to categorise our assets and liabilities for the reports we need to produce. Each category (or sub-category, or sub-sub-category) is an account: each transfer of value from one account to another is a transaction. A transaction is usually associated with a source document (for example, a purchase order, or an invoice, or a receipt) - the so-called paper trail is not necessarily kept on actual carbon laminate these days, but it's still important. In essence, what we do is record the transactions. <h2> Credit and Debit </h2>

We record each financial transaction as a flow of value from one (or several) accounts into another (or several others). Historically, bookkeepers don't get on with the concept of negative numbers - this is possibly because it can be confusing to have your "Income" account get steadily more negative as the year goes on (we'll come back to why this happens), or maybe just because the principles of double-entry bookkeeping were invented in a time and place (Renaissance Italy) that hadn't really yet heard of negative numbers. Whatever. But the upshot is that they made their own words up instead: the account that loses value is said to be "credited"; the account which gains value is "debited".

This is, of course, completely bass-ackwards from the perspective of normal people, though it has been claimed that the problem is that we're backwards. When the bank send you your statement of account it's printed from their point of view, not yours. So, if you deposit £50 in the Royal NatMid, in their eyes that creates a liability to you (after all, it's money they have but you own). The more money you give them the more they can transfer (debit) to Assets/BranchSafe or Assets/Vault or Assets/SubPrimeMortgages, but they have to credit that transaction to Liabilties/AP/YourNameHere. So the effect is that we perceive being in credit with the bank as a good thing: they see it as a bad thing. It's just a matter of perspective.

(I am slightly suspicious of this explanation. "Credit" and "Debit" are both from Latin roots: /creditum/: "a loan, thing entrusted to another", and /debilitum/: "thing owed," neut pp. of /debere/ "to owe". In the end they're just words, but it's still confusing enough to be just more fuel for my scepticism towards the claim that negative numbers are avoided because they cause confusion. Maybe that's just me.) <h2> Double-entry </h2>

The principle of double-entry accounting is that the value credited in a transaction must equal the value debited - value cannot be created or destroyed. The name comes from paper-based systems: if we have two accounts affected by a transaction, we must enter the transaction details into both. Using a computer, of course, we can enter it once and it will appear in both, but that's not the point. We are interested in the principle of "conservation of value", not so much in the mechanism of how we achieved that in the old days.

A simple example: our shop runs out of float in the till, so on Monday we must visit the bank and get some more cash. We record this as a transfer from the bank account (which is credited) to the till account (which is debited).

Here we're moving value between two asset accounts: they both represent monies that we own - just in different places. So it's pretty easy to see that "conservation of value" holds true. But the principle of double-entry bookkeeping is that the debits and credits in any transaction must always balance, so the astute reader will now be wondering how we do that for a transaction that actually makes the company money. For example, if we do some work and get paid, then the value in the transaction is clearly going into the bank account, but where is it coming from? <h2> Income and expenses </h2>

The answer is that we create "Income" accounts which serve as a proxy for the outside world as it affects our company. So, if we get $200 for configuring Joe's web server, our bank account is debited $200 and the outside world, as represented by our Income (or Income/Sales, or whatever subcategorisation we want to use) account, is credited $200. Expenses accounts serve a similar but opposite role: we pay for stuff (like stationery, utilities, salaries) that makes us poorer (our assets are credited) and the outside world richer (our expenses are debited).

Income and Expense accounts are key to the P&L statement that we will produce at the end of the year, because they act as summaries of our interactions with the world - which is what P&L is all about. The USAnian name for them "Income statement" hints as much.

<h2> Accruals </h2>

Another key concept is accruals. In most businesses there is a delay between when we provide something of value (e.g. do some work) and when we actually get paid: there is also often a delay between when we receive something of value and when we have to pay for it. In a cash accounting system there's nothing we can do about that, but in an accruals system we can create "accounts receivable" and "accounts payable" for these sums which are "in the post". This allows our accounts to say that we are worth $4000 because we have that amount expected to come in from Michael next week, even though we haven't got it in the bank yet. So, this makes payment a two-stage process: first we send an invoice and transfer $4000 from Income/Sales to Assets/AR/Michael, then when he pays it four weeks later (or perhaps four months later if he's a public sector body) we transfer $4000 from Assets/AR/Michael to Assets/Bank. We haven't actually made any new money in that second transaction, but at least it now exists in the bank and not just on paper.

Accounts Payable is similar but opposite. We order office furniture on account, it gets sent with an invoice, and we log that transaction as a transfer from Liabilities/AP/IKEA to Assets/Furniture. When the invoice is due (or three weeks later if you have really good credit control) we send them a cheque and we do another transaction from (crediting) Assets/Bank to (debiting) Liabilities/AP/IKEA, which hopefully reduces the balance of the latter account to zero.

Most of the examples later in this post ignore accruals in much the same way and for the same reason as Kernighan and Ritchie ignore error checking: it slightly obscures the pedagogical point, but that doesn't mean you won't do it for real. <h2> Sale of goods </h2>

If you're selling services, the transaction is Income/Sales->Assets/Bank. That's simple. If you're selling goods, though, (1) you have to buy them first

£6 cr. Assets/Bank = dr. Assets/Inventory/Widgets

and then (2) when you sell them you are selling at a different price.

£10 cr. Income/Sales = dr. Assets/Bank
£6 cr. Assets/Inventory/Widgets = dr. Expenses/Cost of sales

The net effect is to increase Income by a tenner and Expenses by an unwell cephalopod (that's "sick quid" to you. Sorry). Thus both effects of the transaction will be represented on the appropriate P&L rows.

<h2> VAT / Sales Tax </h2>

VAT in the UK is not really ever money we have earnt, it's just money we are collecting on behalf of the nice people at HMRC. So, if we are registered for VAT we must collect it on each sale into a holding account which we send them later, but it's not "ours" and doesn't show in Sales.

£20 cr. Income/Sales + 3.50 cr. Liabilities/VAT = 23.50 dr. Assets/Bank

Watch out for the credit/debits in that transaction. We should end up with cash in the bank (a debit), some of which is owed to the VAT man (credit). If they don't sum to zero, you've done something wrong.

Similarly we can also claim back VAT on purchases from our VAT-registered suppliers

11.75 cr. Assets/Bank = 10.00 dr. Assets/Inventory + 1.75 dr. Assets/Input_VAT

At the end of the quarter, we pay HMRC what we owe them, less what they owe us

1.75 cr. Assets/Input_VAT + 1.75 cr. Assets/Bank = 3.50 dr. Liabilities/VAT

Note that this is not reflected in any Expense account - it shouldn't be, because it wasn't in an Income account to start with

<h2> Year end </h2>

We've already talked about producing the Balance Sheet and P&L. The other action we take at end of year is to close the accounts: in the case of Income and Expenses, we will want to start the following year with a clean sheet. How to do this: after producing the end-of-year reports, move the entire contents of Income and Expenses accounts into a summary "Retained Earnings" account, debiting and crediting as appropriate.

<h2> Contingent concepts </h2>

We have not talked about: journals, day books, cash books, general ledgers, T accounts, and trial balances. Most of these are historical practices that are necessary in manual systems either because the latency of entering everything directly in double-entry form is high (so transactions are initially recorded elsewhere instead), or because there is no automatic checking that the accounts are in balance, or because obtaining summaries of groups of accounts (answering queries like "what's the total AP for all suppliers") isn't a trivial bit of SQL.

<h2> Where next? </h2>

This is Part I of a two-part series. In the second part I'm going to write about my experience implementing all this in Ruby, but that will have to wait until I've done the actual implementation.

Hopefully though, this post should provide you with a view of the principles such that you can google for anything else you see and you have a framework to hang it on.

Syndicated 2010-02-09 14:58:49 from diary at telent netowrks

9 Feb 2010 redi   » (Master)

welcome back, hjclub

9 Feb 2010 bagder   » (Master)

a big curl forward

We’re proudly presenting a major new release of curl and libcurl and we call it 7.20.0.

The primary reason we decided to bump the minor number this time was that we introduce a range of new protocols, but we also did some other rather big works. This is the biggest update to curl and libcurl that have been made in recent years. Let me mention some of the other noteworthy changes and bugfixes:

We fixed a potential security issue, that would occur if an application requested to download compressed HTTP content and told libcurl to automatically uncompress it (CURLOPT_ENCODING) as then libcurl could wrongly call the write callback (CURLOPT_WRITEFUNCTION) with a larger buffer than what is documented to be the maximum size.

TFTP was finally converted to a “proper” protocol internally. By that I mean that it can now be used with the multi interface in an asynchronous way and it has far less special treatments. It is now “just another protocol” basically and that is a good thing. Also, the BLKSIZE problem with TFTP that has haunted us for a while was fixed so I really think this is the best version ever for TFTP in libcurl.

In several different places in the code older versions of libcurl didn’t properly call the progress callback while waiting for some special event to happen. This made the curl tool’s progress meter less responding but perhaps more importantly it prevented apps that use libcurl to abort the transfer during those phases. The affected periods included the ftp connection phase (including the initial FTP commands and responses), waiting for the TCP connect to complete and resolving host names using c-ares.

The DNS cache was found to have at least two bugs that could make entries linger in the database eternally and in another case too long. For apps that use a lot of connections to a lot of hosts, these problems could result in some serious performance punishments when the DNS cache lookups got slower and slower over time.

Users of the funny ftp server drftpd will appreciate that (lib)curl now support the PRET command, which is needed when getting data off such servers in passive mode. It’s a bit of a hack, but what can we do? We didn’t invent it nor can we help that it’s a popular thing to use! ;-)

cURL

Syndicated 2010-02-09 11:42:42 from daniel.haxx.se

9 Feb 2010 zeenix   » (Journeyer)

FOSDEM fun

Just like the last 2 years, I joined the FOSDEM fun. Just like every year there were lots of awesome beers, chocolates, waffles and of course conference full of awesome hackers from all over the world & their awesome talks. In short, the same old 'awesome' experience. Also I finally got to meet Jens George (phako) in person. Another thing that went it a totally unusual (and unexpected) direction was my home directory. Here is how it went:

Early Sunday morning (8:00) my alarm goes off to wake me up in time for my 10:15 am talk. I manage to get up after half an hour of snoozing the alarm, get ready (including getting a shower out of almost non-functional shower), back-up my slides on USB stick and manage to get to the conference room 10 minutes before the talk along with my room-mates, Marc-Andre and Juerg. Failed to get anything to eat or even a cup of coffee before that.

I open-up the lid of my laptop to find out that it won't be able to wake-up from suspend. No problems, happens all the time even since I moved to Ubuntu so I apply the usual solution: Forced reboot. When the system boots I get into another issues that I had been having ever since I moved to Ubuntu: Soon after I provide the password for my encrypted partition (home and swap), I am told that home partition could not be mounted. No problems, I again apply the usual solution: Hit escape key to get to a root shell, where I do this:


# cryptsetup remove home
# cryptsetup create home /dev/sda6
Enter passphrase:
# mount /home


Normally, at this point I just hit CTRL+d and normal boot sequence resumes and everyone on the laptop lives happily ever after. However, this time thats not what happens. `mount` says:


mount: wrong fs type, bad option, bad superblock on /dev/mapper/home,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so


This has also happened before but usually this means I mistyped the password and I just have to repeat the above procedure. However, after trying several times, I keep getting the same message. I give-up and lend Marc-Andre's laptop for the slides show. After the presentation, I tried everything and asked everyone I knew who might have any clues but no help came.

Since I get the same questions each time I tell this story to anyone, I thought I append this blog entry by an FAQ:

Q: Why on earth are you encrypting your partitions:
A: Nokia security policy.

Q: What makes you so sure that you haven't just forgotten the password.
A: Because I keep the exact same password for both the partitions and I am able to successfully decrypt one of them.

Q: Have you tried looking at `dmesg | tail`?
A: Yes, nothing in there.

Q: Have you tried fsck?
A: Yes, that can't make any sense out of the "unencrypted" block device either. Because of this, I suspect something is wrong on the actual (encrypted) device, which fsck can not help with.

Syndicated 2010-02-09 11:21:00 (Updated 2010-02-09 11:21:40) from zeenix

9 Feb 2010 badvogato   » (Master)

redi, we are in agreement. All I am saying is to have a trigger and a procedure, before deleting nodes, make a snap shot of trust-matrix and offending node's diary dump. So that if it offers genuine apology, it can download source with the data that has more meaning to the alien node than to the 'norm' community. what's more, if advogato's authentication model expands to local password store & openID or any third party authentication scheme, hjclub can setup a child-virgule site that allow existing advogatoer here to participate there as well. That's how virgule can expand its influence beyond Free software domain and English domain. cheerio.

9 Feb 2010 etbe   » (Master)

Web Site Validation

Over the last few days I’ve got this blog and my documents blog to conform to valid XHTML according to the W3C validation service [1].

One significant change that I made was to use lower-case for HTML tags. For about 15 years I’ve been using capitals for tags to make them stand out from content and my blogs are the latest in a long line of web sites with that. Naturally I wasn’t going to correct 900 posts manually so I ran a series of SQL commands such as the following on my database server (where X is the Wordpress table prefix):

update X_wp_posts set post_content = replace(post_content,'<PRE>','<pre>');

But make sure you have a good backup of your database before running SQL search and replace commands on your blog data.

After running such commands about 90% of my blog posts conformed, so I only needed to edit about 90 posts to correct things. This process gave some real benefits. One issue is that an apostrophe in a URL must be quoted, otherwise some browsers will link to the desired URL and some will link to a truncated URL. Fixing a couple of variations of this problem resulted in some broken links being fixed. Another issue is that you can’t have paragraphs (<p> tags) within list items, fixing this made some of my posts align correctly – it was a tricky fix, in some cases I had to use <br/> to break up text in a list item and sometimes I replaced lists with different sections delimited by <h3> headings (which apparently is rumored to give better SEO).

It would make a really nice Wordpress feature to be able to do W3C validation as part of the publishing process, ideally an attempt to publish or schedule a post would result in a message saying “saved as a draft because it’s not valid XHTML” if the checks failed. The source to the W3C validation software is significantly larger than Wordpress [2], but it seems to me that there are two main types of Wordpress installations, small ones for personal use (which tend to be on fairly idle servers) and big ones that have so much traffic that the resource usage of validation would be nothing compared to the ongoing load.

As there seems to be no way of validating my posts before publication my best option is the W3C button I now have on my blog. This allows me to validate the page at a click so while I can’t entirely avoid the risk of publishing a post with invalid XHTML I can at least fix it rapidly enough that hardly anyone will notice.

It also seems like a useful feature to have aggregators like Venus [3] check for valid HTML and not display posts unless they are valid. It’s not a feature that could be enabled immediately (I’m sure that if you click on this link to the W3C validation service [1] from a Planet feed you will see lots of errors and warnings), but once bloggers have time to fix their installation it would allow preventing some of the common annoyances of Planet installations. It’s not uncommon on popular Planets to have unmatched tags in a post which results in significant amounts of the content being bold, underlined, in italics, or for the greatest annoyance struck-out. I know that this may be a controversial suggestion, but please consider why you are blogging – if you are blogging for the benefit of your readers (which seems to be the case for everyone other than sploggers) then it seems that the readers will benefit more by not having a broken post syndicated than they would benefit from having it syndicated and thus messing up the display of many following posts.

The next thing on my todo list in this regard is to do some tests of accessibility. The work that I have done to pass the XHTML validation tests has helped to some degree – if nothing else the images now all have alt= descriptions, but I expect that it will be a lot of work. The Wordpress Codex has a page about accessibility, I haven’t read all of it yet [4].

Does anyone have any recommendations for free automated systems that check web sites for accessibility? What would be ideal is a service that allows different levels of warnings, so instead of trying to fix all problems at once I could start by quickly fixing the most serious problems on the most popular posts and finish the job at some later date.

Syndicated 2010-02-09 11:13:07 from etbe - Russell Coker

9 Feb 2010 softkid   » (Journeyer)

What is the best bleeding edge

At the moment If you want to run a bleeding edge version of Thunderbird, you need to choose between running 3.2x and 3.1x builds. In fact, it appears that most people willing to run bleeding edge are now running 3.2x builds.

Whilst our most dedicated testers are running 3.2x builds, the engineering team is working on bringing features and bug fixes to the 3.1x branch. There's a discrepancy here. This means that the issues that might exist in 3.1x have a greater chance to be discovered after releases rather than before. It's easy to fix that, instead of running 3.2x builds, we would like our bleeding edge user to use the 3.1x builds (you can find them at http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-comm-1.9.2/). By doing this simple switch you'll help to make the 3.1 series a great series.

Syndicated 2010-02-09 08:01:38 (Updated 2010-02-09 09:12:44) from Ludovic's weblog

9 Feb 2010 robertc   » (Master)

Is a code of silence evil?


Looking at using google apps for my home email, as I want to be able to have my home machines totally turned off from time to time.

Found this interesting gem in the sign up agreement (which I have not yet agreed to :P ):

11. PR. Customer agrees not to issue any public announcement regarding the existence or content of this Agreement without Google’s prior written approval. Google may (i) include Customer’s Brand Features in presentations, marketing materials, and customer lists (which includes, without limitation, customer lists posted on Google’s web sites and screen shots of Customer’s implementation of the Service) and (ii) issue a public announcement regarding the existence or content of this Agreement. Upon Customer’s request, Google will furnish Customer with a sample of such usage or announcement.

This is rather asymmetrical: If I agree to the sign up page, I cannot say ‘I am using google apps’, but google can say ‘Robert is using google apps’. While I can appreciate not wanting to be dissed on if something goes wrong, this is very much not open! A couple of implications: Everyone seeking support for google apps in the apps forums is probably in violation of the sign up agreement; we can assume that anyone having a terrible experience has been squelched under this agreement.

Le sigh.

Syndicated 2010-02-09 08:47:47 from Code happens

9 Feb 2010 mikal   » (Journeyer)

Measuring the popularity of SMTP server implementations on the Internet

I'm interested in measuring the performance of SMTP servers connected to the Internet. Before I can poke around inside a SMTP implementation, I want to ensure that I am using one which lots of people use. To that end I have been running a series of SMTP server surveys for the last several years. This work has been alluded to in the past, but I haven't published any results. This has mainly been because while I have written a number of papers on the topic, I am yet to have one accepted by an academic conference. I've been hesitant to comment about my results because of the requirement for academic publications not be previously published work.

I've decided to change that policy. I'm going to reserve a lot of the deeper analysis for academic publication (if I can make such a thing happen), but I am going to start talking about the work I am doing more in public. To start that off, I should mention what I've been doing...

There have been a number of previous surveys of SMTP servers connected to the Internet, with each survey using a different methodology. So although these results are not directly comparable, a comparison still provides some insight into how the server landscape has changed over the last 12 years. A comparison of published surveys is presented in the table below. Each survey in this table shows the: sample size, which is the number of IP addresses surveyed; sample approach, which is the methodology used to determine which IP addresses to sample and adds bias into the sampling; and the number of responses, which is the number of SMTP servers that responded. The majority of these surveys have relied on random sampling of the IP address space, perhaps with a selection algorithm to limit the results selected. Few of the more recent surveys provide complete information on their probing implementation or the rules they used to identify specific implementations from their observations. It should be noted that non-response from a surveyed IP generally indicates that it is not in fact running a SMTP server accessible from the Internet.

DateSurveyorSample sizeSample methodResponses
27 Nov 1996Bernstein500,000Selective random25,121
214 Aug 1997Bernstein200,000Selective random8,056
211 May 1998Bernstein20,310MX walk17,592
22 Apr 2000Bernstein12,595Selective random10,087
25 Oct 2000Bernstein25,777Random859
227 Sep 2001Bernstein39,206Random937
21 Dec 2002Credentia4,096Random1,837
21 Jan 2003Credentia30,000Random17,540
21 Apr 2003Credentia37,563Random20,410
21 May 2007MailChannels400,000Corporate domain names254,400


Whereas the surveys that I have been running with the assistance of my ever patient PhD supervisor Dr Eric McCreath have been quite a bit larger. Note that larger isn't necessarily better with these sorts of surveys, but my methodology attempts to aim for completeness, and the relative power of PlanetLab makes these computations surprisingly cheap. Details of my surveys so far:

DateSurveyorSample sizeSample methodResponses
January 2008Still / McCreath46,136,113Exhaustive1,973,748
April 2008Still / McCreath92,286,998Exhaustive1,609,111
July 2008Still / McCreath97,545,668Exhaustive1,579,507
October 2008Still / McCreath109,661,889Exhaustive1,801,081
January 2009Still / McCreath110,397,428Exhaustive1,916,719
April 2009Still / McCreath110,706,130Exhaustive1,925,760
October 2009Still / McCreath111,209,212Exhaustive1,800,573


Our survey is implemented by attempting to identify the MTA software running on an SMTP server using the SMTP connection banner. In other words, a collection of IP addresses are connected to on the SMTP port (TCP 25), and an attempt is made from the early stages of the SMTP protocol interaction to determine what SMTP server software is running on that host. The SMTP protocol will often reply to the connection with a status 220 line, referred to as the SMTP banner, this tells the connecting client that the server is ready. The SMTP banner also frequently states what software the server is running. Even if the software in use isn't explicitly named, it is often a string which is unique to a given SMTP implementation. This technique simply connects on the SMTP port, and logs any lines starting with 220. The connection is then closed, with no attempt to transfer an email occurring.

So what results have I found so far? I'm trying to keep these blog posts to less than 1,000 words each, so that's too big a question to answer here. I've found some quite unexpected things along the way, such as an accurate technique for measuring the occurrence of domain parking on the Internet, and I'll discuss those in future posts. Instead, let me leave you with this short graphical summary of the results so far:



This is the history of the currently five most popular implementations over time. You can see that Sendmail has fallen from a position of market dominance, and Exim is currently the most popular SMTP server implementation.

I have a lot more to say about all this work, but as I mentioned earlier I want to keep the length of these posts down. I'll say more in future posts.

Tags for this post: research(S)
Related posts: Initial SMTP survey poster results in a pie chart; Interesting paper: "YouTube Traffic Characterization: A View From the Edge"; RemoteWorker v74; Microsoft Exchange the most popular SMTP server on the Internet?; I think I've worked out the problem with the hotel network; Mikal, the massive domain squatter; Internet traffic; Domain name lookup helper for python?; Normalising mail server package names; Satellite internet at Walmart; Long time not much write; RemoteWorker v70; Announcing early results of my survey of SMTP servers; Noticed that smtpsurvey.stillhq.com is down?; Mikal, tell something I didn't know about SMTP servers on the Internet; The witty worm with Vern Paxson; Why does every man and his dog put man pages online?; Sensis Australian search

Comment

Syndicated 2010-02-08 21:29:00 from stillhq.com : Mikal, a geek from Canberra living in Silicon Valley (no blather posts)

9 Feb 2010 fozbaca   » (Apprentice)

links for 2010-02-08

Syndicated 2010-02-09 03:00:47 from fozbaca.org

9 Feb 2010 Hobart   » (Journeyer)

I lived in Boulder, yes, how'd you guess?

<small>Posted via LiveJournal.app.</small>

Syndicated 2010-02-09 02:07:24 from jon's blog

9 Feb 2010 marnanel   » (Journeyer)

CSS window border themes

View Poll: CSS window border themes


No, it can't be "dialogue": CSS is US English.

Syndicated 2010-02-09 01:20:28 from Monument

9 Feb 2010 sness   » (Journeyer)

algo




Favorite algorithms

"Debugging it was awful though, as one wrong move and you end up with an angry ball of polygonal hate."



Syndicated 2010-02-09 17:23:00 from sness

9 Feb 2010 chbm   » (Journeyer)

Week In Review 5W2010

<content type="application/xhtml+xml" xml:base="http://chbm.net" xml:lang="en" xml:space="preserve" mode="xml">
</content>

Syndicated 2010-02-08 23:34:00 from c h b m . n e t

8 Feb 2010 bagder   » (Master)

My Fosdem 2010

<h2>Friday</h2>

Björn and I left work on the Friday afternoon and took a flight down to Brussels, Belgium. After having checked in to our hotel, we met up with Frank from the Rockbox project and we headed to the Fosdem beer event that took place on a pub quite nearby to the hotel.

The Beer event was crowded. I mean really really crowded. But we still managed to get seated and we got fine belgium beers and we had a good time. We met a few other Swedes that turned out to be the first in a long series of Swedes that were there. Petur from Rockbox joined up there as well and together we went over a fair share of their beer selection…Atomium <h2>Saturday</h2>

For us tech guys, the Saturday morning had no really exciting subjects and weirdly enough the morning had only one track and the massive amount of parallel tracks didn’t start until after lunch. This gave us an opportunity to go sight-seeing, and we visited the city square and the Atomium before we headed into the FOSDEM premises and squeezed our way in to a presentation.

Peter Stuge from the Coreboot project explained to us that we were by far too many people crammed into that little room so if one of the responsible guys would come around a fair lot of us would get thrown out of there. With that heads up given, he started his talk and gave us insights in what coreboot is, what it does and so on. I’ve heard Peter talk about this topic before, but he’s still a good talker and the topic still is techy and interesting enough to listen to.

Embedded software development best practices by Adrien Ampelas turned out to be a bit boring. Basically we got the feeling that Adrien re-used a company slide show or something and told the audience a lot of things I bet the majority of people already knew. Yes we know we must use version control. Yes we know we should send patches upstream. No we don’t Fosdem Entryagree with you that there never exist any reason not to use git.

Sascha Hauer from the Barebox project (the project that was previously known as U-Boot v2) told us about this U-Boot project and what they’re trying to accomplish. It seems like an interesting approach to fix some of the worst mistakes of U-Boot but still leverage on all the things U-Boot did right. It’ll be fun to see if it gets adoption from board makers and companies in general. I guess there’s a lot of investment in U-Boot so lots of things will probably stick with that for a long time ahead…

Flash enable BIOS reverse engineering by Luc Verhaegen gave us an insight in the x86 based reverse engineering they do in the Coreboot project to figure out how to enable flashes and to make them possible to write to when you want to upgrade them to use Coreboot. It was only a quick run-through, but my general feeling was still that compared to Rockbox-style reverse engineering, their tasks actually seem a lot easier! Still interesting, as Luc is a good speaker. <h2>Sunday</h2>

Sunday morning started earlier than yesterday. Interesting talks started right away, and we actually were too slow at breakfast so we missed the first part of the interesting Introduction to RTEMS talk by Thomas Doerfler. RTEMS is a fully open source RTOS that’s been around for ages and that has some very good realtime skills and can get shrunk to a rather small size. A slight downside with it is its slightly odd license, as it is a GPLv2+ license with a rather big exception that is made to allow proprietary applications link with it. It makes it incompatible with regular GPLv2 code.

The RepRap project was presented by Adrian Bowyer and I must admit that these 3D-printers are mighty cool and even more fun to see and witness in the real world than they are to see on tiny pictures on web sites.

Back in the embedded room, Roberto Jacinto told us about apt-get for android – with GUI which pretty much described the Aptoide project. It has nothing in common with apt: it doesn’t do dependencies and it doesn’t use its file formats. It has some pretty significant bugs still, and it generally seemed like a rather immature project that I’m not even sure I agree are on the right track. I’d rather actually see the real apt-get for android, with out without GUI.

The Cross build systems: Present & Future workshop could’ve become interesting. A lot of projects (PTXdist, Buildroot, Crosstool-NG, Openembedded, Emdebian etc) spoke about what they are, what they hope to do and how they’d like to collaborate. Unfortunately it took a bit too long time so by the time all had presented their projects the time was pretty much up. The most controversial and slightly off-topic of them all was Andy Green (formerly involved in Openmoko) who talked about how we all should stop cross-compiling and build directly on the target instead(!) and how booting Linux shouldn’t need a boot-loader and that designing PCBs with NAND is stupid(! again). I didn’t hear anyone agreeing with his ideas.

Next up was my talk on Rockbox. I did it in about 40 minutes and I think I covered a bit of what Rockbox is and how we work when we work with new potential targets. It later struck that I should perhaps have had a slide about what the future holds etc, but hey I think it went pretty smooth anyway! Peter recorded my talk on his n900 so hopefully it’ll soon be available online somewhere. After my talk we met a lot of guys wanting to talk Rockbox, ask about particular players and so on and it was mighty fun and interesting.

Greg Kroah-Hartman did the final talk and he is a very good and engaging speaker that really can catch the big audience in Fosdem’s biggest room. Write and Submit your first Linux kernel patch is his “standard talk” but he’s doing it so good and with such elegance that it is a pleasure to watch and learn from. And I’ll admit I wasn’t aware of the get_maintainers.pl script in the kernel tree. A very useful little thing! <h2>Reflections</h2>

Some conclusions and general thoughts about the event:

Lack of gaps – there’s a problem when all talks in all rooms are made gapless. It makes people get up and leave 5-10 minutes before the end of each talk so that they will get in time to the next talk that will start on the full hour in another room. It causes pretty much all question-sessions towards the end to fail since the questions (and answers) can’t be heard.

Hard to find people – it is such a huge event and lots of people I have no idea what they look like, so trying to meet friends and people I’ve only emailed with or chatted with on IRC is very hard. Name tags would be really cool. I did have some benefitsHaxx from using my shirt with a big Haxx logo on the back since a fair amount of people recognized it and approached me!

Audio systems – the quality of the different rooms varied a lot (not only sound-wise but the sound was what bothered me). Unfortunately for me, the embedded room was one of the worst ones when it came to audio. It was a big room sure, but the biggest room had an excellent audio system and thus proved size is not what matters. In this case, I think a lot was to blame on the actual microphone we had there.

Phone apps – having phone apps with the entire schedule and a little map for each room etc was a great service. The app also reminded us when a talk you had marked as “favorite” was about to start. It was a bit strange though how the android and n900 versions of the app differed. The n900 version was buggy and slow, but it did offer the schedule in a time-based view while the android version only allowed us to view the schedule based on rooms.

Next year – yes. I think it was great fun and I will really try to attend next year again. Hopefully other friends will too, since meeting friends at the place really doubles the fun! Thank you all for a nice event!

Syndicated 2010-02-08 23:01:24 from daniel.haxx.se

8 Feb 2010 ralsina   » (Master)

Marave 0.3 is out!

Version 0.3 of Marave, a distraction-free fullscreen editor is out at http://marave.googlecode.com

This version includes several bugs fixed and features implemented since 0.2:

  • New 'Styles' support, you can change the look of Marave with CSS syntax
  • Debugged themes support, a few themes included
  • Fixed bug saving text color
  • Fixed font changing bug
  • Use the document name in window title
  • "Now playing" notification

Marave is free softare released under the GPL, and should work in all major desktop platforms.

I would love feedback on this release, as well as ideas for Marave's future, so if you want to help, please join the mailing list:

http://groups.google.com/group/marave-discuss

Of course, if you like Marave, feel free to give me money


Syndicated 2010-02-08 21:17:12 from Lateral Opinion

8 Feb 2010 mostafadaneshvar   » (Apprentice)

localized spins

Today I want to add some points to ongoing discussion about localized spins in Fedora Project. As a member of translation team, I think there is no need to create such spins. In translation of any packages there are two surfaces that every localizers can work on. The first one is the translation of GUI. As my experience shows that great majority of users prefer to use the source language rather than their localized GUI. This case is very special for RTL languages, like Persian and Balochi. Because the users   based on their previous experience with LTR interface are not familiar with this new interface. He can spend sometimes on RTL environment for example here, then when he surfs in internet, he uses LTR web pages. So the translated UI is not as favorable as we expect. For example, I myself get confused as I use a translated UI, because I don’t get accustomed a RTL interface at all.
The next item is translation of documentations. This is the point we should emphasis in localizing. A lot of Fedora users are not English native speakers or have difficulties in understanding English. As they get into trouble in using Fedora they are unable to find any local help, I mean in their computers. Localized docs can be useful here.
Shall we have a localized spin? frankly I should say that NO. As I check in Transifex in Fedora I find that we have 89 languages. If the Board authorized creation of a language-based spin for one language or two, the other languages that don’t have any spin up to that time may request a new spin. One day we have a lot of spins. I think it’s better to have a plan in development process to have some sort of modifications in coding for example in Anaconda. As a user select his region, for example, Anaconda  suggest usage of localized version. developers can think about installing of selective packages based on users locale, for example.

Any thought?Idea?

Syndicated 2010-02-08 18:27:58 from مصطفی دانشور (Mostafa Daneshvar) » linux

8 Feb 2010 redi   » (Master)

badvogato, disjoint sets in the trust graph are interesting, but what cert level would the new seeds have? If they do not lose their certification then the spammer can still post their crappy SEO links with advogato's google-juice.

I prefer to say goodbye with the gesture "here's the source code, run your own mod_virgule but don't spam this one"

8 Feb 2010 federico   » (Master)

Mon 2010/Feb/08

  • Luciana was munching on sausage slices. She grabbed the curved end of one sausage, looked carefully at it, and exclaimed, "look, a little vault!".

    I guess that's what she learns in this house.

Syndicated 2010-02-08 11:52:00 from Federico Mena-Quintero - Activity Log

8 Feb 2010 badvogato   » (Master)

It just occurred to me if virgule trust matrix can implement self-split, in this concrete case, once nodes inside one community decided to kick the spammer out with its supporter but instead of killing them the mother matrix should split and make a copy of trust-matrix, reverse all its certs so that the spammer & hjclub become the Other independent Universe on their own with two offending nodes as root seeds. That's almost like to say goodbye with a good gesture: 'May Force be with you Amen'.

8 Feb 2010 redi   » (Master)

badvogato, are you suggesting we should convince the spammers to stop their pathetic attempts at SEO and to become Free Software developers instead?

That would certainly be consistent with the goals of advogato, but I'm not sure it's practical :)

I didn't say the system was ideal, only that it worked. I consider it unfortunate that hjclub's account was deleted. I did try to convince hjclub that certifying the spammer was a bad idea. I don't know if hjclub read that and I don't know whether the spammer's cert was dropped by hjclub, or as a result of hjclub's cert being dropped.

8 Feb 2010 badvogato   » (Master)

redi, i beg to differ. The ideal situation for 'Self-Policing' is for others to exert enough pressure from outside to the inside and 'unknown' states of 'self' so that it will conform to the outside norm by eliminating extremities of itself and so become the other. It is not for the other to eliminate the 'alien' self from outside. Also interestingly, there maybe evidence to suggest that nerve reflex can be faster than nerve initiator, which means preemptive strike will do no good other than self-destruction.

8 Feb 2010 redi   » (Master)

recentspam, episode 94

Since Zaitcev's posting atai lost his Master cert, as a result of a new Apprentice cert. atai responded by removing hjclub's Journeyer certification. I didn't see which order the events happened in, but the spammer and hjclub have both been deleted, probably marked as spam and deleted by other users as soon as they lost their certs.

That might be unfair to hjclub, but the community's response to spammers, and anyone who helps them, is pretty clear. Simple as it is, the self-policing system worked to defeat spammers again. 8191 is right that it would struggle if the spammers weren't utterly incompetent, but so far that hasn't been a problem. Maybe the intersection of SEO spammers and people with half a brain is an empty set?

8 Feb 2010 mikal   » (Journeyer)

Home power measurement

I've been spending some quality time with a Current Cost CC128 and my existing home sensor network. So far I've discovered that I use quite a bit of power, and that I can remotely monitor how many times a day my wife makes a cup of tea. Some example data:



You can see that it was relatively cool compared with days a few weeks ago today. That's more obvious in the graph showing the last two weeks though:



However, it was quite humid today:



Which is why we didn't have the evaporative cooler on, just the fan. That doesn't seem to really affect our power usage, which really needs more analysis:



The 500 watt minimum power draw makes me unhappy. You can see over a week it never goes away:



Tags for this post: blog(S)
Related posts: Extreme Machines: Eirik Raude; More on burial methods; It seems to me; The environmental friendliness of shrimp trawling?

Comment

Syndicated 2010-02-07 23:55:00 from stillhq.com : Mikal, a geek from Canberra living in Silicon Valley (no blather posts)

8 Feb 2010 Hobart   » (Journeyer)

Grumble

Spent ten minutes fiddling with my PS3 and TV settings wondering why I couldn't hear the movie, then noticed the PS3 thinks its outputting '2 channel audio', so Netflix botched the audio for HD Nightmare on Elm Street. :-b

Syndicated 2010-02-08 02:34:38 from jon's blog

8 Feb 2010 fozbaca   » (Apprentice)

links for 2010-02-07

Syndicated 2010-02-08 03:01:08 from fozbaca.org

8 Feb 2010 mcr   » (Journeyer)

Thing I saw at Active Surplus

I was in Toronto at the AGM for http://www.EspressoCode.com/. I had to stop at Active Surplus for switches and what-not for my model railroad. I certainly the a "what-not"

Check it out:

[[http://www.sandelman.ca/mcr/humour/2010-02-04-10-46-whatnot.jpg][Some kind of Pumpy Thing]]

Syndicated 2010-02-07 20:02:00 (Updated 2010-02-08 02:11:25) from Michael's musings

8 Feb 2010 etbe   » (Master)

Security and Hiring

The main sources of information used when hiring someone are their CV, the interview, and references. <h3>CV</h3>

The CV is written by the applicant or sometimes for the applicant. Naturally it says only good things, if a CV notes no skill in a particular area then it may be used to exclude an employee from consideration. But the trend is towards including a reference to anything that you touch, so someone who lists DBA experience may merely have done a couple of CREATE TABLE operations. <h3>Interview</h3>

The interview is a good test of people skills but is often of little value in assessing technical skills. The interviewer asks questions such as “do you know technology X” and the applicant says “I know that really well“. If the company is hiring another person with similar skills to current employees then they can have their current employees sit in on the interview and ask difficult technical questions, but for unknown reasons managers often don’t take that option and get no advice from their technical people. Also if the company is hiring someone with specialised skills (EG they are about to implement a new application and want to hire their first employee to work on it) then it may be impossible for them to assess the technical merit of answers. Probably the best use of the interview is to match answers with the CV, if the applicant doesn’t appear to know the contents of their own CV then they should be rejected.

The biggest problem with interviews is when the questions are all of the form “do you know X“. Someone who really knows it will say “yes” as will someone who doesn’t know enough to realise the limits of their knowledge – and such ignorant people vastly outnumber the skillful people. The real problem is that the people who are moderately skillful will lose out. If someone asks me about my MySQL skills I will tell them that I’m not really good at it. Sure I’ve run replicated servers with tens of thousands of users running 24*7, but that doesn’t mean I’m really good at it – probably most people who will claim to be great at MySQL without qualification would have less experience than me. <h3>References</h3>

Reference checks rely on an unknown person saying good things about the applicant. For starters there is the issue of the number of references which may not be representative of their employment history – EG the applicant could use as a reference the one manager who didn’t sack them.

The next issue is that there is little incentive for the referee to be honest, most people are aware of instances where someone once worked for a friend and can rely on good references for the rest of their career. If a reference is inaccurate then there is no realistic opportunity for redress.

Finally every reference check that I am aware of (checks where I have been the referee or the applicant) has involved the applicant giving the phone number of the referee to the hiring manager! The phone could be owned by a friend or relative of the applicant, so logically a good reference that is based on trusting the applicant to supply the phone number only proves that the applicant is either good or really bad. To make a reference check prove something the recruiter would at a minimum have to phone the number listed in the white-pages for the corporation that used to employ the applicant, asks to speak to the manager of the relevant department, and then gets a reference. Calling a mobile phone number that is supplied by the applicant (which seems to be the standard practice) is essentially trusting the applicant – and trust is the root cause of most security problems!

Really most of this ends up as trusting the applicant to provide honest evidence that they are trustworthy and believing that the applicant’s technical knowledge is good enough to be correct when they say that their technical knowledge is good. It can fail spectacularly when someone isn’t trustworthy enough to provide honest evidence of their integrity or when someone doesn’t have the skills needed to know that their skills are lacking.

As an aside, even if the reference is given accurately and in good faith it may still be misinterpreted. The fact that telephone references are exclusively relied on exacerbates this problem. Ideally references would be in writing with some way of proving their authenticity (maybe using phone verification of the accuracy of the written document). <h3>Solutions</h3>

So how can we solve this? Some people believe that career based social networking software will solve the problems, but as usual I think that software doesn’t magically solve human problems. The first challenge when trying to use social networking to solve the problem is to find someone on your friends list who has relevant knowledge, this may be viable in a small industry (EG when someone from bank A applies for work as bank B in the same city). The next issue is that of false “friends“. I’m sure that I’m not the only person who has been pressured to add people as friends on social networking sites, the non-computer social interactions really don’t prepare people for saying “no you are not my friend” (apart from high-school I guess). With professional social networking sites there are further issues, if you are working on a client site and a manager demands that they be listed as one of your friends then what are you going to do?

So it seems to me that the social networking sites are at best a helper for the gossip network. If you think that a friend of a friend from a social networking site might be able to help you then you first ask your friend if the person in question is really a friend, and if so are they one of the shifty pseudo-friends you only hang out with because their company pays good money. But the problem with the gossip network is that it’s mostly secret and is therefore subject to settling vendettas, I’ve heard of senior managers going out of their way to spread false stories about former employees to settle scores.

The best solution I can think of is for someone who has a reputation to publicly stake it on the accuracy of their references. If I’m going to give a reference then I would be happy to do so via a GPG signed email or a blog post. This doesn’t mean that my references will always be correct, but it would show that I try to give good references.

Syndicated 2010-02-08 00:08:51 from etbe - Russell Coker

7 Feb 2010 ralsina   » (Master)

Marave 0.2 is out!

Version 0.2 of Marave, a distraction-free fullscreen editor is out.

This version includes several bugs fixed and features implemented since 0.1.1:

  • A corrupted Right-click menu (Issue 20)
  • Flickering on background changes
  • More detailed licensing information
  • More tested on Windows
  • Added help (F1)
  • Search & Replace (but replace all is not done)
  • New artwork
  • Status notifications
  • Document Info (Ctrl+I)
  • Better feedback in the UI elements (specially the buttons)
  • Save font size correctly
  • Fix "Starts in the background" problem (Issue 17)

Marave is free softare released under the GPL, and should work in all major desktop platforms.

I would love feedback on this release, as well as ideas for Marave's future, so a mailing list for Marave has been opened:

http://groups.google.com/group/marave-discuss

Of course, if you like Marave, feel free to give me money


Syndicated 2010-02-07 19:49:07 from Lateral Opinion

7 Feb 2010 marnanel   » (Journeyer)

The record shows I took the blows

There is an idea in literature called the motif of harmful sensation. For example, if you look at Medusa, you will turn to stone from the very experience. If you read a book called The King In Yellow, you will go mad simply from the sensation of reading it. If you hear the joke "Wenn ist das Nunstruck git und Slotermeyer? Ja! Beiherhund das Oder die Flipperwaldt gersput!" you will die, as long as you understand German.

Well, apparently the Sinatra song My Way should be added to the list. Being around anyone singing this song will apparently get you killed.

Syndicated 2010-02-07 18:35:35 (Updated 2010-02-07 18:37:06) from Monument

7 Feb 2010 louie   » (Master)

Telling numbers

I’m currently reading a book on modern legal drafting (read: ‘plain english for dummies, I mean, lawyers’). It is very good so far, but I think this is a telling stat about lawyers: 127 pages are devoted to why clear, modern english is a good idea. That is 22 pages more than are devoted to how to write clear, modern english.

Modern Legal Drafting

Modern Legal Drafting, by Peter Butt and Richard Castle

This imbalance isn’t as insane as it sounds at first; there are some not-crazy reasons to re-use old language in legal documents, and explaining why they aren’t actually correct is a useful service. Still… given that some of the complaints about legalese cited by the book are over 200 years old, you would think the profession might at least by now realize that much legalese is a bad idea, even if we haven’t yet learned how to get rid of it…

(Favorite sentence from the book: “My client has discussed your proposal to fill the ditch with his partners.“)

Syndicated 2010-02-05 02:11:49 from Luis Villa's Internet Home » Blog Posts

7 Feb 2010 argp   » (Journeyer)

exploit for CVE-2010-0453

While playing today with kmdb on OpenSolaris I wrote a denial of service (kernel panic) PoC exploit for the UCODE_GET_VERSION ioctl NULL pointer dereference vulnerability. The vulnerability was discovered by Tobias Klein who always publishes very detailed advisories:

http://www.trapkit.de/advisories/TKADV2010-001.txt

You can get my exploit from:

http://census-labs.com/media/cve-2010-0453.c

7 Feb 2010 ruoso   » (Journeyer)

Writing games in Perl - Part 2 - Controlling the Ball

Following the first post on the subject of writing games in Perl, where we created a bouncing ball (I know, it is a rectangle, but I trust your imagination), this post is going to add something very important when dealing with games: input.

Silveira Neto suggested that I should include more specific instructions on how to start the game (and maybe a video), so I recalled that I didn't mention that all the sources for this posts (including the text) is currently hosted at a github repository (if you plan to contribute, please just ask me for commit permissions instead of forking the repo).

So if you want to run the codes posted here, you first need to:

$ git clone http://github.com/ruoso/games-perl.git

You can check for updates by calling

$ git pull origin master

from inside the games-perl directory. Each directory inside games-perl starts with the number of the post. The first post is inside the 1-bouncing-ball directory and the second is in 2-controlling. To run the the first code just get inside the first directory and call:

$ perl ball.pl

The second example code is based on the first, so the script name is the same, so just get into the other directory and run the same line. If you get an error like:

Can't locate SDL/Video.pm in @INC (@INC contains: /etc/perl
/usr/local/lib/perl/5.10.0 /usr/local/share/perl/5.10.0 /usr/lib/perl5
/usr/share/perl5 /usr/lib/perl/5.10 /usr/share/perl/5.10
/usr/local/lib/site_perl .) at ball.pl line 8.

It means you probably don't have the newest SDL, take a look at the first post to see how to get the newest redesigned SDL. <h4>Controlling the Ball</h4>

Enough for the introduction, let's get to the actual code. The first thing we need is understanding SDL Events. If you ever programmed GUI applications or even if you wrote some javascript you are aware of how an event framework looks like. SDL is no exception, you need to wait (or poll) for the events, and each event will contain the information you need to figure out what happened.

In our case, we want to apply additional acceleration to the ball whenever the arrow keys are pressed. But if we have an event-based system, the way to figure out which of those four keys is currently pressed is keeping a state mask and update it when you receive keydown and keyup events.

So what we're going to do is to manipulate the acc_h and acc_v ball attributes depending on the keydown and keyup events. It might look complicated, but the only change we need is (this is inside ball.pl main loop):

  while (SDL::Events::poll_event($event)) {
    if ($event->type == SDL_QUIT) {
      exit;

    } elsif ($type == SDL_KEYDOWN &&
             $sevent->key_sym() == SDLK_LEFT) {
      $ball->acc_h(-1);

    } elsif ($type == SDL_KEYUP &&
             $sevent->key_sym() == SDLK_LEFT) {
      $ball->acc_h(0);

    } elsif ($type == SDL_KEYDOWN &&
             $sevent->key_sym() == SDLK_RIGHT) {
      $ball->acc_h(1);

    } elsif ($type == SDL_KEYUP &&
             $sevent->key_sym() == SDLK_RIGHT) {
      $ball->acc_h(0);

    } elsif ($type == SDL_KEYDOWN &&
             $sevent->key_sym() == SDLK_UP) {
      $ball->acc_v(1);

    } elsif ($type == SDL_KEYUP &&
             $sevent->key_sym() == SDLK_UP) {
      $ball->acc_v(0);

    } elsif ($type == SDL_KEYDOWN &&
             $sevent->key_sym() == SDLK_DOWN) {
      $ball->acc_v(-1);

    } elsif ($type == SDL_KEYUP &&
             $sevent->key_sym() == SDLK_DOWN) {
      $ball->acc_v(0);

    }
  }

So, this is it. Follows a small video of the game.

<object height="344" width="425"><param name="movie" value="http://www.youtube.com/v/On8O7jJP_WI&hl=pt_BR&fs=1&" /><param name="allowFullScreen" value="true" /><param name="allowscriptaccess" value="always" /><embed allowfullscreen="true" height="344" src="http://www.youtube.com/v/On8O7jJP_WI&hl=pt_BR&fs=1&" type="application/x-shockwave-flash" width="425" /></object>

Syndicated 2010-02-07 10:30:30 from Daniel Ruoso

7 Feb 2010 mikal   » (Journeyer)

Dogs of War




ISBN: 0446610895
Aspect (2002), Mass Market Paperback, 320 pages
LibraryThing
<script type="text/javascript">GBS_insertPreviewButtonPopup('ISBN:0446610895');</script>
Another combat anthology, this time edited by David Drake.
  • Or Battle's Sound (Harry Harrison): already read in Battlefields Beyond Tomorrow and Body Armor: 2000.
  • Liberty Port (David Drake): already read in The Complete Hammer's Slammer's Volume 1.
  • Straw (Gene Wolfe): steam punk mercenaries in a future which is more like the past. A good read.
  • Tomb Tapper (James Blish): this one has a nice twist at the end, although I found the underlying premise of the story hard to believe.
  • A Relic of War (Keith Laumer): already read in The Compleat Bolo.
  • Basic Training (Mark L Van Name): this one is a bit sad. Well, all of the stories in this book are sad, but I find this one a bit harder than others because the main character is about the age of my kids.
  • Witch War (Richard Matheson): an interesting little tale which doesn't end up at all like you expect.
  • Transstar (Raymond Banks): I didn't think this story was all that good when I was reading it the other night, but its the one I kept thinking about for the next day. There are lots of interesting issues raised here -- arrogance, response, the cost / benefit of aggression.
  • Time Piece (Joe Haldeman): this story is very much like a small summary of The Forever War, which makes it disappointing in a way... The book is better, and this story glosses over many of the issues.
  • Clash by Night (Henry Kuttner and C.L. Moore): this story is interesting because it prompted a sub-genre all of its own about mercenaries in the future. However, this story is entertaining by not earth shatteringly great.


Tags for this post: book(S) Anthology(S)
Related posts: Battlefields Beyond Tomorrow; Body Armor: 2000; Bolos 1: Honor of the Regiment; The Complete Hammer's Slammers Volume 1; Bolos 4: Last Stand; The Complete Hammer's Slammers Volume 2; The Complete Hammer's Slammers Volume 3; Bolos 2: The Unconquerable; Bolo!; Bolos 3: The Triumphant; The Compleat Bolo; Bolos 5: Old Guard; Bill the Galactic Hero Series; Galactic Dreams; Bolo Strike; Bill the Galactic Hero on the Planet of Bottled Brains; Forever Peace; The Stars Must Wait; Bill The Galactic Hero; Bolo Brigade; Bolos 6: Cold Steel


Comment

Syndicated 2010-02-07 02:25:00 from stillhq.com : Mikal, a geek from Canberra living in Silicon Valley (no blather posts)

7 Feb 2010 etbe   » (Master)

Michael Atkinson Lies

The South Australian government wanted to force bloggers to disclose their name and post-code when commenting on an election [1]. According to Adelaide Now this included posts on Twitter and Facebook [2].

As expected there was a strong public reaction to this and Michael Atkinson (state Attorney General) stated that the law was not going to be enforced and that it would be retrospectively repealed after the election [3]. Which might have been the end of the issue, but he also said “All MPs and all parties voted for Electoral law. Hope Libs, Greens, Family First, Independents etc will join us to support repeal” which seems to be a clear claim that the Greens supported that legislation.

The Greens are pretty good about freedom of speech issues so I immediately enquired as to what was going on. The following is from a Greens media brief issued yesterday by the office of Mark Parnell MLC (the Greens MLC in SA) which was issued before the new law was retracted:

Attorney General Michael Atkinson must urgently clarify the scope of new electoral laws covering public comment on the internet, in the wake of concerns that the impact will be much wider than expected.

“With only weeks to go before the election kicks off, there is an urgent need for the Attorney General to explain exactly what his laws are
intended to capture,” said Greens MLC Mark Parnell.

“When this was debated in Parliament last year, we were told that the law change would only affect ‘electronic versions of a journal’. Now there is
concern that the laws could extend further, restricting public commentary on media sites like AdelaideNow and ABC Online,” he said.

When the Bill was debated in the Lower House, Michael Atkinson originally wanted to include ALL material on the web. However, the Government
backed down on this by the time it got to the Upper House, with Minister Holloway saying:
“The intention is to limit the coverage of section 116 as it applies to the internet to electronic versions of a journal rather than any electronic publication on the internet.”

A ‘journal’ was narrowly defined as ‘a newspaper, magazine or other periodical’.

Now I don’t have a great objection to a law that demands that journalists identify themselves when commenting on an election, and I think that most people would not care about that. It seems that Michael Atkinson is repeatedly changing his claims to try and match popular sentiment.

He has got form for this sort of thing, his past “achievements” include censoring the censorship debate about an R18+ rating for computer games [4].

Syndicated 2010-02-03 02:28:22 from etbe - Russell Coker

7 Feb 2010 sye   » (Journeyer)

atai, hjclub is gone but not forgotten. it is a lesson we can all learn about Google's 10 things they know are true. I know the _very_ opposite of what Google knows or doesn't know can also be true. thanks.

7 Feb 2010 slef   » (Master)

The Phone Co-op 2010

Ed Mayo talking to the Phone Co-op meeting
Ed Mayo talking to the Phone Co-op meeting

Yesterday I had the pleasure to be at the Phone Co-ops annual meeting, as those of you who follow me on identi.ca or elsewhere will already know. The proceedings included a speech from Cooperatives-UK Chief Exec Ed Mayo that included the co-op pub in Salford, spurning David Beckham’s gift and this summer’s Co-operatives Fortnight.

As ever, the bits I liked best were the lively question-and-answer sessions, both on Ed’s speech and the annual report. I took notes and reproduce them below, but I didn’t capture every question, or every answer and none of this is verbatim. Questions to Ed about UK cooperation included:

  • Can people in the co-op pub smoke? – No, there’s a bit of drinking in meetings, but not smoking. Some heated debates in meetings are defused by a break to go refill the glasses.
  • Gordon Brown and Labour are starting to talk about cooperatives more. What should we do? – We need to stand on our own two feet, but expect to be treated equally. A “Mutuals Manifesto” will be out online next month. In regulation, having the telecoms networks opened up to co-ops and not be so limited would help. One interesting possibility is with football co-ops / supporters trusts: it’s been suggested that a fan co-op should have the right to buy a club if supported by 75% of season ticket holders.
  • With all the business collapses and Cadburys being bought, what is the temperature of interest in mutuals outside the movement? – Good. The only AAA-rated bank in the world is the Rabobank co-op, but remember that co-ops are not immune to failure. Cadburys is not a co-op and was always vulnerable to the current investor-led rules changing its ownership. People are not necessarily thinking of co-ops, so we need to reach out. It’s a cynical, apocalyptic age, so we need to prove what we’re doing. coops14 is an opportunity. The co-op bank saw a surge in new accounts after it was highlighted in one newspaper as an alternative to excessive banker bonuses. We need to encourage people to act. Reminded of a quote that “the trouble with the rat race is that even if you win, you’re still a rat”. Research shows that Britain today is a brutal and competitive place to grow up in, with only 45% of children trusting their friends. People have heard of cooperatives. We should try to build a more cooperative nation.
  • Comment from the floor that the private sector won’t take substantial cooperative competition lightly. We must look behind us and never give them a chance.
  • Could the BBC become a co-op? – Yes.

Questions about the annual report asked about:

  • What is our target profit %? – No hard target. This year’s reduction was planned and it is still a profit.
  • Why are we using UKMail instead of the Royal Mail? – £10k price difference. It was a hard choice, debated by the board, but there is no co-op option and no particular preference for state ownership.
  • Why are we considering energy supply? – Telecoms is a deflationary market, so diversification is a wise move and energy supply is a top customer request.
  • What is our churn rate? – 1.5%/year, which is unsatisfactory to us, but still low by industry standards.
  • Was the Avoco acquisition price too high? – a good price by usual industry measurement (multiple of sales)
  • Will we support coops14 Free and Open Source Software events, because there is an overlap? – (I didn’t catch the answer)
  • Can we put leaflets in the shops? – Matt Lane is working on this and other opportunities
  • Can we encrypt bills and emails if customers want it? – exec will take and deliberate on that
  • Can we get some sales materials to Scotland? – details taken to send them out next week

Syndicated 2010-02-07 07:06:49 from Software Cooperative News

7 Feb 2010 kr   » (Journeyer)

Don’t Copy the Call Stack

Some runtimes claim to provide first-class continuations, but implement this by copying the entire call stack. This implementation strategy makes continuations totally unusable in production code, and it should be outlawed. Or maybe such runtimes should be required to call them “shitty continuations” instead of just “continuations”.

Syndicated 2010-02-06 08:00:00 from Keith Rarick

7 Feb 2010 hacker   » (Master)

Ruminations on Baggage

The more and more I expose myself to people from all walks of life, the more I realize… almost everyone tries to offload their baggage onto anyone who will listen. Look, EVERYONE has baggage. Those who say they don’t, are lying. It’s not that you have baggage, or don’t have baggage, or have less or [...] Related posts:
  1. The Logic of Airport Madness: Part 1
  2. The Search for a Better Backpack
  3. Rooting the Dyson Root 6 “Portable” Vacuum Cleaner

Syndicated 2010-02-07 02:13:35 from random neuron misfires

7 Feb 2010 Hobart   » (Journeyer)

Hmmm. I wonder how deep irssi scripts have access to stuff ... Should be possible to learn the magic JavaScript to drive iPhones in "acts just like an app" mode ... Then have an irssi script listen on an https port and let you get to your screen-detached session and get your chat on ...

<small>Posted via LiveJournal.app.</small>

Syndicated 2010-02-07 00:58:37 from jon's blog

6 Feb 2010 atai   » (Journeyer)

Re: hjclub

I have downgraded hjclub's certification from me to observer. Not sure if this is enough to resolve the spam problem?

7 Feb 2010 Hobart   » (Journeyer)

Bioshock themed concert flyer

<small>Posted via LiveJournal.app.</small>

Syndicated 2010-02-07 00:50:55 from jon's blog

6 Feb 2010 Stevey   » (Master)

You seem uncomfortable.

I've been trying to remember to post the pictures I like online for the past few months. So this is a reminder to myself.

This image below didn't turn out quite how I wanted it to:

  • I was hoping for a nicer sihouet upon the lady's face.
  • The tree-branch on the left irritates me.

But that said I keep on coming back to look at it. I like the lighting, and I love the way that the brick wall on the right hand side angles towards the building on the horizon.

Enjoy. Or not. Sunset

A similarly "not perfect" image is this outdoor shot. I have only one irritation with this shot - and that is that the trees are clipped at the top. Meh, such is life.

(I have two styles of photography; semi-random where I snap what is in front of me, and staged where I try to construct a particular picture - the two images above? One of each.)

ObFilm: Bound

Syndicated 2010-02-06 21:13:45 from Steve Kemp's Blog

6 Feb 2010 8191   » (Apprentice)

It's way to easy to do hit-and-run spamming on Advogato. It's so easy to automate, I'm surprised that the recentlog isn't completely dominated by spam. The reason it didn't happen yet is probably that the spammers are, well, dumb (evidence: they don't understand nofollow), but one can't rely on security through stupidity forever. Here's a very simple suggestion, which I volunteer to implement in case the Advogato community agrees: instead of asking for a password on registration, create a random password and mail it to the user; the user should then be able to change the password when she logs in. A password reminding mechanism already exists.

6 Feb 2010 yosch   » (Journeyer)

fontforge support for WOFF

At the request of Chris Lilley from W3C's font activity (among other things), the amazing George Williams has now implemented WOFF support in fontforge. It's available now in CVS but in a release near you probably soon. Thanks Chris and George for your efforts in this area!

A good way to use the new web native format and to view/handle the corresponding metadata.

6 Feb 2010 8191   » (Apprentice)

A user becomes a Journeyer with a single certification, from another user who was, in turn, certified by a single user. I see a problem there. There's way too much trust passing through a single vertex in the graph.

Let's put on our Dijkstra hats and think of a solution. Here's one possibility: create a special vertex - let's call it the "sink". Create edges from every vertex in the graph to the sink (the weight of this edge would need to be tweaked a bit). This would add some "leaking" to trust, and help avoid this sort of situation. The actual masters would continue to be masters, but users with few certifications would see their ratings decreased.

An objection to that solution: it would be hard for vertices that are far from the sources to get their ratings raised. But I don't think there are that many "hops" from the sources to vertices that truly deserve trust.

Edit: looks like I misunderstood how the trust metric works. I assumed that mod_virgule would somehow find the sinks in the graph before running the network flow algorithm, but that wouldn't work, since sinks may not even exist (and they probably don't). All vertices are already connected to a single "supersink" - there's no other way. And the problem of users far from the sources not getting certified already exists.

Anyway, there already is a way to attain the proposed "solution", which doesn't even involve code changes: simply decrease the node capacities as a function of source distance; this is defined in config.xml, inside the <caps> tag.

6 Feb 2010 lucasr   » (Master)

New EOG maintainers

t’s been quite some time that you Felix Riemann and Claudio Saavedra have been doing all the development work in EOG and I think it’s time to pass the ball to them. So, this is just to let everyone know that I’m officially passing the EOG maintainer post to them.
Thanks Felix and Claudio for the great work!

It’s been quite some time that you Felix Riemann and Claudio Saavedra have been doing all the development work in EOG and I think it’s time to pass the ball to them. So, from now on, they are the EOG maintainers. Thanks Felix and Claudio for the great work!

Syndicated 2010-02-06 13:06:16 from Lucas Rocha

New Advogato Features

FOAF updates: Trust rankings are now exported, making the data available to other users and websites. An external FOAF URI has been added, allowing users to link to an additional FOAF file.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

Advogato User Stats
Users13915
Observer9796
Apprentice745
Journeyer2356
Master1014

New Advogato Members

9 Feb 2010 amitsingh0542 (Observer)
9 Feb 2010 hjclub (Observer)
9 Feb 2010 mysand (Observer)
27 Jan 2010 8191 (Apprentice)
19 Jan 2010 malted (Observer)
18 Jan 2010 ittner (Observer)
18 Jan 2010 kawsark (Observer)
16 Jan 2010 feterno (Observer)
9 Jan 2010 cubny (Observer)
29 Dec 2009 mpolla (Observer)
29 Dec 2009 balleyne (Observer)
27 Dec 2009 silvs (Observer)
18 Dec 2009 kscanne (Observer)
15 Dec 2009 torasuku (Observer)
11 Dec 2009 alcor (Observer)
3 Dec 2009 alvinj (Observer)
3 Dec 2009 Trollaxor (Observer)
30 Nov 2009 weakish (Observer)
30 Nov 2009 beafred (Observer)
28 Nov 2009 indyplanets (Observer)

Recently modified projects

9 Feb 2010 Thunderbird
6 Feb 2010 Hail
4 Feb 2010 Plone
3 Feb 2010 QEMU
3 Feb 2010 KVM
22 Dec 2009 GNU Malayalam Project
22 Dec 2009 OpenBSC
14 Dec 2009 ZX-180
8 Dec 2009 Diakonos
22 Nov 2009 BoopCore
9 Nov 2009 Java applets in Wikipedia
4 Nov 2009 SCEW
28 Oct 2009 latexrefman
27 Oct 2009 librep
26 Oct 2009 NX-DOS

New projects

9 Feb 2010 Thunderbird
6 Feb 2010 Hail
4 Feb 2010 Plone
3 Feb 2010 QEMU
3 Feb 2010 KVM
22 Dec 2009 OpenBSC
8 Dec 2009 Diakonos
22 Nov 2009 BoopCore
6 Nov 2009 Java applets in Wikipedia
28 Oct 2009 latexrefman
23 Oct 2009 Bombono DVD
18 Oct 2009 Piwik
17 Sep 2009 Sneer
8 Sep 2009 Atrack
4 Sep 2009 Net::FluidDB