The Programmer's Guide to Financial Book-keeping, Part I

Once upon a time I knew enough about bookkeeping to implement a a rudimentary accounting system for the consulting business I was running at the time. Then I got a real job, and after that I forgot most of it. Recently I've had to relearn it all, and as the accountancy/bookkeeping web pages that I've found on the Internet are decidedly mixed (an honourable mention here for the Gnucash manual, which is actually quite good), this time I'm writing it down.

The intended audience for this is chiefly me and people like me: computer programmer types who have to make their systems talk to accounts departments and accountants. If you are looking for more information on bookkeeping or accountancy from a professional perspective, it is less likely to be useful.

It should not be necessary - though it probably is - to state that I hold no professional qualifications and have had no training in the field, and if you want proper advice you'll have to pay for it from someone entitled to give it. This information is offered as-is, and no warranties as to its correctness, usefulness or completeness are offered.

Feedback welcome - see the page footer for details.

<h2> Definition </h2>

Let us define bookkeeping as: the collection and processing of financial records for an entity, with the object that interested parties can learn (1) as of a specified time, how much money (and other valuable stuff) it owns, against how much it owes to other entities; (2) over a specified period of time, how much has come in and how much has gone out. Bookkeeping deals not just with money but with all kinds of valuable stuff: cash, shares, financial instruments, land, saleable equipment, stock in trade, etc etc - in the rest of this post I'll be lumping it all together as "value". <h2> End results </h2>

In the UK, the end processes of bookkeeping/accountancy for a company or other trading entity are usually produced annually -

• the Balance Sheet - a document of type (1), which lists the assets (stuff we've got) and liabilities (stuff we owe) broken down by category, at the end of the trading year. We start with assets, listed in order from most liquid (e.g. money at hand or in the bank) to least liquid (things we own that would be complicated to sell), then we subtract liabilities (usually ordered from short-term to long-term), then the bottom line is what we're worth. This is often referred to as the Accounting Equation: <center>Assets - Liabilities = Equity</center> although other people will say that Equity is really what the company owes its owners (e.g. the shareholders) so the equity will appear as a liability account and the equation is "Assets = Liabilities". Mathematically it makes no difference.

• the Profit and Loss account, or P&L - also known in the US as the Income Statement. This is a document of type (2) which lists what's come in and what's gone out over the course of the year.

We probably also want quarterly reports for VAT (that's "Sales Tax" in other countries), and ad-hoc reports for credit control (we need to know who owes us money so we can chase them) and management accounting. <h2> Derivation </h2>

Obviously, if your trading entity is you and you alone and there's no regulatory requirement on you to show anyone else the figures, you can choose any categories you like. But for most of us, there are accepted rules about the breakdown that people want to see, what you're allowed to assign to which categories, and what you'd actually want to assign to which categories (which might be a question with different answers depending on whether you're trying e.g. to maximise profit for the investors or minimise it for the taxman). This kind of decision is what you have an accountant for: keeping the numbers is what you have a bookkeeper for. So, look on the difference between those two roles as a policy/mechanism distinction (and a big difference in hourly rate: don't pay an accountant to do a bookkeeper's job) <h2> Accounts and transactions </h2>

So, with the aid of an accountant we can establish how we need to categorise our assets and liabilities for the reports we need to produce. Each category (or sub-category, or sub-sub-category) is an account: each transfer of value from one account to another is a transaction. A transaction is usually associated with a source document (for example, a purchase order, or an invoice, or a receipt) - the so-called paper trail is not necessarily kept on actual carbon laminate these days, but it's still important. In essence, what we do is record the transactions. <h2> Credit and Debit </h2>

We record each financial transaction as a flow of value from one (or several) accounts into another (or several others). Historically, bookkeepers don't get on with the concept of negative numbers - this is possibly because it can be confusing to have your "Income" account get steadily more negative as the year goes on (we'll come back to why this happens), or maybe just because the principles of double-entry bookkeeping were invented in a time and place (Renaissance Italy) that hadn't really yet heard of negative numbers. Whatever. But the upshot is that they made their own words up instead: the account that loses value is said to be "credited"; the account which gains value is "debited".

This is, of course, completely bass-ackwards from the perspective of normal people, though it has been claimed that the problem is that we're backwards. When the bank send you your statement of account it's printed from their point of view, not yours. So, if you deposit £50 in the Royal NatMid, in their eyes that creates a liability to you (after all, it's money they have but you own). The more money you give them the more they can transfer (debit) to Assets/BranchSafe or Assets/Vault or Assets/SubPrimeMortgages, but they have to credit that transaction to Liabilties/AP/YourNameHere. So the effect is that we perceive being in credit with the bank as a good thing: they see it as a bad thing. It's just a matter of perspective.

(I am slightly suspicious of this explanation. "Credit" and "Debit" are both from Latin roots: /creditum/: "a loan, thing entrusted to another", and /debilitum/: "thing owed," neut pp. of /debere/ "to owe". In the end they're just words, but it's still confusing enough to be just more fuel for my scepticism towards the claim that negative numbers are avoided because they cause confusion. Maybe that's just me.) <h2> Double-entry </h2>

The principle of double-entry accounting is that the value credited in a transaction must equal the value debited - value cannot be created or destroyed. The name comes from paper-based systems: if we have two accounts affected by a transaction, we must enter the transaction details into both. Using a computer, of course, we can enter it once and it will appear in both, but that's not the point. We are interested in the principle of "conservation of value", not so much in the mechanism of how we achieved that in the old days.

A simple example: our shop runs out of float in the till, so on Monday we must visit the bank and get some more cash. We record this as a transfer from the bank account (which is credited) to the till account (which is debited).

Here we're moving value between two asset accounts: they both represent monies that we own - just in different places. So it's pretty easy to see that "conservation of value" holds true. But the principle of double-entry bookkeeping is that the debits and credits in any transaction must always balance, so the astute reader will now be wondering how we do that for a transaction that actually makes the company money. For example, if we do some work and get paid, then the value in the transaction is clearly going into the bank account, but where is it coming from? <h2> Income and expenses </h2>

The answer is that we create "Income" accounts which serve as a proxy for the outside world as it affects our company. So, if we get $200 for configuring Joe's web server, our bank account is debited $200 and the outside world, as represented by our Income (or Income/Sales, or whatever subcategorisation we want to use) account, is credited $200. Expenses accounts serve a similar but opposite role: we pay for stuff (like stationery, utilities, salaries) that makes us poorer (our assets are credited) and the outside world richer (our expenses are debited).

Income and Expense accounts are key to the P&L statement that we will produce at the end of the year, because they act as summaries of our interactions with the world - which is what P&L is all about. The USAnian name for them "Income statement" hints as much.

<h2> Accruals </h2>

Another key concept is accruals. In most businesses there is a delay between when we provide something of value (e.g. do some work) and when we actually get paid: there is also often a delay between when we receive something of value and when we have to pay for it. In a cash accounting system there's nothing we can do about that, but in an accruals system we can create "accounts receivable" and "accounts payable" for these sums which are "in the post". This allows our accounts to say that we are worth $4000 because we have that amount expected to come in from Michael next week, even though we haven't got it in the bank yet. So, this makes payment a two-stage process: first we send an invoice and transfer $4000 from Income/Sales to Assets/AR/Michael, then when he pays it four weeks later (or perhaps four months later if he's a public sector body) we transfer $4000 from Assets/AR/Michael to Assets/Bank. We haven't actually made any new money in that second transaction, but at least it now exists in the bank and not just on paper.

Accounts Payable is similar but opposite. We order office furniture on account, it gets sent with an invoice, and we log that transaction as a transfer from Liabilities/AP/IKEA to Assets/Furniture. When the invoice is due (or three weeks later if you have really good credit control) we send them a cheque and we do another transaction from (crediting) Assets/Bank to (debiting) Liabilities/AP/IKEA, which hopefully reduces the balance of the latter account to zero.

Most of the examples later in this post ignore accruals in much the same way and for the same reason as Kernighan and Ritchie ignore error checking: it slightly obscures the pedagogical point, but that doesn't mean you won't do it for real. <h2> Sale of goods </h2>

If you're selling services, the transaction is Income/Sales->Assets/Bank. That's simple. If you're selling goods, though, (1) you have to buy them first

£6 cr. Assets/Bank = dr. Assets/Inventory/Widgets

and then (2) when you sell them you are selling at a different price.

£10 cr. Income/Sales = dr. Assets/Bank
£6 cr. Assets/Inventory/Widgets = dr. Expenses/Cost of sales

The net effect is to increase Income by a tenner and Expenses by an unwell cephalopod (that's "sick quid" to you. Sorry). Thus both effects of the transaction will be represented on the appropriate P&L rows.

<h2> VAT / Sales Tax </h2>

VAT in the UK is not really ever money we have earnt, it's just money we are collecting on behalf of the nice people at HMRC. So, if we are registered for VAT we must collect it on each sale into a holding account which we send them later, but it's not "ours" and doesn't show in Sales.

£20 cr. Income/Sales + 3.50 cr. Liabilities/VAT = 23.50 dr. Assets/Bank

Watch out for the credit/debits in that transaction. We should end up with cash in the bank (a debit), some of which is owed to the VAT man (credit). If they don't sum to zero, you've done something wrong.

Similarly we can also claim back VAT on purchases from our VAT-registered suppliers

11.75 cr. Assets/Bank = 10.00 dr. Assets/Inventory + 1.75 dr. Assets/Input_VAT

At the end of the quarter, we pay HMRC what we owe them, less what they owe us

1.75 cr. Assets/Input_VAT + 1.75 cr. Assets/Bank = 3.50 dr. Liabilities/VAT

Note that this is not reflected in any Expense account - it shouldn't be, because it wasn't in an Income account to start with

<h2> Year end </h2>

We've already talked about producing the Balance Sheet and P&L. The other action we take at end of year is to close the accounts: in the case of Income and Expenses, we will want to start the following year with a clean sheet. How to do this: after producing the end-of-year reports, move the entire contents of Income and Expenses accounts into a summary "Retained Earnings" account, debiting and crediting as appropriate.

<h2> Contingent concepts </h2>

We have not talked about: journals, day books, cash books, general ledgers, T accounts, and trial balances. Most of these are historical practices that are necessary in manual systems either because the latency of entering everything directly in double-entry form is high (so transactions are initially recorded elsewhere instead), or because there is no automatic checking that the accounts are in balance, or because obtaining summaries of groups of accounts (answering queries like "what's the total AP for all suppliers") isn't a trivial bit of SQL.

<h2> Where next? </h2>

This is Part I of a two-part series. In the second part I'm going to write about my experience implementing all this in Ruby, but that will have to wait until I've done the actual implementation.

Hopefully though, this post should provide you with a view of the principles such that you can google for anything else you see and you have a framework to hang it on.

Syndicated 2010-02-09 14:58:49 from diary at telent netowrks

Oh. Ouch, Sorry

I've only just spotted this host didn't come back cleanly after last night's power-cut-induced outage. Well, here it is now

In the meantime, I have found the stunningly attractive Ruby FFI gem and written some rather gross code to play music through PortAudio with it. Unfortunately it turns out that PortAudio's Pa_OpenDefaultStream function is buggy (either the code or the spec) and it defaults to OSS anyway. Still, a step closer

Syndicated 2010-01-22 23:03:51 from diary at telent netowrks

Changing OSS in mid-stream

It was always a given that the Oliver OSS (that's "/dev/dsp" to you) interface would eventually need to be swapped out for something else, just because it doesn't work on many machines other than mine.

Now Oliver has reached the point that it functions as a rudimentary music player for general use (i.e. it presents me a list of my music on the left that I can drag into the playqueue on the right), I realise that "eventually" is sooner than I was expecting, simply because it opens the dsp exclusively and forces all other audio programs (e.g. youtube vids) to fail. In short, it doesn't work acceptably well on my machine either. Fail. I'd forgotten how miserable Linux audio used to be.

In other news

• webkit doesn't support javascript 1.7, which is a grave disappointment as this is the one with proper lexical scope (via let )

• minor edit to previous post for comprehensibility, as re-reading it I realised it made more sense in my head than on the page.

• as tweeted: "there is a fine line between genius and idiot, and I'm still not sure on which side of it to place the jquery api". Selecting document elements using CSS syntax, yes, great idea. Returning those elements in an object sufficiently similar to an array that Firebug prints it as square brackets, but then making it respond to map in an entirely different way to the real Array - no, stupid idea. Hours of my life I won't get back

Syndicated 2010-01-19 22:04:11 from diary at telent netowrks

Testes on testing

Lest the reader assume from my previous post that I'm against automated testing: no, I'm not. In fact, Oliver hacking time over the last couple of days has been all about writing tests for the playqueue and turning the hacked together OSS interface into something that can pass them.

I offer for your consideration, though, that the benefits of writing a test suite are not so much about "having an executable specification" or even catching regressions as they are about making the software easier to test, by (1) decoupling interfaces so that units may be tested without a plethora of complicated test doubles (stubs, mocks) which may themselves contain bugs, and (2) reducing their dependence on complicated state. The easiest code to test is the purely functional, and happily this is also the easiest code to statically reason about (thus reducing our need to write tests in the first place).

There are other considerations: I object to the false confidence of "our change passes tests, so we can feel good because it must be correct" - though I suspect I'm fighting a straw man there anyway - and I am not sure that even attempting to write tests for some classes of bug (say, race conditions) is a more productive use of ones time than, say, Thinking Very Hard at the program to be tested. And there's the whole issue of whether the executable specification (probably written by a computer programmer) is a fair reflection of the business requirement (if written by the "customer", probably much less precise and probably not even consistent), but by that point we just have to accept that, well, TDD won't fix world hunger either. Chiefly my message is that writing (and more importantly, maintaining) tests is Not Free nor axiomatically Good, and therefore is only to be commended when there is some expected benefit from it.

I leave you with two final thoughts that are vaguely related but don't fit into the overall argument anywhere else...

First: correct me if I'm wrong here, but

• Test-Driven Development is what, back in the XP days, they called "test-first programming": you write the unit tests before the code that they test. In Ruby, Test::Unit is/was the tool for running unit tests

• Behaviour-Driven Development is an exercise in redefining concepts, to change the emphasis from unit tests (which typically operate on a particular class) to functional tests (which may span several classes, and address user stories or whole-system behaviour). In Ruby, RSpec was developed with this end.

• Missing the point, therefore, must be the practice of editing all your Test::Unit cases into RSpec syntax to pretend more convincingly to the new orthodoxy. They don't magically turn into functional tests just because test_barf_when_value_negative is now written it "should barf when value negative". Er? Do they?

Second: a powerful driver for a good test suite, from my experience with SBCL, is that of checking the program is not broken by environmental changes (new os, new cpu architecture, whatever). When working on that project I saw test suite breakages far more often in those circumstances than I ever did from hacking the code they were supposed to protect.

Syndicated 2010-01-16 00:42:37 from diary at telent netowrks

Streaming XHR with Ruby and Mongrel

An entirely pun-free post title today, and still it sounds like something you'd see a vet about. Hey ho.

Suppose you are writing a web app in which the server needs to update the client when things change, and you don't want to do it by polling. It turns out there is a technique for this that is probably more than two years old: you make the client do an XMLHttpRequest (aside: that name is almost as bad as its capitalization) to the server, and then the server sends its response v e r y   s l o w l y. The clients XmLhTtPReQuEst object will get an onreadystatechange event every time a new packet arrives, and just has to pull the new data out of xhr.responseText and decide what to do with it.

Well, that's the theory. There are a variety of more-or-less-documented bugs and pitfalls to do with browser compatibility, as there always are (google "Comet", there are lots of resources and none of them I have the personal experience to recommend), but the new wrinkle I observed when doing this yesterday was a quarter-of-a-second lag between the server sending and the client receiving. Odd. Ruby can't be that slow, can it?

Wel, no, it's not. After monkeying with wget and netcat and wireshark and mostly failing to find out what was going on, I did strace -e setsockopt ruby server.rb and connected to it, and lo, what should I find but that something in Ruby or something in Mongrel was setting the TCP_CORK socket option

setsockopt(5, SOL_TCP, TCP_CORK, [1], 4) = 0

TCP_CORK (since Linux 2.2)dd> If set, don't send out partial frames. All queued partial frames are sent when the option is cleared again. This is use‐ ful for prepending headers before calling sendfile(2), or for throughput optimization. As currently implemented, there is a 200 millisecond ceiling on the time for which output is corked by TCP_CORK. If this ceiling is reached, then queued data is automatically transmitted. This option can be combined with TCP_NODELAY only since Linux 2.5.71. This option should not be used in code intended to be portable.

Now I don't know where it's being set - a cursory grep of the mongrel sources says probably not there, but it's simple enough to unset again. So, here's one I made earlier. Note that you can't use the response.start method (or at least, I don't see how) - you have to reach a little deeper into Mongrel::HttpResponse

class StatusHandler < Mongrel::HttpHandler
  def process(request, response)
    fh=nil
    response.status=200
    response.send_status(nil)
    response.header['Content-Type'] = "application/x-www-form-urlencoded"
    response.send_header
    # something inside Ruby or inside Mongrel is setting TCP_CORK,
    # which is bad for latency.  I suspect Ruby C code, because 
    # the interpreter complains there is no definition for Socket::TCP_CORK
    # <linux/tcp.h>:#define TCP_CORK 3 /* Never send partially complete segments */
    response.socket.setsockopt(Socket::SOL_TCP, 3, 0)
    response.socket.setsockopt(Socket::SOL_TCP, Socket::TCP_NODELAY, 1)
    response.write("# gubbins for webkit bug "+("." * 256)+ "\n");
    response.write("# stuff follows\n");

    300.times.each do
      response.write("stuff\n")
      response.socket.flush
      sleep 30
    end   
    response.done
  end  
end

We limit the response to 300 lines in case of browser timeout or connection interruption or just to stop the client-side memory going up unboundedly as responseText grows without let or limit. It's simple for the javascript to kick off another handler when this one dies.

For completeness, here's some client-side code to go with it

// XXX we made the_req global just so that we can look at 
// what's going on in firebug.  It's not required in normal use
var the_req;
function json_watch_stream(url,callback) {
    var req =new XMLHttpRequest();
    the_req = req;
    req.open("GET",url,true);
    req.last_seen=0;
    req.onreadystatechange = function() {
	if(req.responseText) {
	    callback(req.readyState,req.status,
		     req.responseText.substr(req.last_seen))
	    req.last_seen=req.responseText.length;
	}
    };
    req.send(null);
}
function json_start_status_receiver (){
    json_watch_stream
	('/status',
	 function(ready,status,text) {
	    if(ready==4) {
		// server response concluded, need to start again
		json_start_status_receiver ();
	    }
	    text.split("\n").map(function(line) {
		    if(!line) { return; };
		    var data=line.substr(1);
		    switch(line[0]) {
		    case '#': break;
		    case 'O': update_track_timer(data); break;
		    case 'P': update_track_number(data); break;
		    case 'S': stop_track_timer(); break;
		    default: 
			console.log("status stream: unrecognised flag ",
				    line[0],data);
		    }
		});
	});
}

Syndicated 2010-01-13 10:56:29 from diary at telent netowrks