Older blog entries for nikodemus (starting at number 61)

How To Fix A Compiler Bug

I've been asked for an introduction or examples in working on SBCL.

Let's see if we can make sense of lp#738464. The issue is that code such as

(flet ((foo ()))
  (declare (ftype non-function-type foo))
  (foo))

complains about "failed AVER: (NULL (FUNCTIONAL-ENTRY-FUN LEAF))" instead of giving a sensible error.

  1. Replicate the issue.

    Entering the form in Slime lands me in the debugger with the expected error. This is good. It means that the bug is still there, and that the description is valid. If that didn't happen, it doesn't necessarily mean that the bug is gone -- platform specific bugs, bugs appearing only under specific build options, and bugs contingent on specific optimize settings are relatively common.

  2. Study the issue.
    1. What should have happened?

      We should have gotten an error that says "cannot use a non-function type with FTYPE" or something along those lines.

    2. Why did the thing that happened, happen?

      The bad type apparently propagated quite a ways into the system and caused bad things to happen. (No point overanalyzing things.) It is obvious to me that this error does not occur when the system is dealing with the declaration, but much later -- and it's obvious because I've spent a lot of time working on SBCL. If you know nothing about SBCL, here's how you could figure this out:

      • Notice that the backtrace at the point of error says nothing about dealing with declarations, but is full of stuff like SB-C::CHANGE-REF-LEAF, SB-C::LOCALL-ANALYZE-FUN-1, SB-C::REFERENCE-ENTRY-POINT, etc.
      • Figure out where declarations are handled. You could scan through a bunch of code starting from COMPILE and meta-dotting your way forward. You could ask on #sbcl, #lisp, or sbcl-devel. You could use APROPOS or "git grep -e " with various likely things and look if you find something that looks right.

      Let's assume I git-grepped a bit and finally hit upon "git grep -e def.*ftype". One of the matching lines is:

      ir1tran.lisp:(defun process-ftype-decl (spec res names fvars context)

      which looks like it might just have to have something to do with processing FTYPE declarations...

      Before you go and take a look at it, understand this: you don't need to read all of it just now. What you need to do is to make sure it's the thing that processes FTYPE declarations. TRACE is often insanely useful here: hit C-c C-t on top of the function name in Slime, go to REPL, and try the original form from the bugreport again.

      Apropos: M-. inside SBCL sources (for various reasons that would be nice to fix) is sometimes off-by-one-or-two toplevel forms. In this particular case you're likely to end up on process-type-decl (type, not ftype!) So notice that and find the right toplevel form -- or be terribly confused for a while because trace output looks strange. This happened to me just now...

      Having traced the correct definition we see it indeed being called with our NON-FUNCTION-TYPE, so our diagnosis is confirmed: the place that processes the declarations doesn't notice that it's not a function type.

    3. Do I have any idea how to fix this?

      Making process-ftype-decl complain sounds about right. Looking at the code we see that it uses compiler-specifier-type to parse the type. So we need to check that the type returned is a subtype of function.

      If you don't know yet how to do the equivalent of SUBTYPEP on SBCL internal type objects, you can use the same process as earlier to figure this out: read more source, ask around, etc. (The answer is: use CSUBTYPEP).

    4. What if you have no idea how to fix it?

      Spend a while -- 15 minutes or 15 days, whatever -- trying to understand the issue well enough to fix it. Maybe you'll figure it out, maybe you won't. Repeat this excercise over various bugs for a few years, and you'll be an SBCL expert. Understand that sometimes you might need to look for information outside the sources: textbooks, papers, original implementors, etc.

  3. Fix the issue.

    Because I have every intention of actually fixing this just now, I'll assign the bug to myself and mark it as "In Progress". This helps avoid duplicate work.

    We already know the general shape of the fix we're going for:

    (unless (csubtypep ...) ...complain...)

    Now's the time to figure out just what our complaint is. I opt for reporting the invalid type and then ignoring it.

    (unless (csubtypep type (specifier-type 'function))
      (compiler-style-warn "ignoring declared FTYPE: ~S (not a function type)" spec)
      (return-from process-ftype-decl res))

    I edit to above code into the right place in the definition, recompile it with C-c C-c and go to the REPL to test it. Looking good.

    Note on hotpatching the compiler: sure, you can break things easily. It is, however, much faster than doing a full rebuild. Just set up a persistent slime-scratch buffer, and save your work before doing risky stuff. Even if you break something, most of the time you can fix things without restarting -- the rest of the time you do M-x slime-restart-inferior-lisp, or at the very worst you have to go to the terminal to "kill -9" the process. The thing to note is that even if you did do a full build, you'd still run the exact same risks! All that said, there are changes that are hard or impossible to hotpatch: sometimes you do need to do a full rebuild.

  4. Finish up

    I add a test-case for the bug into tests/compiler.pure.lisp:

    (with-test (:name :bug-738464)
      (multiple-value-bind (fun warn fail)
          (compile nil `(lambda ()
                          (flet ((foo () 42))
                            (declare (ftype non-function-type foo))
                            (foo))))
        (assert (eql 42 (funcall fun)))
        (assert (and warn (not fail)))))

    I commit my changes to the local repository with an appropriate commit message. This is a pretty trivial change, so it doesn't need much of an explanation.

    ignore non-function FTYPEs
    
      Fixes lp#738464.
    
      Give a style-warning and ignore the bad type.

    Now I do a full rebuild and run all tests to make sure I didn't break anything.

  5. Send to upstream.

    At this point I would normally write a NEWS entry for the bug, amend the last commit with it, and push to the official repo. However, SBCL is in its montly freeze right now, so I'm holding up for now and just pushed to the pending branch on my tree at Github.

    Non-commiters should at this point just do "git format-patch -1" and add their patch to the Lauchpad bug report, and add the tag "review" to the bug. (NEWS is conflict magnet, so it's easier all around if the person who commits the patch writes the entry.)

Obviously most any point here could be expanded a lot -- but hopefully this provides a starting point of sorts to people interested in digging into SBCL. Fixing tiny buglets like this is an excellent way to get your bearings.

Syndicated 2011-08-17 07:36:35 from Nikodemus Siivola

New crowdfunding goal: MADEIRA

My crowdfunding campaign has been a roaring success: all goalposts I initially set have been reached in 3 days out of 22!

So, time for a new goal.

If the campaign reaches $12k, in addition to the SBCL specific work I will also implement Madeira, a new portability layer for Common Lisp implementations, focusing on threading and parts not covered by CFFI or USOCKET.

The current plan is for Madeira to support:

  • Basic things like access to command-line arguments, environment, EXIT, etc.
  • Bordeaux-Threads -like functionality, but better defined, with sharper edges.
  • CAS. Remains to be seen how general I can make the support, though.
  • Running external programs. (Probably somewhere between TRIVIAL-SHELL and SBCL's RUN-PROGRAM in functionality.)

Another way to describe Madeira would be to say that it should cover most everythingtaht I keep having to #+sbcl when writing portable code.

Gentle hacker, spread the word: it's not just SBCL users who stand to benefit now.

Thanks You again, Everyone.

Syndicated 2011-08-13 09:27:18 from Nikodemus Siivola

Flabbergasted

I'm having my morning cup of black, and just had a look at the SBCL crowdfunding campaign I put up yesterday.

The initial funding goal was reached in less than a day! I'm awed and humbled.

This means crowdfunding looks like a very viable model for funding future SBCL development, though of course the real question is how repeatable this is. I'm sure I'm at least to a degree cashing in on work I've already done as opposed to what the campaign pitch actually says, but the future will tell...

In the meanwhile, please keep spreading the word! The campaign is now at the point where locking improvements and adding timeouts to all blocking functions in the threading API have been funded.

If it reaches $6000, it means getting read/write locks and semaphore notification objects as well.

Syndicated 2011-08-11 06:24:30 from Nikodemus Siivola

SBCL Crowdfunding

My first attempt at crowdfunding SBCL development is live!

Crowdfunding SBCL threading improvements at IndieGoGo.com. Go and take a look.

Syndicated 2011-08-10 14:03:48 from Nikodemus Siivola

Donations by Popular Request

Several people have told me I need a donations button on my website, so I went ahead and added one.

That crowdfunding thing? Expect it to go live next week.

My current plan is to run a fundraiser for assorted threading improvements first, and a second one for external format work after that.

The primary goal of the threading work is to make mutexes, semaphores, and condition variables interrupt safe on non-Linux POSIX platforms (primarily Darwin and Solaris/x86[-64]). Secondary goals are timeouts on all blocking synchronization constructs, adding read/write-locks, semaphore notification objects, and finally a general cleanup/refactoring of the threading API.

The external-format project's goal is adding support for newline conversions and BOM marks, and also making external format conversions a good deal faster. Possible secondary goals are adding support for Unicode-aware case-conversions and similar, plus adding a general vector output and input streams similar to string output and input streams provided by ANSI CL.

Syndicated 2011-08-06 12:52:17 from Nikodemus Siivola

SBCL Merchandice

I have opened a cafepress shop with some Lisp and SBCL branded merchandice -- currently mugs and T-shirts, including zbir's infamous save-lisp-and-die -design.

I've been intending to do something like this for ages now, but never got around to it before. Everything has a markup of around 50%, and the proceeds go towards funding SBCL development.

The crowdfunding thing is still on. More on that soon.

Syndicated 2011-08-03 18:43:16 from Nikodemus Siivola

Crowdfunding SBCL

I'm planning a couple of crowdfunding campaigns on indiegogo.com to raise money for SBCL work.

One campaign will be about external-format improvements, and the other about improving threading on non-Linux POSIX platforms.

Trying to cauge interest a bit, so here's a small questionnaire: SBCL Crowdfunding Questionnaire

Syndicated 2011-08-02 15:58:38 from Nikodemus Siivola

Amazon ebook Surcharges

What's up with the $2 surcharge on ebooks Amazon applies when shopping from Finland? (Or most non-US countries for that matter.)

My understanding is that this is used to bankroll the Whispernet, which is completely, ridiculously disproportionate.

Since buying my Kindle late last year, I've bought some 60 ebooks from Amazon -- paying $120 is surcharges. That's more than enough to pay for a year of unlimited mobile broadband in Finland.

So yeah, I'm feeling pretty butthurt here.

For more on this, see: Amazon Hold Back The Growth Of E-Books Around The World. (TL;DR: $0.99 ebook becomes a $3.44 ebook -- and none of the price difference goes to the author or the publisher.)

Since it's highly unlikely that Amazon will do anything about this unless there's a good deal of publicity about this, please consider sharing this.

Syndicated 2011-07-14 13:49:05 from Nikodemus Siivola

Will Somone Think of The Namespaces?

There is a single namespace for packages in Common Lisp.

There is also a single ASDF system namespace.

Please, make sure you use the same prefixes in both!

To pick a random bad example: system name cl-unification, package name unify. Do not do this.

Just pick one, please. It's getting crowded.

Syndicated 2011-06-14 10:37:14 from Nikodemus Siivola

Slime-Indentation and Names Styles

Slime's slime-indentation contrib now also provides support for named indentation styles. Add a line like this to source files to specify a style:

;; -*- common-lisp-style: modern -*-

Predefined styles are: basic, classic, modern, and sbcl. See C-h v common-lisp-style for more details.

You can also define your own styles:

;;; in .emacs, after slime-setup.
(define-common-lisp-style "personal"
   "My own eccentric style."
   (:variables
     (lisp-lambda-list-keyword-alignment t)
     (lisp-lambda-list-keyword-parameter-alignment nil)
     (lisp-loop-indent-forms-like-keywords t))
   (:indentation
     (if (4 2 2))))

See C-h f define-common-lisp-style for details.

If you define a style for use in the source-files of an open source project, please consider submitting it for inclusion.

Ps. in case you're wondering, you can gain access to all these goodies and more by adding slime-indentation to your slime-setup call:

;; Maybe slime-indentation will be part of slime-fancy at some
;; point, but it isn't yet.
(slime-setup '(slime-fancy slime-indentation))

Syndicated 2011-06-09 16:08:35 from Nikodemus Siivola

52 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!