Perl 6 Design Minutes for 07 May 2008

<summary type="xhtml"></summary>

Syndicated 2008-05-08 22:42:28 from pudge

Good Error Messages are Important

Parrot r27355 was fun to write.

One of the persistent error messages Parrot emits for compiler writers is Null PMC access in invoke(). If you've had your hands deep in the guts of Parrot, you know what that means -- you tried to call a Sub PMC when you don't have a Sub PMC, you have no PMC. (If you don't know what that means, this entry is for you.)

Sometimes this means that there's a problem in Parrot. We've fixed almost all of those problems though, so the error usually comes from elsewhere. If you're writing a compiler, or running a compiler built on Parrot, the error usually means "You tried to call a function that doesn't exist."

Parrot's optimizer does something interesting at the end of compilation time. You've probably heard that Parrot's compiler, IMCC, translates PIR into PBC. That is, it turns source code into bytecode, which Parrot can either serialize t to disk or execute immediately. That bytecode is just a chunk of linear data in memory. It's not really a data structure. (Okay, it's a C array, but that doesn't make it a data structure.)

After IMCC has finished building a standalone chunk of bytecode, it performs a constant fixup phase. The notable part of this phase is that it edits the bytecode in place to replace all named invocations of functions known at fixup time with offset invocations.

The previous code looks something like:

invoke known_function
null    # padding
null    # padding

If IMCC has already seen known_function by this time, the direct invocation of known_function can continue. There's no runtime lookup necessary; all functions already compiled and ready are available in the bytecode.

If IMCC hasn't seen that function, runtime lookup is necessary, and so this function replaces the bytecode earlier with the equivalent of:

<nobr> <wbr></nobr>.local pmc func
func = find_name 'unknown_function'
invoke func

(I've simplified what actually happens slightly, because the concepts are more important than the details. Hopefully you see why the padding is necessary. If not, just imagine trying to splice additional opcodes into what may presumably be a lengthy C array -- like I said, barely a data structure.)

The problem with this second form occurs when find_name returns a NULL PMC, which it can legitimately do. In that case, the invoke opcode tries to invoke a NULL PMC and fails, and Parrot throws an exception saying "There's nothing here to invoke." There's the error message.

It's clear why that happens, but it's not useful. It would be more useful to see the name of the function you tried to invoke in the error message. Unfortunately, by the time Parrot calls the invoke op, that name is long gone.

My first idea was to rewrite the dynamic lookup form into something resembling:

<nobr> <wbr></nobr>.local pmc func
func = find_name 'unknown_function'
if defined func goto call_it
die "Can't invoke 'unknown_function'"

call_it:
invoke func

Unfortunately, I didn't have the space in the bytecode stream to insert that many ops, and I had no desire to move chunks of memory around in that C array. I could have added more padding after an invocation, but to be fair I'm only mostly sure that it exists there in the first place.

I had room for one op with a destination PMC and a string constant argument. I added an experimental op called find_sub_not_null which does the same thing as find_name but throws an exception which includes the requested name if Parrot can't find a PMC of that name.

This isn't entirely an ideal situation. It's a special case op, and I prefer to remove ops where possible. It's also nearly code duplication, though it's effectively three lines of code in an op, which isn't awful. I still want to be able to perform these kinds of transformations in PBC itself, but we need a different way to generate PBC and perform op-level transformations in PIR before we can do this effectively.

There are always tradeoffs, though. Doing this check in C is slightly faster than doing it in PIR. The standard Perl 5 rule of optimization applies even in Parrot -- the fewer ops, the mostly faster you can go. As well, I was able to improve the warning message today, rather than at some point in the future when we have better PBC optimization possibilities.

After all, I can always remove this op in the future.

Syndicated 2008-05-07 05:42:03 from pudge

What are You Going to Do, SMS at Me?

<summary type="xhtml"></summary>

Syndicated 2008-05-04 00:23:56 from pudge

What a GC Bug Looks Like in Parrot

<summary type="xhtml"></summary>

Syndicated 2008-04-28 07:19:51 from pudge

Perl 6 Design Minutes for 23 April 2008

The Perl 6 design team met by phone on 23 April 2008. Larry, Allison, Patrick, Jerry, Will, Jesse, Nicholas, and chromatic attended.

Jerry:

• we're in a period where everyone's trying to break Parrot
• they're adding new features and accidentally breaking thing
• but they're fixing it
• it's a good part of the cycle
• people fix it
• we don't have a build farm, so we can't test everywhere before committing to trunk

Patrick:

• I thought that was the point of the release cycle

Jerry:

• some people have suggested that we always keep trunk building and passing tests
• but we don't have the means to do that
• especially when we're playing with config
• moving on, the big news is that TPF has six slots in Google's Summer of Code
• one of them is fleshing out the Perl 6 test suite
• we've needed someone to spearhead that
• having a funded new contributor is wonderful
• two Parrot-related projects
• one is generating NCI stubs
• Kevin Tew, a long-time contributor
• the other is the incremental GC specified in the PDD
• that's Andrew Whitworth
• there's also an ASF project for integrating the GC from Apache Harmony into Parrot
• they've wanted to release it as a standalone library
• Parrot's the first test of a standalone system

Nicholas:

• nice that it doesn't count against TPF's slot list

Jesse:

• and it's nice visibility for Parrot from another group

Jerry:

• I finally have six weeks of no plans to travel
• should be able to devote more time to Parrot and Rakudo
• looking forward to that

Larry:

• getting some hacking in on my two pet bugaboos, the longest token matcher and match object generator
• I refactored the matcher
• it still uses TGE
• instead of lumping all of the expect term possible tokens (that is, all of them) into one bucket it separates them into buckets based on their first letter
• it's a one-level tree
• we can build a much smaller DFA for the regexes that start with that letter
• it caches that, of course
• can get an instant reject if the next token can't possibly start with that letter
• also flattened out all of the rules such that the list of tokens is easy to read
• if the first probe with the DFA engine fails, I can take that small set of tokens that start with the same letter, run all of those rules, and sort from longest to shortest
• preserves the token matching order without building huge DFA structures
• as a backoff strategy, that will scale pretty well
• refactored the parameter passing on the matcher side (STD 5)
• instead of passing an initial array of random things, I have parameters
• constructs match objects more correctly
• in the sense that it gets all the information it's supposed to have
• also has some attachments where it shouldn't have
• I'm cleaning that up
• that should scale pretty well

Jerry:

• is there a drop in memory usage?

Larry:

• I haven't measured
• I'm sure that not feeding 800 regexes to TRE at once will make it allocate 17 megabytes on the stack
• it might still be allocating too much for some of the larger things
• I'm still aiming for correct, as opposed to fast
• just trying to bear fast in mind
• the longest token matcher now returns a linked list of states
• not a string
• should be a lot faster; easier to cache
• functionally it's the same as before
• one of those things you don't even have to measure to know it'll be faster
• trying to avoid the bugaboo of premature optimization by doing what I know will be efficient to begin with
• all the while trying to make the thing work
• it has a good chance of being pretty speedy
• my talk in Tokyo will be about all of the places where the current grammar allows extensibility
• it'd be nice to be able to demonstrate some of that

Allison:

• getting work done again
• launched the Strings PDD
• list of tasks for concurrency that I'm breaking down into smaller pieces
• may post what I have now, and leave other people to break them down

Patrick:

• are they parallelizable?<nobr> <wbr></nobr>:)

Allison:

• many of them are
• there are some bigger things, like switching the exception system over to the event handler
• otherwise, just life stuff

Patrick:

• had paying work come up this past week, so not a lot of actual coding
• need to type the milestones document
• it's all in my head
• managed to remove a lot of unused code thanks to chromatic's post about possible optimizations
• mostly just cleanup
• but helped me figure out things which will feed into my redesign of PGE for longest token matching
• should be able to return to direct Rakduo hacking later this week

Will:

• various Parrot cleanups
• TPF quarterly grant proposals are due at the end of the month
• haven't seen anything come in yet

Allison:

• they're queued in an RT queue
• I don't know if grant members have access to that queue

Will:

• we do
• please, get your proposal in now, sooner than later
• that goes for you on the call as well as people reading the minutes

c:

• mostly spent the past week optimizing Parrot and Rakudo
• looks like it's the building speed is twice as fast as when I started
• runtime is faster too, but the optimization is compilation time
• found some infelicities that need more design thought
• but I'm happy to put these improvements in now and take them out later when the design improves around them
• hope to start adding new features again soon

Patrick:

• most of the test execution time is in compilation
• how useful would it be to compile Rakudo to standalone PIR?

c:

• I'd find it useful
• but I'd find about ten things useful with all I work on
• so not a blocker at the moment

Jesse:

• how far will that get you toward native executables?

Patrick:

• the existing trick for building perl6 would work
• but it's not the same

c:

• if it takes an hour or two, it would help me with debugging and profiling
• if it takes more time, it's not that important

Patrick:

• we have to figure out runtime deployment issues for the Perl 6 runtime library

Will:

• we could add the requirement to run from languages/perl6/ right now

c:

• that's fine by me for now

Patrick:

• that's an afternoon job, not too bad
• what do we need to do to get the Perl 6 and Parrot pages up to date?

Will:

• I'll work on that

Nicholas:

• why is C99 useful to Parrot and the compiler tools?

c:

• front-end parsing for C header files to build NCI declarations automatically
• the backend is pretty easy, that's thunk generation
• the front-end keeps people from having to write boilerplate code by hand
• generate the front-end once, where you have the headers, and then you can run the generated code anywhere even if you don't have the headers

Jesse:

• how does that compare to Python's ctypes?

c:

• as I understand it, they have the nice backend stuff
• not so sure about the front-end
• my P5NCI is nicer, if incomplete
• just haven't had time to work on it...

Jesse:

• if you put in for a TPF grant, that would be very useful

c:

• get me a clone first, and you have a deal

Jerry:

• Allison, we talked about implementing return
• that requires tying in exceptions to the concurrency scheduler?

Allison:

• yes
• just not implemented yet
• when we did exceptions, we didn't have concurrency
• so it's on the top of my list to tie them together

Will:

• Tcl's already using exceptions to handle return, break, and continue

Allison:

• right now, you can't have an exception handler which is a full subroutine

Patrick:

• I'm not sure we need one for that feature
• every subroutine block decorated as such in PAST puts an exception handler in the block
• if any nested block throws a return type exception, it grabs the arguments, does what it has to, and then does a Parrot return

Allison:

• if that's what you need, go ahead and do it
• I thought there are some features you didn't have yet

Patrick:

• I thought there might be some opcodes I needed, like handled
• but we can do something now
• might not be completely optimal
• but it's just packaging things up now
• I have something I think will work
• it's not trivial, but I'm 80% confident
• just a matter of sitting down and doing it

Allison:

• the concurrency stuff will be there before the next release
• might not want to roll it in before the release
• but it'll be there soon

Patrick:

• I want to get return in for the April 15 milestone we're behind on

Jerry:

• have you put in tickets for the breakdown of specific tasks?

Allison:

• I've never done tickets for that
• just sent mail to the list of the tasks
• handed them out to people as they volunteered

c:

• can you put them on a wiki page?
• some of the other committers wanted that

Allison:

• I can do that

Syndicated 2008-04-24 21:16:01 from pudge