Older blog entries for gary (starting at number 258)

Future archaeology

Andrew Hughes pointed out yesterday that the ARM interpreter and JIT are slated for removal in IcedTea6-1.11 unless someone steps up to maintain it. Currently there’s only one place where the all information about what’s required is collated—inside my head—so I thought I’d better write it up before I start forgetting. It’s entirely possible the interpreter will be removed, but it’s also possible that someone will end up trying to resurrect it months or years down the line. If you are that person and you are reading this then you owe me a beer ;)

The first change that broke the ARM code was the fix for PR icedtea/323, aka Sun bug 6939182. I described the required fix here:

“[In the ARM code] last_Java_sp is set to the address of the top Zero frame wherever the frame anchor is set up. It needs changing such that last_Java_sp is set to thread->zero_stack()->sp() (and the new field last_Java_fp gets set to what last_Java_sp used to be set to).”

The second change that broke the ARM code was the fix for PR icedtea/484, aka Sun bug 6951784. I described the required fix here:

“I have had to change the calling convention within Zero and Shark. All method entries (the C function that executes the method) now return an integer which is the number of deoptimized frames they have left on the stack. Whenever a method is called it is now the caller’s responsibility to check whether frames have been deoptimized and reenter the interpreter if they have.”

The third change, currently in progress, reverts the last commit by the ARM code’s author, Ed Nevill: fix for fast bytecodes with ARM/Shark. This piece of code was accidentally incorporated in one of the webrevs when Zero was upstreamed, and isn’t conditionalised correctly. It can cause problems when the ARM code is not present, and there’s no neat fix. Given that the ARM code has been broken for five days shy of a year now I’ve asked for it to be removed from OpenJDK. This is Sun bug 7030207. If the ARM code is resurrected, this patch will require reinstating (with more specific conditionalisation please!)

The fourth change, currently in the future, is JSR 292. Explicit method handle stuff should just work–it’ll be handled by Zero–but the ARM interpreter and JIT will need updating to support three new instructions: invokedynamic, fast_aldc and fast_aldc_w. The latter two are internal instructions, in case you wondered why you’d never heard of them before!

Ok, that is all.

Syndicated 2011-03-24 12:14:33 from gbenson.net

ARM interpreter

I just discovered that the ARM-specific interpreter stuff that Ed Nevill wrote (and then abandoned) last year has a hack that disables it when run with -XX:+PrintCommandLineFlags. I guess this is one problem when you have 6,000 lines of assembler nobody understands: you don’t know what secret weird shit is buried in there.

Syndicated 2011-03-14 14:28:01 from gbenson.net

JSR 292 and Zero

Maybe you’ve heard about JSR 292: Supporting Dynamically Typed Languages on the Java™ Platform? Well, it’s VM changes, and slated for OpenJDK 7 so I figured I ought to take a look at it before it suddenly appears and breaks Zero all over the place.

I’ve been working on it for a couple of weeks now over in the old Shark forest. It’s by no means stable, but if you want to have a play with it then here’s how:

  1. Build yourself a recent(ish) copy of OpenJDK 7, one that has the JSR 292 stuff in the class library. I had a copy of the jdk7-hotspot-comp forest lying around, so I used that, but I expect you could use IcedTea7:
    hg fclone http://hg.openjdk.java.net/jdk7/hotspot-comp
    cd hotspot-comp
    export ALT_JDK_IMPORT_PATH=/path/to/some/existing/jvm
    export ALT_BOOTDIR=$ALT_JDK_IMPORT_PATH
    export DISABLE_NIMBUS=true
    export ALLOW_DOWNLOADS=true
    . jdk/make/jdk_generic_profile.sh
    make
  2. Maybe go and have a cup of coffee while it builds…
  3. Clone yourself a copy of the Shark forest:
    hg fclone http://icedtea.classpath.org/hg/shark
  4. Edit the Makefile in there, changing JAVADIR to point to the JVM you just built.
  5. Also change JUNITJAR to point to a JUnit 4 jarfile. The location there is where the Fedora junit4 package puts it, so if you have that installed you should be ok.
  6. If you aren’t building on x86_64 then you’ll need to edit build.sh too. Set ZERO_LIBARCH, ZERO_ENDIANNESS, ZERO_ARCHDEF and ZERO_ARCHFLAG to appropriate values for your system.
  7. Run make.

If you got your editing right it’ll build a new HotSpot, and create a copy of the JVM you built with the new HotSpot dropped in. It’ll then run the OpenJDK 7 JSR 292 unit tests on it.

They’ll fail, of course. Currently there’s no support for invokedynamic yet: I’m still working on the method handles code that underpins it. Method handles look like ordinary methods, except when you call a method handle the VM is presented with a chain of transformations that need applying to the call’s arguments and return value to translate between what the caller supplied and what the eventual callee is expecting. The bad news is that there are some 40 (!) different transformations, of which I’ve implemented maybe 15. The good news is that (I think!) I’ve figured out the framework of it all, so now it’s mostly a case of run the code, read the “unimplemented” message it spits out, and implement the thing it was complaining about. Just like the old days :)

Syndicated 2011-03-03 15:52:39 from gbenson.net

Shark now in OpenJDK 7

It’s taken a while, but all the pieces of Shark’s build system finally percolated through into an OpenJDK 7 release (build 112, released on October 1). Sadly a couple of HotSpot interfaces changed in the interim so you need to grab this changeset to get it working. We’ll get there eventually!

Syndicated 2010-10-12 08:29:13 from gbenson.net

Shark build passes TCK

An IcedTea build of OpenJDK using Shark passed the Java SE 6 TCK today. Fedora 12, x86_64, LLVM 2.6, icedtea6-7674917fa451. Dr Fun is here!

Syndicated 2010-06-10 15:08:48 from gbenson.net

Zero and Shark in IcedTea

Over the past few months I’ve been working on Shark in it’s own forest. This has allowed me to track upstream HotSpot (and the goal is to upstream Shark, so it’s the correct place to base it) but it’s meant that the Shark (and Zero) in IcedTea6 are old. I’m trying to update Zero and Shark in Icedtea6, but it’s a nightmare.

Zero in IcedTea6 has the ARM interpreter which can’t go upstream. Upstream has all the JSR 292 stuff which can’t go downstream. Between these two are fixes that need synchronizing, in both directions. One of the fixes, 6939182 (aka PR icedtea/324) requires the ARM interpreter to be updated before it can be committed, so that needs keeping separate… except that the changes to Shark to support that one are pretty invasive and hard to strip out.

It’s a real mess. I’m pretty close to giving up on it.

Syndicated 2010-05-07 08:37:10 from gbenson.net

New stack overflow code

It’s funny, but I’d kind of thought that the version of Zero that got upstreamed was pretty much cut and dried. It hadn’t had any real changes for months, after all, and the S/390 compatibility work required was so minimal—in terms of code changed, if not in terms of time taken!—that I didn’t think there was much more to do. But in this last couple of months I’ve made a ton of fixes, including one big change that’s upstream (improved stack overflow handling) and one that’s hopefully heading that way (some frame anchor fixes).

The new stack overflow code is pretty awesome (if I say so myself!) The old code jumped through some pretty bizarre hoops in order to throw the exception from the exact frame of the method being called. This is more or less how it happens in “classic” HotSpot, which decides which method you’re in from the ABI stack, but wasn’t workable for Zero which might have run out of Zero stack with no room to put the frame of the method from which you want to throw the exception. Argh! So I wrote this look-ahead detection thing, to throw the exception when you still had some stack to put your frame on, which worked, but was easily fooled. A valid Java method could have a frame far larger than the default thread stack size on most systems, so the lookahead worked on the assumption that a frame probably wouldn’t be that big. There were a number of places where the stack could overflow, in which case you’d hit an Unimplemented and the VM would bomb out. Not good, and not very secure either.

The new code is awesome. I realised that it doesn’t really matter exactly which frame the StackOverflowError got thrown from, only that it got thrown. I wrote a little method, ZeroStack::overflow_check, which is inline and can be called from anywhere—any VM state, etc. It does the check, and throws the exception, via ZeroStack::handle_overflow. That one is pretty complicated, but I did say I wanted the overflow check to be callable from anywhere, so it has to set up the frame anchor if it isn’t already, and cope with whatever thread state it finds itself in. The best thing about ZeroStack::handle_overflow? You can call it from Shark, which let me blow away all the ugly stack overflow code I was griping about in my last post.

Those Unimplementeds in the stack overflow detector were my least favourite thing about Zero, my guilty secret (in as much as open source code published on the internet is secret). And now they’re gone! No more ugly code in Zero! Until a few days later, when I discovered some stuff in the frame anchor that was reversed. But that’s for another blog…

Syndicated 2010-04-21 11:23:57 from gbenson.net

Stack overflow detection

I added stack overflow detection to Shark last week. It works, but it made me realise just how odd Zero’s overflow detection is. Knowing what I know now, I think I can make it cleaner, better and faster, for Zero and Shark. That’s my project for this afternoon…

Syndicated 2010-03-24 13:26:16 from gbenson.net

Shark

I’m back on Shark, after a four month hiatus. A minor milestone: it can build itself again.

Syndicated 2010-03-12 09:29:02 from gbenson.net

JNI wrapper compilation

I now have a version of Shark with a basic implementation JNI wrapper compilation. Sadly I can’t say if it’s faster or not yet because it’s totally unstable!

The problem is this. When HotSpot wishes to compile a normal (interpreted) method, the thread initiating the compile simply adds it to a queue and carries on doing whatever it was it was doing. A separate thread, the compiler thread, loops over this queue, compiling methods one at a time. This means there’s only ever the one thread making LLVM calls, and everything is rosy.

When HotSpot wishes to compile a native (JNI) method, the thread initiating the compile bypasses the queue and does it immediately. It acquires a lock, the adapter hander library lock, so there’s only every one native method being compiled at once, but in the meantime the compiler thread is in all likelihood busy compiling some normal method or another, so there’s two threads making LLVM calls. LLVM doesn’t like this, not LLVM prior to 2.6 at any rate, and even then not without being written with a separate LLVMContext for each thread.

The obvious fix for this is for Shark to acquire a lock before compiling either a normal or a native method, ensuring that only one thread is calling into LLVM at once. This doesn’t work, however, as the compiler thread runs _thread_in_native. The benefit of this is that the compiler thread does not have to halt for safepoints (and the rest of the VM doesn’t have to wait for the compiler thread to halt) but the drawback of this is that threads running _thread_in_native may not own locks. You can’t make the compiler thread run other than _thread_in_native, not without losing the large chunk of the server compiler that Shark shares, and you can’t hack a lock in there anyway (by using a pthread mutex, say, rather than a HotSpot lock) because it’ll deadlock the first time a safepoint occurs when the compiler thread holds the lock (compiling a normal method) and a Java thread is blocking trying to take it (to compile a native one).

I’ve been circling around this issue for a couple of days now, but it looks like the only solution is to require LLVM 2.6 and rearrange everything to use separate contexts.

Syndicated 2009-10-28 10:57:11 from gbenson.net

249 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!