Advogato blog for rlougher

The long and winding road of JSR 292

Mon, 31 Dec 2012 23:09:35 GMT

As the biggest change to the JVM since the introduction of Java, I've been acutely aware that I can't ignore JSR 292 and simply leave it unimplemented in JamVM. This has been reinforced recently by announcements such as Nashorn that indicate that more and more developments in the future will require JSR 292.

However, my record of implementing it has been dismal. I first started about this time last year (end of December 2011). At the time I hoped to get it finished for FOSDEM in February. This was a tall order, but by FOSDEM I had invokeExact working, and could run several simple examples (all with OpenJDK 7). I understood the general framework, and the way in which method handles were chained together (invocation involved following the chain, performing transformations along the way, until the final target method was reached).

The problem was the number of transformations the VM needed to implement. Certain ones such as unboxing (REF_TO_PRIM) and retyping were straight-forward. But more complex ones such as argument spreading were extremely time consuming, and there were even more exotic ones requiring "ricochet frames" (I never did work out exactly what they were). Anyway, I didn't deliberately abandon the work, I just stopped and didn't touch JamVM for 6 months.

The breakthrough came from reading Roman Kennke's blog of getting Zero working with OpenJDK 8. Of particular interest was the removal of the adapter code from Zero (like JamVM, it was incomplete). It was no longer needed as the transformations are now done in Java, via LambdaForms. These are compiled into bytecodes via the JSR 292 runtime, again coded in Java.

So the main stumbling block of my last attempt was gone. However, unlike with Zero, in addition to invokedynamic the rest of the JSR 292 support within the VM has to be implemented (class file changes, resolution of method handles/types, call site bootstrap methods, stack walking, field injection, etc.). It's taken a couple of weeks work, with a lot of debugging over Christmas (it's been so wet in the UK there's been nothing much else to do) but all but one of the jtreg tests for java.lang.invoke pass. In fact, all tests were passing until I updated my sources a few days ago and found an extra 6 tests.

Passed: java/lang/invoke/6987555/Test6987555.java
Passed: java/lang/invoke/6991596/Test6991596.java
Passed: java/lang/invoke/6998541/Test6998541.java
Passed: java/lang/invoke/7157574/Test7157574.java
Passed: java/lang/invoke/7196190/ClassForNameTest.java
FAILED: java/lang/invoke/7196190/GetUnsafeTest.java
Passed: java/lang/invoke/7196190/MHProxyTest.java
Passed: java/lang/invoke/lambda/LambdaAccessControlDoPrivilegedTest.java
Passed: java/lang/invoke/lambda/LambdaAccessControlTest.java
Passed: java/lang/invoke/AccessControlTest.java
Passed: java/lang/invoke/BigArityTest.java
Passed: java/lang/invoke/CallSiteTest.java
Passed: java/lang/invoke/ClassValueTest.java
Passed: java/lang/invoke/InvokeDynamicPrintArgs.java
Passed: java/lang/invoke/InvokeGenericTest.java
Passed: java/lang/invoke/JavaDocExamplesTest.java
Passed: java/lang/invoke/MethodHandlesTest.java
Passed: java/lang/invoke/MethodTypeTest.java
Passed: java/lang/invoke/PermuteArgsTest.java
Passed: java/lang/invoke/PrivateInvokeTest.java
Passed: java/lang/invoke/RicochetTest.java
Passed: java/lang/invoke/ThrowExceptionsTest.java
Test results: passed: 21; failed: 1

The changes aren't pushed to git yet, as JamVM will currently only work with OpenJDK 8. Getting it to work with OpenJDK 6/7 (without JSR 292) won't be difficult, but GNU Classpath will be more tricky.

JamVM no claim to notability?

Fri, 30 Dec 2011 01:27:14 GMT

JamVM has a wikipedia page. I didn't create it, and I'm not egotistical enough to maintain it in any way. However, I was less than impressed to see that somebody had taken it upon themselves to put the page forward for deletion. The reasons being that it hasn't had a recent release, and that it has no claim to notability.

I have tried to show that neither of these claims are true. For example, JamVM is the default VM on Ubuntu/ARM 11.10. I think this is both notable and recent! However, this doesn't seem to count, the debate being fixated on a claim on the page regarding Dalvik from a blog.

To be honest, I'm so disgusted with the process that I no longer care if the page is deleted. But if anybody else cares, please put a word in for JamVM.

OpenJDK/JamVM Git repository

Wed, 16 Feb 2011 03:08:54 GMT

JamVM's got a shiny new Git repository, which contains the port to the OpenJDK class-library. Details are here: http://developer.berlios.de/git/?group_id=6545

The repository can be checked out anonymously with:

git clone git://git.berlios.de/jamvm

JamVM 1.6.0 will be released off of this in the near future. JamVM 1.6.0 will be a combined release, supporting both GNU Classpath and OpenJDK class-libraries, with GNU Classpath support being built by default. I still need to run some tests to make sure that the refactored codebase hasn't introduced any regressions w.r.t. GNU Classpath and JamVM 1.5.4.

So what can be done with the OpenJDK port? As discussed in my FOSDEM talk, it is ready for a first release. There's stuff which hasn't been implemented, but it runs everything I've tested it with (jedit, eclipse, derby, etc.).

Some words on running the port. JamVM is not yet integrated into the IcedTea build process (although it supports the same --with-java-runtime-library switch as Cacao). Instead, the easiest way to test the port is to build JamVM, and copy the libjvm.so file into an existing IcedTea/OpenJDK installation.

After cloning the git repository, do:

./autogen.sh --with-java-runtime-library=openjdk

This will generate the autoconf/automake files and configure JamVM to build support for OpenJDK.

Then do make, make install as usual. This will put libjvm.so into /usr/local/jamvm/lib.

This can then be copied onto an existing IcedTea installation (or a copy of one), e.g. on x86_64 (as root):

cd /usr/lib/jvm
cp -r java-6-openjdk jamvm-openjdk
cp /usr/local/jamvm/lib/libjvm.so jamvm-openjdk/jre/lib/amd64/server

You can then run it by running the normal java wrapper, e.g.:

/usr/lib/jvm/jamvm-openjdk/jre/bin/java -version

java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9.5) (6b20-1.9.5-0ubuntu1)
JamVM (build 1.6.0-devel, inline-threaded interpreter)

(the inline-threaded interpreter is the other name for the code-copying JIT)

Debian Linux on cheap MIPS mini netbook

Wed, 4 Aug 2010 00:08:39 GMT

Computing-wise, I've taken a break from the JamVM/OpenJDK port for a couple of days while I play with my latest toy : a cheap mini-netbook based on a Chinese MIPS clone. It's branded CnMbook, but it's available (or was) under dozens of names.

Yes, it's been available for a while. I first investigated it over a year ago as I wanted a MIPS machine on which I could do the port of the code-copying JIT (I did the MIPS port of JamVM back in 2007, and I've not touched it since). But it was 170 pounds, which was too much for the tiny spec, and I got an EeePC instead.

However, I still don't like Intel, even though it finally became my main architecture in 2008 (albeit AMD). And I like it even less as a mobile processor, as it's the last area where it doesn't dominate.

So what lead to this sudden surge of interest? I've been waiting for years for the fabled ARM-based netbooks, and a couple of weeks ago I saw a cheap WinCE mini-netbook in a local discount shop which I was passing. My brother (the author of SquashFS) thought it shouldn't be too difficult to get Linux working on it. This lead to a weekend long investigation, which showed up some startling results.

The mini-netbooks appear to be based on one of two ARM9-based SoCs (so these still aren't the low-cost Cortex A8 ones I've been waiting for). Either the Anyka AK7802 or Via's WonderMedia VT8505 SoC. The amazing thing? There's no publicly available Linux kernel (with source) for any of them. There is a binary-only Android kernel for the VT8505 but no public source. For the AK7802 there's nothing, and as Anyka have not made any SoC documentation publicly available it's likely that there never will be.

In contrast, Ingenic, the maker of the MIPS clone in the CnMbook have made available the full source to their modified kernel, u-boot bootloader, rootfs and applications. It came with CELinux, and an ancient 2.4.20 kernel. But it didn't take my brother long to get a modern-ish 2.6.24 kernel running on it (he's working on 2.6.31), and with Debian MIPS Linux it makes a nice little portable development system. Compiling is rather slow, but vim runs well on console, which is all you need!

The cost? 65 quid off ebay for an ex-display model as you can't buy them anymore, the ARM9 WinCE machines having completely replaced them.

I'll next post my experience of running JamVM/OpenJDK on it :)

... a thousand words (JamVM/OpenJDK update 2)

Tue, 27 Jul 2010 02:08:11 GMT

Firstly apologies to the people who commented on my first progress update (18th May). I'd hoped to do a blog update way before now, but I've had a lot less time to work on JamVM/OpenJDK port than I expected...

Xerces Ranby asked:

Does JamVM still produce those quick and fast startup times when using the OpenJDK class libraries compared to the fast startup times obtainable when using GNU Classpath classes?

[best of 3 runs]

rob@traken:~/JAM/tests$ time /usr/lib/jvm/jamvm-openjdk/jre/bin/java -showversion hello
java version "1.6.0_0"
OpenJDK Runtime Environment (IcedTea6 1.6.1) (6b16-1.6.1-3ubuntu1)
JamVM (build 1.5.5-openjdk, inline-threaded interpreter)

Hello World!

real 0m0.046s
user 0m0.030s
sys 0m0.000s

rob@traken:~/JAM/tests$ time jamvm -showversion hello
java version "1.5.0"
JamVM version 1.5.5-devel
Copyright (C) 2003-2010 Robert Lougher
...
Build information:
Execution Engine: inline-threaded interpreter
Compiled with: gcc 4.5.0 20100211 (experimental)
Boot Library Path: /usr/local/classpath/lib/classpath
Boot Class Path: /usr/local/jamvm/share/jamvm/classes.zip:/usr/local/classpath/share/classpath/glibj.zip

Hello World!

real 0m0.048s
user 0m0.030s
sys 0m0.020s

gnu_andrew asked:

Will JamVM still support GNU Classpath, as CACAO does?

Yes, most definitely. I still consider GNU Classpath as JamVM's main class-library as it's where my chief loyalty lies. On more practical grounds, even after the OpenJDK port is functionally complete it will still be a long time before it is as tested as JamVM/GNU Classpath. FWIW, many embedded systems seem to be quite happy with GNU Classpath. GNU Classpath is considerably smaller "out of the box" and much easier to build...

As far as development is concerned, I've taken a different approach to Cacao. Cacao implements the class-library differences within the VM-specific code using #ifdefs. While there's nothing wrong with that, I personally think it makes the code harder to read, and it's harder to get an overview of the changes.

Instead, I've tried to abstract the differences into a classlib interface. At times this has taken some thought and quite a lot of code re-arranging. If anything it's made the code cleaner, as a lot of the messier details are hidden (in general, I'm not a fan of information hiding, but removal of some of the VMFoo details makes the intent clearer).

Having said that, the classlib interface is mostly driven by the differences between GNU Classpath and OpenJDK as I find them. I'd like to think the interface is reasonably generic, but it will probably need changing if another class-library came along...

Currently the classlib interface has 40 functions, the gnuclasspath directory has 8 files, totalling 2552 LOC, and openjdk 10 files totalling 3491 LOC.

Christian Thalinger (hello, twisti!), linuxhippy (hello, Clemens!), Michael Starzinger (hello, Michi!), Stefan Ring:

Yeah, it's taken a long time, and lots of prevaricating. It's not been quite as boring and tedious as I expected; some of it I've actually enjoyed :) Debian might even re-instate JamVM (sticking pins in the Debian T-shirt I bought at FOSDEM, while looking for the Fedora 13 CDROM).

A picture is worth...

Tue, 27 Jul 2010 00:07:41 GMT

JamVM and OpenJDK - Progess Update

Tue, 18 May 2010 00:10:09 GMT

Porting JamVM to use the OpenJDK/IcedTea class-library has been on my TODO list for quite a while. It appears on my FOSDEM slides for 2008 and again in 2009. It doesn't this year, as I didn't do a talk. Maybe that's why I've finally got round to doing it. I told Twisti I'd get "Hello World" running in 3 months. I've missed by just over a week, mainly because I only started 2 months ago.

root@traken:/usr/local/jamvm/bin# java -Xbootclasspath:/usr/lib/jvm/java-6-openjdk/jre/lib/rt.jar -Dsun.boot.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/amd64 -showversion hello
java version "1.6.0_0"
OpenJDK Runtime Environment (IcedTea6 1.6.1) (6b16-1.6.1-3ubuntu1)
JamVM (build 1.5.5-openjdk, inline-threaded interpreter)

Hello World!
root@traken:/usr/local/jamvm/bin#

There's still quite a lot to do, including finishing off support for reflection and sun.misc.Unsafe but threading, exception handling, library handling, class-loading and the VM bootstrap sequence should be done.

Filesystem Collaboration

Sat, 18 Jul 2009 00:06:18 GMT

My twin brother is the author of SquashFS. A very influential Linux filesystem used on practically all Live CDs. He reads the Linux Magazine interview with Valerie Aurora who says:

hoping that if the existing Linux file systems developers could just talk to each other, we could come up with a way to improve funding for Linux file systems development. I don’t know that it helped, but if it did, I consider this to be my most important contribution to Linux file systems.

So decides to link up via LinkedIn, and gets refused...

Netifera uses JamVM as remote probe

Tue, 12 May 2009 02:08:15 GMT

Netifera is a very interesting 100% Java-based Open Source network security platform. The next version will include a probe, which is a "deployable software agent that makes it possible to run all netifera platform tools remotely".

The probe uses JamVM and a cut-down version of GNU Classpath. This is downloaded onto the remote site to run the netifera tools. My thanks to Claudio Castiglia (of Netifera) for sending me the following link to a webcast showing the probe in action:

http://netifera.com/video/netifera_java_virtual_machine_as_shellcode

JamVM and GNU Classpath are mentioned starting 14:54 into the webcast in the discussion of the probe architecture.

JamVM on Beagle Board

Tue, 12 May 2009 01:09:42 GMT

I've been intending to get a Beagle Board for quite a while now, to replace a Neo1973 as the development platform for JamVM on ARM. The Neo has an ARM920T core, and I've been particularly wanting to test JamVMs code-copying JIT on a more modern implementation (the Beagle Board has a Cortex A8). But I've been waiting for the Rev C board, as this has working EHCI USB host, and 256MB RAM (previous revisions had 128MB). The board arrived 2 weeks ago, but I haven't had much time to set it up until now...

So far, the results are quite encouraging. SciMark 2.0 shows a 55% speed improvement:

While jBYTEmark shows an 82% improvement on Integer tests:

The option -Xcodestats prints out the size of the JIT code-cache when the VM exits. This shows 77K was used after running SciMark 2.0, and 178K after jBYTEmark. This is using the code-profiling introduced in JamVM 1.5.2.