NQP on JVM bootstrapped, soon will land in NQP master

The work to get NQP running and bootstrapped on the JVM has reached an interesting milestone, and I thought I’d take a few moments from working on it to talk about what has taken place since my last post here, as well as taking a look at what will be coming next.

Getting NQP on JVM Bootstrapped

When I last posted here, I had reached the point of having an NQP cross-compiler to the JVM that covered quite a lot of NQP’s language features. Cross-compilation meant using NQP on Parrot to parse NQP source, build an AST, and then turn the AST into a kind of JVM assembly language. This in turn was then transformed into a JVM bytecode file by a small tool written in Java, which could then be executed on the JVM.

Since the last post, the cross-compiler became capable of cross-compiling NQP itself. This meant taking the NQP source files and using the cross-compiler to produce an NQP that would run on the JVM – without depending on Parrot. This also enabled support for eval-like scenarios. I reached this stage last month; the work involved tracking down a range of bugs, implementing some missing features, and doing a little more work to improve NQP’s portability. So, we had an NQP that ran on the JVM. Port done? Well, not quite.

A little vacation later, I dug into the next stage: making NQP on the JVM able to compile (and thus reproduce) itself. While I’d already implemented the deserialization used to persist meta-objects (representing classes, grammars and so forth), for this next step I had to implement the serialization side of things. Thankfully, there are a bunch of tests for this, so this was largely “just” a matter of working through making them pass. Finally, it was time to work through the last few problems, and get NQP on JVM able to build the NQP sources – and therefore able to compile new versions of itself. I decided to do this work as part of moving the JVM support into the main NQP repository.

Since we’ve only had NQP running on one backend (Parrot) up until now, certain aspects of the repository structure were not ideal. Before starting to bring in the JVM support, I first did a little bit of reorganization to segregate the VM specific components from the VM independent ones. Happily, much of NQP’s implementation falls into the latter category. Next came gradually building up the bootstrapping build process, working a file at a time, tracking down any issues that came up. This was a little tedious, especially given a couple of the problems were separate compilation leakages (where things from the running compiler would get confused with the version of the compiler that it was compiling). It was pretty clear that this was the problem from the errors I was seeing, but such problems show up long after things actually go wrong, requiring some careful analysis to hunt down. With those leaks plugged, and a few other relatively small bugs fixed, I had a working NQP on JVM…compiled by NQP on JVM.

The work from there has been to fill out the rest of the build process, adding in the second bootstrap stage and the test targets. The good news: the NQP produced by NQP on JVM passes all the tests that the original cross-compiled version did, so we’ve got no regressions there as a result of the bootstrap.

This work is currently sat in the jvm-support branch of the NQP repository. After the upcoming NQP release, it will be merged.

Supporting invokedynamic

Amongst all of this progress, we’ve also gained infrastructure to support using the invokedynamic instruction. This is a mechanism that enables those implementing non-Java languages on the JVM to teach its JIT about how their dispatch works. Most of the hard work here was done by donaldh++. I’d initially built things using BCEL in order to do code generation. While it served well up to a point, it turns out that ASM has much better support for invokedynamic, as well as being a little faster. So, donaldh got things switched over, and I soon was able to emit invokedynamic.

So far, we are not using it a great deal (just for making QAST::WVal compile to something a bit – or potentially a lot – cheaper), but in the future it will be used for things you’d typically think of as invocations (sub and method calls). I’ll write in more detail about it as things evolve.

What next?

With NQP now ported, the focus will fall on Rakudo itself. Quite a lot of preparations have already been made; for example, many pir:: ops have been replaced with nqp:: ones, multiple dispatch has been ported to NQP from C (fixing some bugs along the way), the way HLL boundaries work has been updated to cope with a fully-6model world (this also let me fix a long-standing introspection bug).

The path through getting Rakudo ported will largely follow the build order. This means starting with the module loader, then the compiler, followed by the MOP and its bootstrapping. After that comes the setting – the place where the built-ins live. There’s around 13,650 lines worth of that, so of course I expect to take it a little at a time. :-) I’ll try to remember to get a progress update here in a couple of weeks time.

This entry was posted in Uncategorized. Bookmark the permalink.

6 Responses to NQP on JVM bootstrapped, soon will land in NQP master

  1. nigehamilton says:

    Great work! Inspiring and very exciting to see how things are progressing. ;-)

  2. Matt Oates says:

    I’ve been waiting with baited breath for this update. Hard to believe you’ve managed to come this far since around November?! Amazing work, great to see its unearthing bugs and improving things as a whole not just for Java! Thanks for putting so much passion and effort into the project, people on the outside/periphery really do appreciate it. Out of interest how easy/hard would Java interop be? Do NQP and later Rakudo have completely different ideas compared to Java on what an object is, in a memory mapped sense? Thanks again for all the hard work!

  3. Jed Harris says:

    What performance are you seeing for NQP on JVM?
    InvokeDynamic (people are calling it Indy) looks like an excellent extension of the JVM design for Perl 6’s exotic needs — lazy lists, sparse arrays, junctions, etc. Eager to see the ways that can be used.

  4. grondilu says:

    I suspect a JVM port would be a great news, at least in terms of performance, but what I’m most curious about is: does this mean we could call all the external libraries that have been ported to JAVA, from perl6? I don’t dare imagining it.

  5. Brandon says:

    Very exciting to see this coming along.

  6. TimBunce says:

    Delighted. Keep up the good work.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.