YAPC::EU 2013 Slides

I’m just back from this year’s YAPC Europe in Kiev. I’ve liked Kiev since the first time I visited many years ago, and after around two years since the last visit, I was glad of an excuse to return. It was the same beautiful, rather hilly city I remembered, though decidedly warmer than I remember it – probably because this is the first visit I made there in summer. In the company of the many good Perl folks attending YAPC, I enjoyed plenty of nice food, and even caught some good Belgian beers thanks to the wonderful Belgian Beer Cafe.

This year I submitted three talks, expecting one or two would be accepted. Instead, all three were! So, I talked about:

  • Rakudo on JVM – this session explained the motivation for adding a JVM backend, the compiler architecture that enabled it, how it was implemented, the current status, the support so far for Java interoperability and the plan from here. Seems to have been well received.
  • Concurrency, Parallelism and Asynchrony – this session showed the work I have been doing to build basic support for parallel, asynchronous and concurrent programming in Perl 6. This was the best attended and also, I believe, the most discussed of my talks this year. There is still much work to do in this area, but what’s done so far caught some interest. Once I’m recovered from YAPC, I’ll dig back into it.
  • MoarVM – this session talked about the motivation for building a new VM, its overall design, the current status and what’s in store. It’s also the first talk I ever gave on MoarVM. The most lightly attended and most hastily prepared, but still it seemed to be appreciated by those who attended.

Enjoy the slides, and hopefully the videos will make it online soon too.

I also agreed to attend this years Austrian Perl Workshop, where the hills will be alive with the sound of Perl 6 in the lovely Salzburg, sometime in November. :-)

Posted in Uncategorized | 1 Comment

Rakudo JVM News: More tests, plus Thread and Promise prototypes

Last time I wrote, the Rakudo on JVM port was passing around 92% of the specification tests that Rakudo on Parrot can. In the last couple of weeks, we’ve continued hunting down and fixing failures. I’m happy to report that we have already passed the 98% mark – well beyond the 95% I was aiming for by the July release! I’m optimistic that we may be able to push that up to 99% in the coming days. Either way, we’re closing in on the goal spectest wise, meaning the focus should soon move to getting tools like Panda working, followed by the module ecosystem. Happily, arnsholt++ has already started working on the NativeCall support that many modules depend on.

One of the reasons for adding a JVM backend is to unblock work on Rakudo’s support for asynchronous, parallel and concurrent programming. With a YAPC::EU talk on these topics looming, and hating to take to the stage without anything to demonstrate, I’ve started working on this area. It’s early days yet, but here is a quick look at what’s already possible.

There is some basic support for doing stuff in threads.

say "Creating a couple of threads...";

my $t1 = Thread.start({ sleep 1; say "Thread 1 done"; });
my $t2 = Thread.start({ sleep 2; say "Thread 2 done"; });

say "Waiting for joins...";
.join for $t1, $t2;
say "Joined!";

This does what you’d naturally expect. However, threads are kind of like the assembly language of parallel programming: occasionally you want to work at that level, but usually it’s much better to work in terms of higher level constructs. Thus, while you can do the above, I don’t suggest it. So what’s available at a higher level? Well, so far, promises are.

say "Creating two promises...";

my $a = async { sleep 2; say "omg slept 2"; 27 }
my $b = async { sleep 1; say "omg slept 1"; 15 }

say "Scheduler has $*SCHEDULER.outstanding() tasks";
say "Waiting for results...";
say $a.result + $b.result;

The async construct evaluates to a Promise object (name subject to change; Future or Task are other options we could steal from other languages). A Promise is an object that represents a piece of ongoing work. It is not backed by a thread of its own; instead, it is scheduled onto a pool of threads that are spun up on demand, up to a limit. Alternatively, a Promise could be backed by some kind of asynchronous I/O. The point is that it doesn’t much matter what the exact nature of the work is, just that there’s a common way to talk about concurrent work and write combinators over them.

When you ask for the result of a Promise and it is not available, you will block until it is available. If the inside of the async block died, then the exception will be thrown at the point the result method is called. There is also a method “then” for chaining on extra work to be done once the promise is either completed or fails due to an exception:

my $a2 = $a.then(-> $res { say "Got $res.result() from promise a" });

This returns another Promise, thus allowing chaining. There is also a sub await that for now just calls the result method for you, on a whole list of promises if you pass them. Here’s an example:

say [+] await dir('docs/announce').map({
    async { .IO.lines.elems }

This creates a Promise per file in the directory that will count the number of lines in the file. Then, await will wait for each of the promises to give a result, handing them back as they come in to the reduction. Note that in the future, this could probably just be:

say [+] hyper dir('docs/announce').map(*.IO.lines.elems)

But we didn’t implement that yet, and when it does happen then it will most likely not work in terms of simply creating a promise per element.

Remember that promises are much lighter than threads! We’re not spinning up dozens of threads to do the work above, just spreading the load over various threads. And yes, the parallel version does run faster on my dual core laptop than the one that isn’t using async/await.

Future plans for promises include combinators to wait for any or all of them, an API for backing them with things other than work in the thread pool, and making await a bit smarter so that it can suspend an ongoing piece of work in the thread pool when it blocks on another promise, thus freeing up that thread for other work.

Of course, all of this is early and experimental; any/all of the above can change yet, and it’s a long, long way from being exercised to the degree that many other parts of Rakudo have been. Expect changes, and expect many more things to land in the coming months; on my list I have asynchronous I/O and an actors module, and I know pmichaud++ has been thinking a bit about how to evolve the list model to support both race and hyper.

Anyway, that’s the latest news. Next time, hopefully there will be yet more spectest progress and some nice sugar for sorear++’s ground work on Java interop, which is what I used to build the threads/promises implementation.

Posted in Uncategorized | 9 Comments

Rakudo on JVM Progress Update

In the spirit of “release early, release often”, last month’s Rakudo compiler release was the first one to come with very basic support for running on the JVM. It supported a small number of the features that Rakudo on Parrot does. Of note, it could pass the sanity tests, a small test suite we keep in the Rakudo repository. The sanity tests ensure that the compiler is functional enough to compile Test.pm, which is a prerequisite for running the specification tests. Essentially, they make sure that we can crawl our way to the starting line, where the real race begins. :-)

So, since that release, we’ve been working hard on getting Rakudo on JVM able to pass more of the specification test suite, gradually increasing its capabilities, hunting bugs, adding missing bits of infrastructure and guts code in order to get features working, and so forth. Progress has been rather swift, and today’s automated test run by Coke++ shows that Rakudo on JVM is now passing 92% of the tests that Rakudo on Parrot does!

So what does this number really mean? It means that Rakudo passes 92% of the individual test cases on the JVM that it does on Parrot. To put that into perspective, that’s more tests than either of Niecza (82%) or Pugs (36%), meaning that Rakudo on JVM is now the second most spectest-passing Perl 6 implementation. Not bad, to say around 8 months ago not a single line of implementation code related to Rakudo JVM targeting had been written.

However, the raw number should be taken with a pinch of salt. Here’s why. The test suite is not especially uniform in terms of tests it dedicates to each feature. For example, there are hundreds of passing tests dedicated to providing good coverage of Unicode character property matching in regexes, and trigonometry easily has many hundreds too. By contrast, the slurp function has more like dozens of tests – since you can comfortably cover it that way. However, these are failing still, and I’m pretty sure that more Perl 6 users depend on slurp that Unicode character properties and trig.

Anyway, my aim is that we’ll be some way past the 95% mark in time for the July compiler release. And yes, I’ll make sure slurp gets fixed before then! :-) I suspect the last couple of percent will be the most tedious, but it feels good to be reaching towards them.

In the coming weeks, I expect the focus to start shifting from the compiler to the ecosystem: getting Panda working with Rakudo on JVM, and starting to run the module tests and work through the issues we find there. Calling Java libraries and work on parallelism are also planned, not to mention digging in to optimization work, since this initial phase has very much been focused on “get things working”. All in all, plenty to be working on, and plenty to look forward to.

Finally, a week or so ago I spent a couple of hours being interviewed by Nikos Vaggalis for Josette Garcia’s blog. It was a very wide-ranging interview and I managed to spout sufficiently in response to the questions that it’s being published over three posts in the coming weeks. You can find the first part of the interview here; enjoy!

Posted in Uncategorized | 3 Comments

MoarVM: A virtual machine for NQP and Rakudo

Over the course of the last year, we’ve been working to make both NQP and Rakudo more portable. This has primarily been done to enable the JVM porting work. While the JVM is an important platform to target, and initial work suggests it can give us a faster and more stable Rakudo, there are some use cases, or users, that the JVM will not cater to so well. Startup time will likely remain a little high for one-liners and quick scripts. Other potential users simply don’t want to use the JVM, for whatever reason. That’s fine: there’s more than one way to do it, and our strategy has been to enable that by adding support for the JVM without breaking support for Parrot. Additionally, pmurias will be working on a JavaScript backend for a GSoC project.

Today I’d like to introduce some work that, all being well, will lead to an additional “way to do it” arriving over the next several months. A small team, composed of myself, diakopter, japhb and masak, have been quietly working on taking the design of the 6model object system and building a runtime around it. Thus, we’ve created the “Metamodel On A Runtime” Virtual Machine, or the “MOAR VM”, which we’ve taken to writing as “MoarVM”.

This is not a release announcement. At present, MoarVM runs neither NQP nor Rakudo, though a cross-compiler exists that allows translating and passing much of the NQP test suite. We’re revealing it ahead of YAPC::NA, so it can be openly discussed by the Perl 6 team. The goal from the start has been to run NQP, then run Rakudo. The JVM porting work has established the set of steps that such an effort takes, namely:

  1. Build an NQP cross-compiler that targets the desired platform. Make it good enough to compile the MOP, built-ins and the classes at the heart of the regex/grammar engine. Make it pass most of the NQP test suite.
  2. Make the cross-compiler good enough to cross-compile NQP itself.
  3. Close the bootstrap loop, making NQP self host on the new platform.
  4. Do the Rakudo port.

At the moment, the JVM work is well into the final step. For MoarVM, we’ve reached the second step. That is to say, we already have a cross-compiler that compiles a substantial range of NQP language constructs into MoarVM bytecode, including the MOP, built-ins and regex-related classes. Around 51 of the NQP test files (out of a total of 62) pass. Work towards getting the rest of NQP to cross-compile is ongoing.

Since anybody who has read this far into the post probably has already got a whole bunch of questions, I’m going to write the rest of it in a question-and-answer style.

What are the main goals?

To create a VM backend for Rakudo Perl 6 that:

  • Is lightweight and focused on doing exactly what Rakudo needs, without any prior technical or domain debt to pay off.
  • Supports 6model and various other needs natively and, hopefully, efficiently.
  • Is a quick and easy build, with few dependencies. I was rather impressed with how quick LuaJIT can be built, and took that as an inspiration.
  • Enable the near-term exploration of JIT compilation in 6model (exploring this through invokedynamic on the JVM is already underway too).

What’s on the inside?

So far, MoarVM has:

  • An implementation of 6model. In fact, the VM uses 6model as its core object system. Even strings, arrays and hashes are really 6model objects (which in reality means we have representations for arrays and hashes, which can be re-used by high-level types). This is the first time 6model has been built up from scratch without re-using existing VM data structures.
  • Enough in place to support a sizable subset of the nqp:: op space. The tests from the NQP test suite that can be translated by the cross-compiler cover a relatively diverse range of features: the boring easy stuff (variables, conditionals, loops, subs), OO stuff (classes, methods, roles, mixins, and, naturally, meta-programming), multiple dispatch, most of grammars (including LTM), and various other bits.
  • Unicode strings, designed with future NFG support in mind. The VM includes the Unicode Character Database, meaning that character name and property lookups, case changing and so forth can be supported without any external dependencies. Encoding of strings takes place only at the point of I/O or when a Buf rather than a Str is requested; the rest of the time, strings are opaque (we’re working towards NFG and also ropes).
  • Precise, generational GC. The nursery is managed through semi-space copying, with objects that are seen a second time in the nursery being promoted to a second generation, which is broken up into sized heaps. Allocations in the nursery are thus “bump the pointer”, the copying dealing with the resulting fragmentation.
  • Bytecode assembly done from an AST, not from a textual format. MoarVM has no textual assembly language or intermediate language. Of course, there’s a way to dump bytecode to something human-readable for debugging, but nothing to go in the opposite direction. This saves us from producing text, only to parse it to produce bytecode.
  • IO and other platform stuff provided by the Apache Portable Runtime, big integer support provided by libtommath, and re-use of existing atomic ops and hash implementations. We will likely replace the APR with libuv in the future. The general principle is to re-use things that we’re unlikely to be able to recreate ourselves to the same level of quality or on an acceptable time scale, enabling us to focus on the core domain.

What does this mean for the Rakudo on JVM work?

Relatively little. Being on the JVM is an important goal in its own right. The JVM has a great number of things in its favor: it’s a mature, well-optimized, widely deployed technology, and in some organizations the platform of choice (“use what you like, so long as it runs on the JVM”). No matter how well Moar turns out, the JVM still has a couple of decades head start.

Additionally, a working NQP on JVM implementation and a fledgling Rakudo on JVM already exist. Work on making Rakudo run, then run fast, on the JVM will continue at the same kind of pace. After all, it’s already been taking place concurrently with building MoarVM. :-)

What does this mean for Rakudo on Parrot?

In the short term, until MoarVM runs Rakudo well, this shouldn’t really impact Rakudo on Parrot. Beyond that is a more interesting question. The JVM is widely deployed and battle-hardened, and so is interesting in its own right whatever else Rakudo runs on. That’s still not the case for Parrot. Provided MoarVM gets to a point where it runs Rakudo more efficiently and is at least as stable and feature complete, it’s fairly likely to end up as a more popular choice of backend. There are no plans to break support for Rakudo on Parrot.

Why do the initial work in private?

There were a bunch of reasons for doing so. First and foremost, because it was hard to gauge how long it would take to get anywhere interesting, if indeed it did. As such, it didn’t seem constructive to raise expectations or discourage work on other paths that may have led somewhere better, sooner. Secondly, this had to be done on a fairly limited time budget. I certainly didn’t have time for every bit of the design to be bikeshedded and rehashed 10 times, which is most certainly a risk when design is done in a very public way. Good designs often come from a single designer. For better or worse, MoarVM gets me.

Why not keep it private until point X?

The most immediate reason for making this work public now is because a large number of Perl 6 core developers will be at YAPC::NA, and I want us to be able to openly discuss MoarVM as part of the larger discussions and planning with regard to Perl 6, NQP and Rakudo. It’s not in any sense “ready” for use in the real world yet. The benefits of the work being publicly known just hit the point of outweighing the costs.

What’s the rough timeline?

My current aim is to have the MoarVM backend supported in NQP by the July or August release of NQP, with Rakudo support to come in the Rakudo compiler release in August or September. A “Star” distribution release, with modules and tools, would come after that. For now, the NQP cross-compiler lives in the MoarVM repository.

After we get Rakudo running and stabilized on MoarVM, the focus will move towards 6model-aware JIT compilation, improving the stability of the threads implementation (the parallel GC exists, but needs some love still), asynchronous IO and full NFG string/rope support.

We’ll have a bunch of the right people in the same room at YAPC::NA, so we’ll work on trying to get a more concrete roadmap together there.

Where is…

Posted in Uncategorized | 5 Comments

Rakudo on JVM progress update, and some questions answered

It’s time for another progress update on the ongoing JVM work. Last time I posted here, we’d reached the point of having a self-hosting NQP ready to merge into the master branch of the NQP repository. That has now happened, so the May release of NQP will come with support for running on the JVM (note, this does not mean the May release of Rakudo will come with this level of capability, and a JVM-based Star release with modules is further still!) . In this post, I will discuss some of the things that have happened in the last few weeks and also try to answer some of the questions left behind in the comments last time.

The Rakudo Port

With NQP pretty well ported (there are some loose ends to tie up, but it’s pretty capable), the currently ongoing step is porting Rakudo. At a high level, Rakudo breaks down into:

  • The core of the compiler itself: the grammar (for parsing), actions (which assign semantics to the things we parsed), world (which takes care of the declarative aspects of programs), optimizer (tries to cheat without getting caught) and a few other small pieces to support all of this. This is written in NQP.
  • The Perl 6 MOP (meta-object protocol) implementation, which defines what classes, roles, enums, subsets and so forth mean. This is also written in NQP.
  • The bootstrap, which uses the MOP to piece together various of the core Perl 6 types. It does Just Enough to let us start writing Perl 6 code to define the rest of the built-ins. Also written in NQP.
  • The core setting, which is where the built-in types, operators and methods live. This is written in Perl 6.
  • A chunk of per-VM code that does lower-level or performance-sensitive things.

The first 3 of these need…an NQP compiler. And, it turns out that we have one of those hosted on the JVM these days. But could it handle compiling the Perl 6 grammar? It turns out that, after some portability improvements to the various bits of Perl 6 compiler code, the answer was a resounding “yes”. In fact, the grammar was compiled down to JVM bytecode sufficiently accurately that I didn’t encounter a single parse failure on the entire CORE.setting (though there were lots of other kinds of failures that took a lot of work – more on that in a moment). As for the rest of the compiler code, there were bits of lingering Parrot-specific stuff throughout it, but with a little work they were abstracted away. By far the hardest was BOOTSTRAP, which actually runs a huge BEGIN block to do its setup work and then pokes the results into the EXPORT package. This is kinda neat, as it means the setup work is done once when building Rakudo and then serialized. Anyway, onto the next pieces.

Compiling the Perl 6 setting depends on the Perl 6 compiler working. Since the first thing the setting does is use the bootstrap, which in turn uses the MOP, it immediately brings all of the above three pieces together. While we talk about “compiling” the setting, there’s a little more to it than that. Thanks to various BEGIN time constructs – such as traits, constant declarations and, of course, BEGIN blocks – all of which show up in the CORE setting – we actually need to run chunks of the code during the compilation process. That’s right – we run bits of the file we’re in the middle of compiling while we’re compiling it. Of course, this will be nothing new to Perl folks – it’s just most Perl programmers probably don’t worry about how on earth you implement this. :-) Thankfully, it’s a solved problem in the NQP compiler toolchain, and the stuff that makes NQP BEGIN blocks work largely handles the Rakudo ones too.

Anyway, all of this means that even getting the setting to finish the parse/AST phase requires doing enough to run the various traits and so forth. And that in turn brings in the fifth piece: the per-VM runtime support. This includes signature binding, container handling and a few other bits. Thankfully, it no longer involves multiple dispatch, since that is written in NQP these days (apart from some caching infrastructure, which is shared with NQP’s multiple dispatch, and thus was already done). Getting through the parse/AST phase of the setting doesn’t need all of the runtime support to be implemented, but it did require a decent chunk of it. Of course, at the start everything is missing, so getting from line 1 to line 100 was O(week), from 100 to 1000 O(day) and each thousand or so from there O(hour). It’s 13,000 or so lines in all.

The parse/AST step is just the first (though biggest) phase of compiling Perl 6 code, however. Next comes the optimizer, followed by code generation. In theory, the optimizer could have been bypassed. I planned to do that, then discovered it basically worked for the set of optimizations that didn’t need signature binding to participate in the analysis, so I left it in. Code generation is part of the backend, and so is shared with NQP. So it shoulda just worked, right? Well, yes, apart from code generation is the place where nqp:: ops get resolved to lower level stuff. And Perl 6 uses a lot more of them than NQP. Note that they only have to be mapped to somewhere; the JVM is late bound enough that it won’t complain unless you actually hit the code path that tries to use something that is not yet implemented. In reality I did a bit of both: implementing those that would surely be hit soon or that were trivial, and leaving some others for later.

So, some time yesterday, I finally got to the point of having a CORE.setting.class. Yup, a JVM bytecode file containing the compiled output of near enough the entire Perl 6 core setting. So, are we done yet? Oh, no way…

Today’s task was trying to get the CORE setting to load. How hard could that be? Well, it turns out that it does a few tasks at startup, most of which hit upon something that wasn’t in place yet. Typically, it was more of the runtime support, though in some cases it was unimplemented nqp:: ops. Of course, there were a handful of bugs to hunt down in various places to.

Anyway, finally, this evening, I managed to get the CORE setting to load, at which point, at long last, I could say:

perl6 -e "say 'hello world, from the jvm'"
hello world, from the jvm

Don’t get too excited just yet. It turns out that many other simple programs will happily explode, due to hitting something missing in the runtime support. There’s still plenty of work to go yet (to give you an idea, trying to say a number explodes, and a for loop hits something not yet implemented), but this is an important milestone.


A couple of the comments in response to my last post asked about interop with Java. There’s two directions to consider here: using Java libraries from Perl 6, and using Perl 6 code from Java. Both should be possible, with some marshalling cost, which we’ll no doubt need to spend some time figuring out how to get cheap enough it’s not a problem. It may well be that invokedynamic is a big help here.

The Java stuff from Perl 6 direction can probably be made fairly convenient to use by virtue of the fact that Perl 6 has a nice, extensible MOP. The fact the object you’re making a call on lives in Java land can be just a detail; we can hide it behind the typical method call syntax, and should even be able to populate a 6model method cache with delegation methods that do the argument mapping. I’m sure there will be plenty of interesting options there. I suspect we’ll want to factor it a little like NativeCall – some lower level stuff in the runtime, and some higher level sugar.

Going the other way will be more “fun”. I mean, on the one hand the marshalling is just “in the other direction”, which we’d need to do for values coming back from Java land anyway. But trying to work out how to make it feel nice from Java land could be trickier. I don’t believe the “.” operator is very programmable, which probably leaves us with string lookups or code-generated proxy thingies. Or maybe somebody will come up with a Really Great Solution that I hadn’t thought of.

My JVM related Perl 6 dev focus for now will be getting Rakudo to work decently and start getting Perl 6 modules working on the JVM also, but interop with Java land is certainly on the roadmap of things I think should happen. As with all things, I’m delighted to be beaten to it, but will work on it if it goes undone for too long. :-)


The first thing to say on this is that it’s too early to have a really good idea. The final pieces of the gather/take transform (which has global consequences) have yet to land, which will certainly have some negative impact and will need to happen soon. At the same time, I’ve been very much focused on making things work on the JVM at all over making them especially clever or optimal. Numerous things can be done in ways that will not only perform better in a naive sense, but that will also be much easier for the JVM’s JIT to do clever stuff with. There are many, many things we will be able to do in this area.

Since I only have a sort-of-working Perl 6 compiler, I can’t say that much about Perl 6 performance. The only result I have to share is that the CORE setting parse completes in around a third of time that it does on Parrot (noting it’s not only about parsing, but also some code generation and running of stuff). This is not especially great – of course, we need to do better than that – but it’s certainly nice that the starting point before I really dig into the performance work is already a good bit faster.

The other result I have is NQP related. nwc10++ has been doing performance testing with a Levenstein benchmark of each commit to the NQP JVM work, which is really great in so far as it gives me a rough idea if I accidentally regress (or improve ;-)) something performance wise. There, we saw a larger factor performance win, around 15 times faster than the same program running in NQP on Parrot.

The big negative news on performance is startup time. Part of this is just the nature of the JVM, but I know another enormous part of it is inefficiencies that I can do something about. I’ve plenty of ideas – but again, let’s make things work first.

From Here

Things will be a little quieter from me over the next week and a bit, due to a busy teaching schedule. But now we have a fledgling Rakudo on JVM, and from here it’s going to be making it gradually more capable, first passing the sanity tests, then moving on to the spectests and the ecosystem. There are ways to help for the adventurous. Some ideas:

  • Profile the code generation phase, which is one of the pieces that is slower than expected. Try to figure out why.
  • Have a look at how multiple dispatch stuff is currently set up, and see if the dispatch logic could possibly be shuffled off behind invokedynamic.
  • Try something. See how it explodes. See if you can fix it. (Yes, generic I know. :))
  • Have a look at a “make install” target for Rakudo on JVM.

I’ll be speaking on the JVM work at the Polish Perl Workshop the weekend after next, and hope to have something a bit more interesting than “hello world” to show off by then.

Posted in Uncategorized | 9 Comments

NQP on JVM bootstrapped, soon will land in NQP master

The work to get NQP running and bootstrapped on the JVM has reached an interesting milestone, and I thought I’d take a few moments from working on it to talk about what has taken place since my last post here, as well as taking a look at what will be coming next.

Getting NQP on JVM Bootstrapped

When I last posted here, I had reached the point of having an NQP cross-compiler to the JVM that covered quite a lot of NQP’s language features. Cross-compilation meant using NQP on Parrot to parse NQP source, build an AST, and then turn the AST into a kind of JVM assembly language. This in turn was then transformed into a JVM bytecode file by a small tool written in Java, which could then be executed on the JVM.

Since the last post, the cross-compiler became capable of cross-compiling NQP itself. This meant taking the NQP source files and using the cross-compiler to produce an NQP that would run on the JVM – without depending on Parrot. This also enabled support for eval-like scenarios. I reached this stage last month; the work involved tracking down a range of bugs, implementing some missing features, and doing a little more work to improve NQP’s portability. So, we had an NQP that ran on the JVM. Port done? Well, not quite.

A little vacation later, I dug into the next stage: making NQP on the JVM able to compile (and thus reproduce) itself. While I’d already implemented the deserialization used to persist meta-objects (representing classes, grammars and so forth), for this next step I had to implement the serialization side of things. Thankfully, there are a bunch of tests for this, so this was largely “just” a matter of working through making them pass. Finally, it was time to work through the last few problems, and get NQP on JVM able to build the NQP sources – and therefore able to compile new versions of itself. I decided to do this work as part of moving the JVM support into the main NQP repository.

Since we’ve only had NQP running on one backend (Parrot) up until now, certain aspects of the repository structure were not ideal. Before starting to bring in the JVM support, I first did a little bit of reorganization to segregate the VM specific components from the VM independent ones. Happily, much of NQP’s implementation falls into the latter category. Next came gradually building up the bootstrapping build process, working a file at a time, tracking down any issues that came up. This was a little tedious, especially given a couple of the problems were separate compilation leakages (where things from the running compiler would get confused with the version of the compiler that it was compiling). It was pretty clear that this was the problem from the errors I was seeing, but such problems show up long after things actually go wrong, requiring some careful analysis to hunt down. With those leaks plugged, and a few other relatively small bugs fixed, I had a working NQP on JVM…compiled by NQP on JVM.

The work from there has been to fill out the rest of the build process, adding in the second bootstrap stage and the test targets. The good news: the NQP produced by NQP on JVM passes all the tests that the original cross-compiled version did, so we’ve got no regressions there as a result of the bootstrap.

This work is currently sat in the jvm-support branch of the NQP repository. After the upcoming NQP release, it will be merged.

Supporting invokedynamic

Amongst all of this progress, we’ve also gained infrastructure to support using the invokedynamic instruction. This is a mechanism that enables those implementing non-Java languages on the JVM to teach its JIT about how their dispatch works. Most of the hard work here was done by donaldh++. I’d initially built things using BCEL in order to do code generation. While it served well up to a point, it turns out that ASM has much better support for invokedynamic, as well as being a little faster. So, donaldh got things switched over, and I soon was able to emit invokedynamic.

So far, we are not using it a great deal (just for making QAST::WVal compile to something a bit – or potentially a lot – cheaper), but in the future it will be used for things you’d typically think of as invocations (sub and method calls). I’ll write in more detail about it as things evolve.

What next?

With NQP now ported, the focus will fall on Rakudo itself. Quite a lot of preparations have already been made; for example, many pir:: ops have been replaced with nqp:: ones, multiple dispatch has been ported to NQP from C (fixing some bugs along the way), the way HLL boundaries work has been updated to cope with a fully-6model world (this also let me fix a long-standing introspection bug).

The path through getting Rakudo ported will largely follow the build order. This means starting with the module loader, then the compiler, followed by the MOP and its bootstrapping. After that comes the setting – the place where the built-ins live. There’s around 13,650 lines worth of that, so of course I expect to take it a little at a time. :-) I’ll try to remember to get a progress update here in a couple of weeks time.

Posted in Uncategorized | 6 Comments

NQP on JVM gets Grammars, Multiple Dispatch

Having just reached an interesting milestone, I thought I’d blog a quick progress update on my work to port NQP to the JVM, in preparation for also doing a port of Rakudo Perl 6.

The big news is that the grammar and regex engine is pretty much ported. This was a sizable and interesting task, and while a few loose ends need to be wrapped up I think it’s fair to say that the hard bits are done. The port includes support for the basics (literals, quantifiers, character classes, positional and named captures, calls to other rules, and so forth) as well as the more advanced features (transitive Longest Token Matching for both alternations and protoregexes, and embedded code blocks and assertions). Missing are anchors, conjunctions and a small number of built-in rules; none of these are particularly demanding or require primitives that don’t already exist, however. It’s also worth pointing out that the NQP code to calculate the NFAs used in Longest Token Matching also runs just fine atop of the JVM.

Another interesting feature that I ported a little while ago is multiple dispatch. This was some effort to port, since the original implementation had been done in C. While it’s sensible to have a close-to-the-VM dispatch cache, there’s little reason for the one-off candidate sorting work (not a hot path) to be done in C, so I ported the code for this to NQP. This meant that on the JVM side, I just needed to implement a few extra primitives, and could then run the exact same candidate sorting code.

I think it’s worth noting again that I’m really doing two things in parallel here: hunting down places where NQP couples too tightly to Parrot and loosening the coupling, and also doing the porting to the JVM. The first half of this work is relevant to all future ports. In many cases, I’m also finding that the changes give us architectural improvements or just cleaner, more maintainable code. I wanted to point this out especially because I’m seeing various comments popping up suggesting that Rakudo (or even Perl 6) is on a one-way road to the JVM, forsaking all other platforms. That’s not the case. The JVM has both strengths (mature, a solid threading story, widely deployed, the only allowed deployment platform in some development shops, increasing attention to supporting non-Java languages through things like invokedynamic) as well as weaknesses (slow startup time, lack of native co-routine support, and the fact that it was originally aimed at static languages). Rakudo most certainly should run on JVM – and it most certainly should run on other platforms too. And, as I wrote in my previous post, we’ve designed things so that we are able to do so. Perl has always been a language where There’s More Than One Way To Do It. Perl also has a history of running on a very wide range of platforms. Perl 6 should continue down this track – but the new reality is that a bunch of the interesting platforms are virtual, not hardware/OS ones.

By now, the JVM porting work is fast approaching a turning point. Up until now, it’s been about getting a cross-compiler and runtime support in place and working our way through the NQP test suite. This phase is largely over. The next phase is about getting NQP itself cross-compiled – that is, cross-compiling the compiler, so that we have an NQP compiler that runs on the JVM, supporting eval and able to run independently.

Posted in Uncategorized | 9 Comments