A quick JVM backend update

Things have been moving along quite rapidly on the JVM backend since my last post. Sadly, I’m too sick to hack on anything much this evening (hopefully, this turns out to be a very temporary affliction…) but I can at least just about write English, so I figured I’d provide a little update. :-)

Last time I blogged here, I was able to compile various QAST trees down to JVM bytecode and had a growing test suite for this. My hope was that, by some inductive handwaving, being able to compile a bunch of QAST nodes and operations correctly would mean that programs made up of a whole range of them would also compile correctly. In the last week or so, that has come to pass.

Having reached the point of having coverage of quite a lot of QAST, I decided to look into getting an NQP frontend plugged into my QAST to JVM backend. In the process, I found that NQP lacked the odd VM abstraction here and there in the common prelude that it includes with every QAST tree it produces. Thankfully, this was easily rectified. Even better, I got rid of a couple of old hacks that were no longer required. With those things out of the way, I found that this common prelude depended on a couple of operations that I’d not got around to implementing in the JVM backend. These were also simple to add. And…here endeth the integration story. Yup, that was it: I now had a fledgling NQP cross-compiler. An NQP compiler running on Parrot, but producing output for the JVM.

This result is rather exciting, because…

  • It’s using exactly the same parse and action stages as when we’re targeting Parrot. No hacks, no fork. The QAST tree we get from the NQP source code that goes in is exactly the one we get when targeting Parrot. Everything that happens differently happens is beyond that stage, in the backend. This is an extremely positive sign, architecturally.
  • With a couple of small additions to handle the prelude, I was immediately able to cross-compile simple NQP programs and run them on the JVM. There’s no setting or MOP yet, but the basics (variables, loops, subroutines with parameters, even closures) Just Worked.
  • The program I wrote to glue the JVM backend work and the existing NQP frontend together was about 30 lines of NQP code.
  • This whole integration process was about an afternoon’s worth of work.

Since I got that working, my focus has been on getting nqp-mo (the NQP meta-objects) to cross-compile. This is where classes and roles are implemented, and thus is a prerequisite for cross-compiling the NQP setting, which is in turn a prerequisite for being able to start cross-compiling and running the NQP test suite. The NQP MOP is about 1500 lines of NQP code, and at this point I’ve got about 1400 of them to cross-compile. So I’m almost there with it? Well, not quite. Turns out that the next thing I need to port is the bounded serialization stuff. That’s a rather hairy chunk of work.

Anyway, things are moving along nicely. The immediate roadmap is to get the bounded serialization to the point where it’s enough for the NQP MOP, then move on to getting a minimal setting cross compiling. Beyond that, it’ll be working through the test suite, porting the regex compilation and seeing what else is needed to cross-compile the rest of NQP.

Posted in Uncategorized | 5 Comments

A Bunch of Rakudo News

Seems it’s high time for some news here. It’s not that I didn’t do any blogging about Perl 6 in December; it’s just that all of those posts were over on the Perl 6 advent calendar. Anyway, now it’s a new year, and I’m digging back into things after an enjoyable Christmas and New Year’s break in the UK with family and friends. Here’s a bunch of things that already happened but I didn’t get around to talking about here yet, and some things that will be coming up.

Better Parse Errors In 2012.12

Ever had Rakudo tell you there’s a problem on line 1, when it’s really on line 50? Or wished that even in the common case where it gets the line right, it would tell you exactly where on the line things went wrong? Or how about the time it told you “Confused” because you got a closing paren too many?

Many of my contributions to the 2012.12 Rakudo release centered around improving its reporting of parse errors. STD, the standard Perl 6 grammar, has had much better error reporting than Rakudo for a while. Therefore, I spent a bunch of time aligning our error reporting more closely with what STD does. Some of this is cosmetic: you get the colored output and the indication of the parse location. But while these cosmetic changes will be the most immediately visible thing, the changes go far deeper. Of note, a high water mark is kept so we can be a lot more accurate in reporting where things came unstuck, and we track what was expected so we can produce better errors. Just doing the cosmetic stuff without being able to give it a better location to report wouldn’t have helped so much. :-)

One other change is that we don’t bail out on the first thing that’s wrong when it’s possible to survive and continue parsing. When this is possible, up to 10 errors will be reported (since that’s typically a screen worth). Of course, some things just hose the parse and we can’t continue in any sensible way.

Hopefully, these improvements will make using Rakudo feel a lot nicer. Already on channel, I can feel the feedback we’re giving about parse errors when people use the evalbot is often a lot more pleasant and informative. Of course, there’ll be more improvements in the future too, but this is a big step forward.

Faster Auto-Threading

The junction auto-threader could sometimes be insanely slow. As in, ridiculously so. After hearing a bunch of reports about this, I decided to dig in and work out why. A rewrite later, the little benchmark I was using with it ran almost 30 times faster. Not so bad… :-) This change also made it into the 2012.12 release.

JVM Backend Preparations Underway

I’ve talked plenty about plans for NQP and Rakudo to run on things besides Parrot for a while now. Over the last year or two, we’ve laid a lot of the groundwork for this. What’s been especially pleasing is that it’s also made Rakudo a better quality Perl 6 implementation on Parrot, thanks to the many architectural improvements. Of note, in many places we’ve closed semantic gaps between what Perl 6 wants and the primitives we were building it out of; the new QAST (Q Abstract Syntax Tree) is a great example.

Anyway, with NQP now being written pretty much entirely in NQP, and many of the right abstractions in place, it felt like time to start slowly picking away at getting 6model ported to the JVM and work on turning QAST trees into Java bytecode. I quietly started on this in November, and mentioned the existence of the work on #perl6 in December. I was delighted to see Coke++ jump in and start working through the Low Hanging Fruit – a file where I’m putting tasks that should be relatively easy to pick off. I actually had to re-fill it, after the last round were depleted. ;-) By now, quite a few bits of QAST are supported and the 6model on JVM implementation is coming along nicely. Yes, this means it’s already capable of doing basic meta-programming stuff.

Note that this work isn’t at the stage where it’s of use for anything yet. You can’t even write a say(“hello world”) in NQP and have it run on the JVM yet, since all the work so far is just about turning QAST trees into JVM bytecode and building the runtime support it needs. That may seem like a curious way to work, but once you do enough compiler stuff you find yourself thinking quite naturally in trees. It meant I didn’t have to worry about creating some stripped-down NQP that could emit super-simple trees to be able to test really simple things. After all, the goal is to run NQP itself on the JVM, and then Rakudo, and only then will things be interesting to the everyday user.

To address a couple of immediate concerns that some may have…

  • No, this is not a case of “stop running on Parrot, start running on JVM”. It’s adding an additional backend, much like pmurias++ has been working on adding a JavaScript backend for NQP. Of course, I expect resource allocation in the future to be driven by which backends users desire most. For some, the JVM is “that evil Oracle thing” and they don’t want to touch it. For others, the JVM is “the only thing our company will deploy on”. Thus I expect this work to matter more to some people than others. That’s fine.
  • No, targeting multiple backends doesn’t mean performance-sucking abstractions everywhere. It’s a pretty typical thing for a compiler to do. As usual, it’s about picking the right abstractions. The debugger was implemented as an additional Rakudo frontend without a single addition to Rakudo or NQP or anything anywhere in the stack. That was possible because things were designed well. I’m sure the process of getting things running on the JVM will flag up a few places where things aren’t designed as well as they need to be, but already I’m seeing a lot of places where things are mapping over very nicely indeed.
  • No, this doesn’t mean that all other Rakudo development will grind to a halt. I’ve been working on the JVM backend stuff in November and December; both months saw a huge amount of Rakudo progress too. Things will go on that way.

Type System Improvements

Rakudo does a lot of things well when it comes to supporting the various kinds of types Perl 6 offers, but there are some weak areas. Here are some of the things I plan to focus on:

  • Getting the definedness constraint stuff working better (the :D and :U annotations). At the moment, they’re supported as a special case in multiple dispatch and the signature binder. In reality, they’re meant to be first class and work everywhere. You may not think you care about these much. Actually, you probably at least indirectly do, because once the optimizer is able to analyze them, it’ll be able to do a bunch more inlining of things than it can today. :-)
  • Getting coercion types in place. Again, this is turning the special-cased “as” syntax into the much more general coercion type syntax (for example, Int(Str) is the type that accepts a Str and coerces it into an Int).
  • Getting native types much better supported. At the moment, you can use them but…there are pitfalls. Having them available has been a huge win for us in the CORE setting, where we’ve used them in the built-ins. But they’re still a bit “handle with great care”. I want to change that.
  • Implementing compact arrays.
  • Improving our parametric type and type variable support. Many things work, but there are some fragile spots and some bugs that want some attention.

I intend to have some of these things in the January release, and a bunch more in the February one. We’ll see how I get along. :-)

Posted in Uncategorized | 3 Comments

Lots of improvements coming in the Rakudo November release

The November release is still a couple of weeks off, but it’s already looking like one of the most exciting ones in a while. Here’s a rundown of the major improvements you can expect.

User Defined Operator Improvements

The way Rakudo handles parsing of user-defined operators has been almost completely reworked. The original implementation dated back quite a long way (we’re talking largely unchanged since probably 2010 or so), and as you might imagine, we’ve the tools to do a lot better now. The most significant change is that we now parse them using a lexically scoped derived language. This is done by mixing in to the current cursor’s grammar – something we couldn’t have done some years ago when NQP lacked roles! This means that the additions to the grammar don’t leak outside of the scope where the user-defined operators are defined – unless they are explicitly imported into another scope, of course.

    sub postfix:<!>($n) { [*] 1..$n };
    say 10!; # 3628800
say 10!; # fails to parse

Before, things were rather more leaky and thus not to spec. Perl 6 may allow lots of language extensibility, but the language design takes great care to limit its scope. Now Rakudo does much better here. As well as making things more correct, the nasty bug with pre-compilation of modules containing user defined operators is gone. And, last but not least, the precedence and associativity traits are now implemented, so user defined operators can pick their precedence level.

sub infix:<multiplied-with>($a, $b) is equiv(&infix:<*>) {
    $a * $b
sub infix:<to-the-power>($a, $b) is equiv(&infix:<**>) {
    $a ** $b
say 2 multiplied-with 3 to-the-power 2;

Quote Adverbs and Heredocs

The other thing that’s had a big do-over is quote parsing. This meant a lot of quoting related things that hadn’t been easy to attack before became very possible. So possible, in fact, that they’ve been done. Thus, we now support heredocs:

say q:to/DRINKS/

This outputs:


Notice how it trims whitespace at the start of the string to the same level of degree of indentation as the end marker, so you can have heredocs indented to the same level your code is, if you wish. The :to syntax is an example of a quote adverb, and we now support those generally too. For example, a double-quoted string normally interpolates a bunch of different things: scalars, arrays, function calls, closures, etc. But what if you want a quoting construct that only interpolates scalars? You do something like:

my $value = 42;
say Q:s'{ "value": $value }';

Alternatively, you could take a quoting construct that normally interpolates everything and just switch off the closure interpolation:

my $value = 42;
say qq:!c'{ "value": $value }';

Last but not least, the shell words syntax now works. This lets you do quoting where things are broken up into a list by whitespace, but you can quote individual parts to prevent them
getting split, and also do interpolation.

say << Hobgoblin 'Punk IPA' >>.perl; # ("Hobgoblin", "Punk IPA")

Sadly, I’ve got too bad a headache today to enjoy any of the listed beers. But hey, at least now you know some good ones to try… :-)

Operator Adverbs

Well, weren’t these some fun to implement. STD has parsed them for a long while. You’d think that’d make it easy to steal from, but no. Before you can do that, you have to figure out that not only does it parse adverbs as if they were infixes, but then pushes them onto the op stack in the operator precedence parser and does a reduce such that they come out looking like postfixes, which in turn need an AST transform to turn the postfix inside-out so what was originally parsed as a fake infix becomes a named argument to the thing preceding it. I’m still not sure if this is beautiful or horrifying, but now it’s in place. This means we can at last handle the adverb-y syntax for doing things like hash key existence checks or deletion:

my %h = a => 0;
say %h<a>:exists; # True
say %h<a>:delete; # 0
say %h<a>:exists; # False


Recent work by masak++ has got macros to the point where they’re powerful enough to be potentially useful now. See his blog post for the sometimes-gory details, illustrated with exciting adventures involving wine barrels and questionably pluralized bits of Russian landscape. Really.

NFA Precomputation

This is decidedly a guts change, but worth a mention because it’s probably the main performance improvement we’ll see in the November release: the transitive NFAs that are used by the parser are now computed at the time we compile the Perl 6 grammar, not on-demand as we start parsing. They are then serialized, so at startup we just deserialize them and dig into the parsing. They’re not entirely cheap to construct, and so this saves a bit of work per invocation of the compiler. In the spectests, it made an almost 10% difference to the time they take to run (many test files are quite small, and we run over 700 test files, so reducing invocation overhead adds up).

By the way, if you’re confused on what these transitive NFAs are about, let me try and explain a little. In order to decide which of a bunch of protoregexes or which branch of an alternation to take, the declarative parts of a grammar are analyzed to build NFAs: state machines that can efficiently decide which branches are possible then rank them by token length. It’s not only important for a correct parse (to get longest token matching semantics correct), but also important algorithmically. If you look at the way Perl 6 grammars work, the naive view is that it’s recursive descent. That makes it nice to write, but if it really was evaluated that way, you’d end up trying loads of paths that didn’t work out before hitting upon the correct one. The NFA is used to trim the set of possible paths through the grammar down to the ones that are actually worth following, and their construction is transitive, so that we can avoid following call chains several levels deep that would be fruitless. If you’ve ever used the debugger to explore a grammar, and wondered how we seem to magically jump to the correct alternative so often, well, now you know. :-)

Other Things

There are a few other bits and pieces: INIT phasers now work as r-values, you can use the FIRST/NEXT/LAST phasers in all the looping constructs now (previously, they worked only in for loops), version control markers left behind in code are gracefully detected for what they are, and a bunch of proto subs in the setting have been given narrower signatures which makes various things you’d expect to work when passed to, say, map, actually do so.

Oh, and whatever else we manage to get around to in the next couple of weeks. :-)

Posted in Uncategorized | Leave a comment

Rakudo Debugger Updates

A while ago I wrote about Rakudo getting an interactive debugger. The feedback I’ve got has been a happy mixture of “this is useful” and “I’d really like it to X” – so I’ve been working on some improvements. I showed them off at my Nordic Perl Workshop talk the weekend before last, in a fun session where I used the debugger to hunt down some problems in a small tool that had a couple of modules. It was nice not only to demonstrate the debugger itself, but also because I could show some neat Perl 6 code along the way.

So, what new things can you expect from the debugger that will be bundled in the Rakudo Star release that will be emerging in the next few days? Here’s a quick rundown.

Attributes that are in scope can now be introspected, just by typing their name. Additionally, the self keyword is recognized as a variable name by the debugger, meaning you can look at it directly (just type “self” and press enter) or even to look at public attributes (“self.name”).

Before, if you were on a line of code that was going to make a call, you would always step in to the callee on pressing enter. Now, if you type “s” and press enter, you will step over the call. Got bored of debugging the current routine and want to let it run its course, and break on the next statement after it returns? Just type “so” and press enter, and you will step out of the current routine.

Trace points are perhaps the most powerful of the new features. They enable you to add print statements to your code, without actually adding print statements to your code. A trace point is like a break point, but instead of breaking, it just evaluates an expression and logs it. Later on, you can view the log.

To go with Rakudo’s improving support for the P5 adverb on regexes, which allows the use of Perl 5 regex syntax from within Perl 6, the debugger now also supports single-stepping through those Perl 5 regexes.

There are also a range of small fixes and tweaks that avoid some of the noise you could get before. For example, fails no longer count as exception throws that cause a rt (run until throw) to break, and sigspace in rules is no longer single-stepped, just jumping straight to the next interesting atom.

What of plans for the future? There are some other ideas already in the issues queue. Beyond those, I’m planning to make a web-based API frontend to the debugger, to go alongside the command line one. This should allow a browser-based debugging interface to be built, but should also enable tools like Padre to integrate with the Rakudo Debugger.

Enjoy the updates!

Posted in Uncategorized | Leave a comment

I’ll be speaking at “We Actually Build Stuff”

I do a lot of my talking about working on the Rakudo Perl 6 compiler inside the Perl community. Happily, I’ll be able to talk about my experiences doing so to a much more general audience soon: I’ve had a talk accepted at We Actually Build Stuff! Organized by Greg Young – somebody I’ve learned a lot from and been inspired by – it brings together people who have actually worked on sizable and/or interesting projects to present their lessons learned and experiences. I think that’s a great idea for a conference, and I’m excited to be getting to share my experiences there, as well as learn a load from other people’s too. The speaker list besides me is pretty awesome. Also, it’ll be my first time in Lithuania, which is the only Baltic country that I’ve yet to visit. :-)

Posted in Uncategorized | Leave a comment

YAPC::Asia 2012

Three years ago I paid a visit to Japan to attend YAPC::Asia. It was also my first time in Japan, and so in addition to a very enjoyable and well attended Perl conference, I got to enjoy the incredible Japanese cities, the beautiful nature, great food (and this is coming from somebody who can’t eat fish and seafood – yes, there are still plenty of nice things to eat) and some of the most polite and pleasant people I’ve come across anywhere. I’m still a bit mystified what made me leave coming back again a whole three years – but it goes without saying that I’m very glad I did. Japan is as awesome as I remembered. :-)

Naturally, Perl 6 has come a long way since I was last here. Last time, I talked about Perl 6 at the level of interesting snippets of code that you could run that solved small problems. These days, with the language having a growing ecosystem of modules and compilers capable of running them, it felt natural to focus on that. Thus, my talk was Exploring Perl 6 Through Its Modules. The code in it is really nothing new to those who have followed Perl 6 very closely; it just shows off various modules along with some annotations and explanations of how they put Perl 6 to good use. Many people don’t follow along so closely, however, so I think for most people attending the talk had quite a bit of new and interesting stuff.

Of all the Perl events I’ve been to, YAPC::Asia is by a long way the biggest. This year, once you counted speakers in, there was over 800 people there! That’s incredible for a community-organized conference, and the organizers did a great job of making it happen. Also impressive is that they got the videos of the talks up really soon afterwards – so you can even watch my talk! I suggest having the PDF of the slides too for easier following.

Anyway, I’ll now get on with enjoying the Japanese countryside for a few days before returning to Sweden and resuming life as normal. Maybe I’ll even be lucky enough to catch a glimpse of Fuji, :-)

Posted in Uncategorized | 2 Comments

The Moving to Moose Hackathon, and NQP on QAST

It’s already been almost a week and a half since I returned from the Moving To Moose Hackathon. Annoyingly, the first thing that happened on my return is that I got sick. This has combined badly with a very busy work schedule, meaning I’ve struggled to get the rest I need to really recover. As a result, I’ve mostly fallen off the Perl 6 grid over the last week. Anyway, I’ll be fine, though it may take another week before I’m really in decent shape and able to be productive on Rakudo and NQP again.

Anyway, let me talk about the days I spent at the hackathon. It was held at the Preikestolen Mountain Lodge, which gives it the title of “most beautiful hackathon venue I’ve ever been at”, by some margin. Getting to such a remote place with public transport would not have been entirely easy; thankfully, Oslo.pm did an incredible job of organizing their own transport, hiring a minibus that was even adorned with the lovely Oslo.pm logo. The lodge itself was by a lake surrounded by hills, and one of the hackrooms had delightful views. The lodge restaurant also had beautiful views, and it was lovely to be up there at dinner and watch the sunset. Talk about an inspiring location…

I’ve not really been that involved with Moose, or the Perl 5 MOP project, so you may wonder how I ended up at the hackathon. The reason was that I was invited to share my experiences working on 6model, and thus had various interesting discussions about that. I designed and built 6model in relative isolation (though taking many existing meta-object systems as inspiration), thus it was nice to be sat in a room with interesting discussions about MOPs going on. Overall I feel good about what is happening with the Perl 5 MOP work, and having witnessed the work on it in progress I think it’s going good places. I think we’ll be able to get good interop between Perl 5 and Perl 6 objects also.

So what did I hack on? Having a good chunk of time where I could focus entirely on Perl related things meant it was a good chance to get one of the Big Tricky Things on the Rakudo and NQP roadmap taken care of: the moving of NQP to use our new backend, QAST. I’d already got Rakudo largely moved over to using this. In some senses, you’d think getting Rakudo using QAST would be a more difficult task; in reality, Rakudo actually is much more loosely coupled to the VM than NQP has traditionally been. Mostly that’s because a bit more design effort has gone into Rakudo, since it’s the more important product. NQP has more filled the role of “the thing we use to build the product”, and thus has mostly been improved on an as-needed basis. (Both pmichaud and I think NQP can and should be an interesting product in its own right, as a toolchain for compiler construction. But Rakudo has priority.)

Anyway, the Big Result is that NQP is now moved over to the new backend toolchain. Just by doing that, we’ve managed to get some performance improvements thanks to better code generation. Thus you can expect the next release of Rakudo to compile your programs a little faster (we’re talking 5% – 10%, which is welcome even if not huge). This has in turn meant much other code can be eliminated, NQP needs less memory to build, and various hacks have gone away. For example, in the course of the work I re-unified multi-method and multi-sub dispatch in NQP, which had diverged a little some time ago.

This also means the last large dependency that NQP had on code written in Parrot’s Intermediate Language is now gone. This is not only good for maintainability (it’s much better to maintain code in a high level language), but also removes the final big blocker for porting NQP to other VMs. Having spoken to plenty of people on this matter during events over the summer, I sense there’s a lot of enthusiasm for it – especially from those whose views I consider most worth listening to – and I look forward to digging into it.

Other things that this has unlocked is working on giving NQP further native type support and an optimizer. These together will also enable us to improve the performance of NQP execution, in turn facilitating further improvements to the compilation performance of Rakudo. I’m hopeful of getting to those in the course of the coming month; they won’t land for the September release, but having them for the October one seems quite comfortably achievable. We’ll see. :-)

Anyway, I’d like to finish up by thanking Oslo.pm for organizing yet another awesome hackathon. It was very well organized and run, and I’m really grateful for all the time, effort and money they put in to it. Not to mention that they’ve a really nice bunch of individuals also. Great work! :-)

Posted in Uncategorized | Leave a comment