I’ll be speaking at “We Actually Build Stuff”

I do a lot of my talking about working on the Rakudo Perl 6 compiler inside the Perl community. Happily, I’ll be able to talk about my experiences doing so to a much more general audience soon: I’ve had a talk accepted at We Actually Build Stuff! Organized by Greg Young – somebody I’ve learned a lot from and been inspired by – it brings together people who have actually worked on sizable and/or interesting projects to present their lessons learned and experiences. I think that’s a great idea for a conference, and I’m excited to be getting to share my experiences there, as well as learn a load from other people’s too. The speaker list besides me is pretty awesome. Also, it’ll be my first time in Lithuania, which is the only Baltic country that I’ve yet to visit. :-)

Posted in Uncategorized | Leave a comment

YAPC::Asia 2012

Three years ago I paid a visit to Japan to attend YAPC::Asia. It was also my first time in Japan, and so in addition to a very enjoyable and well attended Perl conference, I got to enjoy the incredible Japanese cities, the beautiful nature, great food (and this is coming from somebody who can’t eat fish and seafood – yes, there are still plenty of nice things to eat) and some of the most polite and pleasant people I’ve come across anywhere. I’m still a bit mystified what made me leave coming back again a whole three years – but it goes without saying that I’m very glad I did. Japan is as awesome as I remembered. :-)

Naturally, Perl 6 has come a long way since I was last here. Last time, I talked about Perl 6 at the level of interesting snippets of code that you could run that solved small problems. These days, with the language having a growing ecosystem of modules and compilers capable of running them, it felt natural to focus on that. Thus, my talk was Exploring Perl 6 Through Its Modules. The code in it is really nothing new to those who have followed Perl 6 very closely; it just shows off various modules along with some annotations and explanations of how they put Perl 6 to good use. Many people don’t follow along so closely, however, so I think for most people attending the talk had quite a bit of new and interesting stuff.

Of all the Perl events I’ve been to, YAPC::Asia is by a long way the biggest. This year, once you counted speakers in, there was over 800 people there! That’s incredible for a community-organized conference, and the organizers did a great job of making it happen. Also impressive is that they got the videos of the talks up really soon afterwards – so you can even watch my talk! I suggest having the PDF of the slides too for easier following.

Anyway, I’ll now get on with enjoying the Japanese countryside for a few days before returning to Sweden and resuming life as normal. Maybe I’ll even be lucky enough to catch a glimpse of Fuji, :-)

Posted in Uncategorized | 2 Comments

The Moving to Moose Hackathon, and NQP on QAST

It’s already been almost a week and a half since I returned from the Moving To Moose Hackathon. Annoyingly, the first thing that happened on my return is that I got sick. This has combined badly with a very busy work schedule, meaning I’ve struggled to get the rest I need to really recover. As a result, I’ve mostly fallen off the Perl 6 grid over the last week. Anyway, I’ll be fine, though it may take another week before I’m really in decent shape and able to be productive on Rakudo and NQP again.

Anyway, let me talk about the days I spent at the hackathon. It was held at the Preikestolen Mountain Lodge, which gives it the title of “most beautiful hackathon venue I’ve ever been at”, by some margin. Getting to such a remote place with public transport would not have been entirely easy; thankfully, Oslo.pm did an incredible job of organizing their own transport, hiring a minibus that was even adorned with the lovely Oslo.pm logo. The lodge itself was by a lake surrounded by hills, and one of the hackrooms had delightful views. The lodge restaurant also had beautiful views, and it was lovely to be up there at dinner and watch the sunset. Talk about an inspiring location…

I’ve not really been that involved with Moose, or the Perl 5 MOP project, so you may wonder how I ended up at the hackathon. The reason was that I was invited to share my experiences working on 6model, and thus had various interesting discussions about that. I designed and built 6model in relative isolation (though taking many existing meta-object systems as inspiration), thus it was nice to be sat in a room with interesting discussions about MOPs going on. Overall I feel good about what is happening with the Perl 5 MOP work, and having witnessed the work on it in progress I think it’s going good places. I think we’ll be able to get good interop between Perl 5 and Perl 6 objects also.

So what did I hack on? Having a good chunk of time where I could focus entirely on Perl related things meant it was a good chance to get one of the Big Tricky Things on the Rakudo and NQP roadmap taken care of: the moving of NQP to use our new backend, QAST. I’d already got Rakudo largely moved over to using this. In some senses, you’d think getting Rakudo using QAST would be a more difficult task; in reality, Rakudo actually is much more loosely coupled to the VM than NQP has traditionally been. Mostly that’s because a bit more design effort has gone into Rakudo, since it’s the more important product. NQP has more filled the role of “the thing we use to build the product”, and thus has mostly been improved on an as-needed basis. (Both pmichaud and I think NQP can and should be an interesting product in its own right, as a toolchain for compiler construction. But Rakudo has priority.)

Anyway, the Big Result is that NQP is now moved over to the new backend toolchain. Just by doing that, we’ve managed to get some performance improvements thanks to better code generation. Thus you can expect the next release of Rakudo to compile your programs a little faster (we’re talking 5% – 10%, which is welcome even if not huge). This has in turn meant much other code can be eliminated, NQP needs less memory to build, and various hacks have gone away. For example, in the course of the work I re-unified multi-method and multi-sub dispatch in NQP, which had diverged a little some time ago.

This also means the last large dependency that NQP had on code written in Parrot’s Intermediate Language is now gone. This is not only good for maintainability (it’s much better to maintain code in a high level language), but also removes the final big blocker for porting NQP to other VMs. Having spoken to plenty of people on this matter during events over the summer, I sense there’s a lot of enthusiasm for it – especially from those whose views I consider most worth listening to – and I look forward to digging into it.

Other things that this has unlocked is working on giving NQP further native type support and an optimizer. These together will also enable us to improve the performance of NQP execution, in turn facilitating further improvements to the compilation performance of Rakudo. I’m hopeful of getting to those in the course of the coming month; they won’t land for the September release, but having them for the October one seems quite comfortably achievable. We’ll see. :-)

Anyway, I’d like to finish up by thanking Oslo.pm for organizing yet another awesome hackathon. It was very well organized and run, and I’m really grateful for all the time, effort and money they put in to it. Not to mention that they’ve a really nice bunch of individuals also. Great work! :-)

Posted in Uncategorized | Leave a comment

A Rakudo Debugger

I’m currently in the midst of a 2-week chunk of time focused on Perl stuff. It started with the Perl Reunification Summit, which I’ll write more about in a future post, and was followed by YAPC::Europe in Frankfurt. After about 36 hours at home after YAPC, I was on my way to Norway for the Moving To Moose Hackathon, organized by the industrious Oslo.pm. And that’s where I am now!

At YAPC::Europe I presented a project I’d been quietly working on in the couple of weeks running up to YAPC: building a debugger for Rakudo. Nobody there knew I’d been working on it, so it was a nice little surprise for everyone. :-) Happily, it got a very positive reception, and Pm is currently working on the next Rakudo Star build, which will include it.

Let’s take a very quick look at it. When you enter the debugger, it takes you to the first non-declarative line of code that is to be executed (which may be in a BEGIN block or a macro body). It highlights it in yellow and shows some context. In the screenshot below, we’re some way into a program, about to execute the first line of a subroutine. We can enter the name of a variable to introspect its content.

It is also possible to eval code in the current context, or evaluate complex expressions. Another use of the debugger is to understand what went wrong when an exception occurs.

Here, we can see what went wrong, and have been able to introspect the variables at the point where the exception was thrown.

The debugger is also able to single-step through regexes. It also recognizes when it is in a regex (it works for rules and tokens in grammars too) and shows what part of the string has been matched so far.

Here, the yellow indicates the next regex atom to try and match, and the green shows the portion of the string matched so far.

Other features include debugging inside of eval’d code, breakpoints and breaking whenever an exception is thrown. Last bot not least, there’s a REPL mode that works much like the normal Rakudo REPL, apart from it lets you step through the expression you entered:

I have plans for more features in the future, including trace points (so you can log the result of evaluating an expression every time a certain place in the code is hit). If you find bugs or have ideas, just file a GitHub issue…or submit a patch. Much of the code is, after all, written in Perl 6 (the rest is in NQP).


Posted in Uncategorized | 4 Comments

Rakudo QAST switch brings memory reductions

In my previous post I mentioned that the work to get Rakudo using QAST was going nicely. QAST is a refreshed abstract syntax tree design and implementation, written in NQP (the Perl 6 subset we write much of Rakudo in) instead of PIR. I’m happy to say that this work has now been merged into the mainline Rakudo and NQP branches, and thus will be included in the next release. There were no spectest regressions, and just one module needed updating (and that’s because it was using a Parrot-specific construct that has got a bit stricter; all of the other pure Perl 6 modules were fine) As part of this work, POST – a much simpler and more low level tree that gets transformed into Parrot Intermediate Code – has also been replaced.

One of the goals of this work was to reduce memory consumption during compilation, especially of CORE.setting. The results on this front look pretty good. Exact numbers vary depending on 32-bit vs 64-bit and other details. I’ve seen various numbers flying around on channel that all suggest a notable improvement; better still, nwc10++ took a moment to take some decent measurements before and after and from them it seems we’re looking at something around a 37% reduction in memory consumption during compilation of CORE.setting compared to the previous release. Since we use the standard Perl 6 compiler to compile the setting, this means the improvement applies to compilation of scripts and modules in general.

While I was fairly comfortable I could get memory usage down some amount, I was a bit concerned that we may take a speed hit. I mean, this work has involved re-writing a bunch of code written in low-level PIR into rather higher level NQP. That carries obvious risks of a slow down – especially since NQP currently lacks an optimizer. Happily, though, we’ve actually got a little bit faster at compilation. This is thanks to more efficient data structures (of note, using natively typed attributes), the AST being a little better angled towards the needs of Perl 6 compilation, and through an implementation that learned from PAST compilation profiles. The most dramatic speedup was to the optimization phase, which now completes in a third of the time it used to! The gains in other phases are much, much more modest, but it’s still movement in the right direction in a project that carried a real risk of a regression speed wise.

While this work has been merged, this isn’t the end of the road for QAST-related tasks. Here’s what I will be looking at from here, both in time for the August release and probably a bit beyond that:

  • Getting inlining of routines back in place; this is a re-implementation rather than an update. Happily, the re-implementation will be much, much cleaner and should allow for better inlining support. This is currently top of my todo list, so should be in place in the next couple of days.
  • Some more optimizations here and there, which should get us some more (albeit probably relatively smaller) wins in memory usage and speed.
  • Getting NQP switched over to using QAST.
  • Giving NQP natively typed variables, which should let us write a bunch of code more efficiently in the compiler.
  • Giving NQP an optimizer, which should hopefully get us some more speed wins.

In other exciting news, the arrival on QAST means masak++ is digging into taking macros some steps further. Maybe there’ll be something nice there for the next release too. :-)

Posted in Uncategorized | Leave a comment

The Rakudo move to QAST: progressing nicely

It’s been a little while since I wrote anything here. After all the work getting the new regex engine bootstrapped and alternations having longest token matching semantics, I’ve been taking things just a little bit easier. Only a little bit though…things have still been moving along nicely. :-)

My current focus is on getting Rakudo switched over to QAST, our refreshed abstract syntax tree design and implementation. What is an AST, you may wonder? Basically, it’s just a tree representation of the behavior of a program. As we parse programs in Perl 6, the grammar engine builds a parse tree. This is very tied to the syntax of the program, whereas an AST is all about semantics. A chunk of code known as the actions transforms pieces of the parse tree into an abstract syntax tree. Not all elements of your program are represented in the AST, however. Declarations are handled differently, through a mechanism called the world. If you pre-compile a module, the declarative bits are serialized; the AST, on the other hand, represents things that will actually run, so it needs to be turned into code for the target runtime.

Thus, the biggest part of QAST isn’t the AST nodes themselves. It’s the code that takes them and produces lower level code for a target VM; in Rakudo’s case, that is currently just Parrot, but we’re looking forward to targeting other VMs in the not too distant future also.
QAST is the successor to PAST (Q is just P++), and has a great deal in common with it. This is because actually, PAST is pretty nice. So what makes QAST nicer?

  • It’s implemented in NQP, not PIR. This makes it much more pleasant to work on.
  • It’s much more closely integrated with other parts of the compiler tool chain, including the bounded serialization support and 6model-based type system. All of the old stuff for talking about types as just string names is dead and buried.
  • Since the AST nodes are now 6model objects, we can use natively typed attributes in many places. This should lead to memory usage improvements during compilation.
  • The nodes being 6model objects means we can just serialize them using the existing serialization support. Why do we want to serialize AST nodes, you may ask? There are two immediate cases we have. One is masak++’s macros work, when a macro lives in a pre-compiled module. The other is because we want simple routines that can be inlined into other compilation units by the optimizer to carry their AST along with them. At the moment we support this but…it’s a very restricted hacky solution. Now with QAST I can toss a bunch of code I never really liked and simultaneously get a more capable optimizer. Win.
  • It’s a bit smarter about compiling lexical accesses, which will result in better code in some cases.
  • It handles native types better.
  • The nqp::op abstraction layer is now incorporated into QAST itself. This means that the optimizer and other bits of the compiler can work completely in terms of abstract operations, independent of any particular virtual machine. This will help with porting, but incidentally has made a various bits of code cleaner too.
  • The nqp::op layer has also been unified with the notion of operations in general (what used to be pasttype). In addition to this, operations can be overridden at a HLL level (meaning Rakudo can supply its view of operations where they differ from the simpler view NQP takes). This has been put to use already in the updated box/unbox model.
  • Exception handlers were somewhat coupled to blocks in PAST. In QAST the handlers model is a bit different; in my view, it’s simpler and more flexible than what came before it also.
  • VM specific things are now all arranged under a QAST::VM, which provides an escape hatch where it’s needed.
  • Serialization of declarative elements now takes place during the final QAST compilation phase. This is significant because it opens the door to implementing optimizations that twiddle with declarative bits of the program. In the way things were set up in PAST, by the time the optimizer got hold of things, it was already too late.

There’s various other cleanups and improvements too, but I think the list above captures a lot of the niceness.

So, what’s the current status? I decided to go with updating Rakudo first, and do NQP later on. The overall plan of attack was:

  1. Get QAST nodes defined and a QAST::Compiler in place that covers a lot of what PAST::Compiler could do before.
  2. Keep using the original PAST compiler to build a perl6 executable, the core setting and Test.pm. Then also start building a parallel version of the compiler (qperl6.exe) that would use QAST.
  3. Get the basic sanity tests passing using qperl6 (note, the core setting is compiled with the original PAST-based perl6 executable still)
  4. Next up, start attacking the spectests, and make a significant dent into them.
  5. Once it’s capable of doing so, start using qperl6 to compile Test.pm also.
  6. Continue tackling more of the spectests, to get most of them passing again.
  7. Get the optimizer up and running with QAST, and make sure it doesn’t cause any regressions to the spectests.
  8. Get qperl6 able to compile the core setting.
  9. Make the qperl6 code be the main compiler code (so the perl6 executable is now using the QAST toolchain).
  10. Sync up latest changes from the main development branch, triage the remaining handful of spectest issues and module space issues.
  11. Merge! Beer!

Note that these steps aren’t at all similar in size. At the moment, thanks to much hacking by masak++ and myself, design input from pmichaud++ and feedback from moritz++ and tadzik++, we’re up to steps 6 and 7. This means that Rakudo using QAST is able to compile, run and pass the majority (I’d say about 95%) of the spectests, and that we have Test.pm being compiled with QAST-based Rakudo also. My current work is updating the optimizer and taking care of the last few things that I know absolutely have to be sorted out before trying out compiling the core setting. I’m currently optimistic that by the time the weekend is over, I’ll be onto step 9. I’m hopeful that by the following weekend, step 11 will have happened, and thus the August Rakudo release will be QAST based.

The standard for this branch landing is the same as for qbootstrap and altnfa, the last two sizable branches we landed: no regressions in the spectests and module space (unless, of course, they relied on bugs that this branch fixes). So from a user perspective, there’s nothing much to fear from this change, and plenty of nice things to look forward to that it will make possible.

Posted in Uncategorized | 1 Comment

LTM for alternations

This month’s Rakudo development work has already seen us switch to the new QRegex grammar engine for parsing Perl 6 source, unifying it with the mechanism for user-space grammars and regexes. A week and a bit on, another major improvement in this space has also landed: alternations now participate in Longest Token Matching, as per spec. What does this mean? To give a simple example:

> say "beer" ~~ /be|.*/

Here, the second branch of the alternation wins – because it matches more characters than the first branch. This is in contrast to sequential alternation (which you are likely more used to), which is done with the || operator in Perl 6:

> say "beer" ~~ /be||.*/

The || may remind you of the short-circuiting “or” operator, which is exactly what a sequential alternation in a regex does: we try the possibilities in order and pick the first one that matches. On the other hand, the | is a reminder of the “any” junction constructor, which is analogous to what happens in a regex too: we process all of the branches with a parallel NFA, trimming impossible options as we go, and the one that matches most characters will win. If multiple match, we take them in order of length until one matches.

Note that – just like with protoregexes – the thing we actually use the NFA on is the declarative prefix. Perl 6 regexes are a mixture of declarative and procedural; the switch between them is seamless. The declarative bits are amenable to processing with an NFA.

Longest token matching is not only a Perl 6 user-space feature, but also used when parsing Perl 6 – and this goes for alternations too. In fact, the ability to quickly decide which branch to take out of a bunch of possible options is also important for parsing performance. STD, the standard grammar, is written so that trying things sequentially will usually give a correct parse. However, there are exceptions, and up until now they have been problematic. With this work, we now come closer to parsing things the way the standard grammar does. In fact, a lot of the tweaks I had to make in order to get the Perl 6 grammar to parse things correctly again after implementing longest token matching for alternations were a case of aligning it more closely with STD, which is decidedly encouraging.

So, the branches in NQP and Rakudo containing this work have landed. Once again, it was a fairly deep and significant change, and pulling it off has involved various other improvements along the way (such as making tie-breaking by declaration order work reliably). Happily, the improvements we’ve made because we dogfood the grammar engine to parse Perl 6 source will also make things better for those writing grammars in Rakudo. I merged it this evening, with no regressions in the spectests or in module space tests.

While I’ve put in most of the commits on this work, it certainly wasn’t a one person effort. pmichaud++ is once again to thank for the excellent design work behind this, and moritz++, tadzik++ and kboga++ have both helped with testing, fixing tests that had bad assumptions about LTM semantics and fixing Pod parsing to work with the new alternation semantics.

The next release is still two weeks off. I expect to spend my tuits, which should be in reasonable supply, on various follow-up tweaks as a result of the regex engine work, pre-compilation improvements and diving into the QAST work, which I’m hopeful will land in time for the July release. Meanwhile, stay tuned: I expect pmichaud++ will have some nice news about what he’s been cooking up for the June release coming up soon. :-)

Posted in Uncategorized | Leave a comment