Lots of improvements coming in the Rakudo November release

The November release is still a couple of weeks off, but it’s already looking like one of the most exciting ones in a while. Here’s a rundown of the major improvements you can expect.

User Defined Operator Improvements

The way Rakudo handles parsing of user-defined operators has been almost completely reworked. The original implementation dated back quite a long way (we’re talking largely unchanged since probably 2010 or so), and as you might imagine, we’ve the tools to do a lot better now. The most significant change is that we now parse them using a lexically scoped derived language. This is done by mixing in to the current cursor’s grammar – something we couldn’t have done some years ago when NQP lacked roles! This means that the additions to the grammar don’t leak outside of the scope where the user-defined operators are defined – unless they are explicitly imported into another scope, of course.

{
    sub postfix:<!>($n) { [*] 1..$n };
    say 10!; # 3628800
}
say 10!; # fails to parse

Before, things were rather more leaky and thus not to spec. Perl 6 may allow lots of language extensibility, but the language design takes great care to limit its scope. Now Rakudo does much better here. As well as making things more correct, the nasty bug with pre-compilation of modules containing user defined operators is gone. And, last but not least, the precedence and associativity traits are now implemented, so user defined operators can pick their precedence level.

sub infix:<multiplied-with>($a, $b) is equiv(&infix:<*>) {
    $a * $b
}
sub infix:<to-the-power>($a, $b) is equiv(&infix:<**>) {
    $a ** $b
}
say 2 multiplied-with 3 to-the-power 2;

Quote Adverbs and Heredocs

The other thing that’s had a big do-over is quote parsing. This meant a lot of quoting related things that hadn’t been easy to attack before became very possible. So possible, in fact, that they’ve been done. Thus, we now support heredocs:

say q:to/DRINKS/
    Yeti
    Leffe
    Tokyo
    DRINKS

This outputs:

Yeti
Leffe
Tokyo

Notice how it trims whitespace at the start of the string to the same level of degree of indentation as the end marker, so you can have heredocs indented to the same level your code is, if you wish. The :to syntax is an example of a quote adverb, and we now support those generally too. For example, a double-quoted string normally interpolates a bunch of different things: scalars, arrays, function calls, closures, etc. But what if you want a quoting construct that only interpolates scalars? You do something like:

my $value = 42;
say Q:s'{ "value": $value }';

Alternatively, you could take a quoting construct that normally interpolates everything and just switch off the closure interpolation:

my $value = 42;
say qq:!c'{ "value": $value }';

Last but not least, the shell words syntax now works. This lets you do quoting where things are broken up into a list by whitespace, but you can quote individual parts to prevent them
getting split, and also do interpolation.

say << Hobgoblin 'Punk IPA' >>.perl; # ("Hobgoblin", "Punk IPA")

Sadly, I’ve got too bad a headache today to enjoy any of the listed beers. But hey, at least now you know some good ones to try… :-)

Operator Adverbs

Well, weren’t these some fun to implement. STD has parsed them for a long while. You’d think that’d make it easy to steal from, but no. Before you can do that, you have to figure out that not only does it parse adverbs as if they were infixes, but then pushes them onto the op stack in the operator precedence parser and does a reduce such that they come out looking like postfixes, which in turn need an AST transform to turn the postfix inside-out so what was originally parsed as a fake infix becomes a named argument to the thing preceding it. I’m still not sure if this is beautiful or horrifying, but now it’s in place. This means we can at last handle the adverb-y syntax for doing things like hash key existence checks or deletion:

my %h = a => 0;
say %h<a>:exists; # True
say %h<a>:delete; # 0
say %h<a>:exists; # False

Macros

Recent work by masak++ has got macros to the point where they’re powerful enough to be potentially useful now. See his blog post for the sometimes-gory details, illustrated with exciting adventures involving wine barrels and questionably pluralized bits of Russian landscape. Really.

NFA Precomputation

This is decidedly a guts change, but worth a mention because it’s probably the main performance improvement we’ll see in the November release: the transitive NFAs that are used by the parser are now computed at the time we compile the Perl 6 grammar, not on-demand as we start parsing. They are then serialized, so at startup we just deserialize them and dig into the parsing. They’re not entirely cheap to construct, and so this saves a bit of work per invocation of the compiler. In the spectests, it made an almost 10% difference to the time they take to run (many test files are quite small, and we run over 700 test files, so reducing invocation overhead adds up).

By the way, if you’re confused on what these transitive NFAs are about, let me try and explain a little. In order to decide which of a bunch of protoregexes or which branch of an alternation to take, the declarative parts of a grammar are analyzed to build NFAs: state machines that can efficiently decide which branches are possible then rank them by token length. It’s not only important for a correct parse (to get longest token matching semantics correct), but also important algorithmically. If you look at the way Perl 6 grammars work, the naive view is that it’s recursive descent. That makes it nice to write, but if it really was evaluated that way, you’d end up trying loads of paths that didn’t work out before hitting upon the correct one. The NFA is used to trim the set of possible paths through the grammar down to the ones that are actually worth following, and their construction is transitive, so that we can avoid following call chains several levels deep that would be fruitless. If you’ve ever used the debugger to explore a grammar, and wondered how we seem to magically jump to the correct alternative so often, well, now you know. :-)

Other Things

There are a few other bits and pieces: INIT phasers now work as r-values, you can use the FIRST/NEXT/LAST phasers in all the looping constructs now (previously, they worked only in for loops), version control markers left behind in code are gracefully detected for what they are, and a bunch of proto subs in the setting have been given narrower signatures which makes various things you’d expect to work when passed to, say, map, actually do so.

Oh, and whatever else we manage to get around to in the next couple of weeks. :-)

Posted in Uncategorized | Leave a comment

Rakudo Debugger Updates

A while ago I wrote about Rakudo getting an interactive debugger. The feedback I’ve got has been a happy mixture of “this is useful” and “I’d really like it to X” – so I’ve been working on some improvements. I showed them off at my Nordic Perl Workshop talk the weekend before last, in a fun session where I used the debugger to hunt down some problems in a small tool that had a couple of modules. It was nice not only to demonstrate the debugger itself, but also because I could show some neat Perl 6 code along the way.

So, what new things can you expect from the debugger that will be bundled in the Rakudo Star release that will be emerging in the next few days? Here’s a quick rundown.

Attributes that are in scope can now be introspected, just by typing their name. Additionally, the self keyword is recognized as a variable name by the debugger, meaning you can look at it directly (just type “self” and press enter) or even to look at public attributes (“self.name”).

Before, if you were on a line of code that was going to make a call, you would always step in to the callee on pressing enter. Now, if you type “s” and press enter, you will step over the call. Got bored of debugging the current routine and want to let it run its course, and break on the next statement after it returns? Just type “so” and press enter, and you will step out of the current routine.

Trace points are perhaps the most powerful of the new features. They enable you to add print statements to your code, without actually adding print statements to your code. A trace point is like a break point, but instead of breaking, it just evaluates an expression and logs it. Later on, you can view the log.

To go with Rakudo’s improving support for the P5 adverb on regexes, which allows the use of Perl 5 regex syntax from within Perl 6, the debugger now also supports single-stepping through those Perl 5 regexes.

There are also a range of small fixes and tweaks that avoid some of the noise you could get before. For example, fails no longer count as exception throws that cause a rt (run until throw) to break, and sigspace in rules is no longer single-stepped, just jumping straight to the next interesting atom.

What of plans for the future? There are some other ideas already in the issues queue. Beyond those, I’m planning to make a web-based API frontend to the debugger, to go alongside the command line one. This should allow a browser-based debugging interface to be built, but should also enable tools like Padre to integrate with the Rakudo Debugger.

Enjoy the updates!

Posted in Uncategorized | Leave a comment

I’ll be speaking at “We Actually Build Stuff”

I do a lot of my talking about working on the Rakudo Perl 6 compiler inside the Perl community. Happily, I’ll be able to talk about my experiences doing so to a much more general audience soon: I’ve had a talk accepted at We Actually Build Stuff! Organized by Greg Young – somebody I’ve learned a lot from and been inspired by – it brings together people who have actually worked on sizable and/or interesting projects to present their lessons learned and experiences. I think that’s a great idea for a conference, and I’m excited to be getting to share my experiences there, as well as learn a load from other people’s too. The speaker list besides me is pretty awesome. Also, it’ll be my first time in Lithuania, which is the only Baltic country that I’ve yet to visit. :-)

Posted in Uncategorized | Leave a comment

YAPC::Asia 2012

Three years ago I paid a visit to Japan to attend YAPC::Asia. It was also my first time in Japan, and so in addition to a very enjoyable and well attended Perl conference, I got to enjoy the incredible Japanese cities, the beautiful nature, great food (and this is coming from somebody who can’t eat fish and seafood – yes, there are still plenty of nice things to eat) and some of the most polite and pleasant people I’ve come across anywhere. I’m still a bit mystified what made me leave coming back again a whole three years – but it goes without saying that I’m very glad I did. Japan is as awesome as I remembered. :-)

Naturally, Perl 6 has come a long way since I was last here. Last time, I talked about Perl 6 at the level of interesting snippets of code that you could run that solved small problems. These days, with the language having a growing ecosystem of modules and compilers capable of running them, it felt natural to focus on that. Thus, my talk was Exploring Perl 6 Through Its Modules. The code in it is really nothing new to those who have followed Perl 6 very closely; it just shows off various modules along with some annotations and explanations of how they put Perl 6 to good use. Many people don’t follow along so closely, however, so I think for most people attending the talk had quite a bit of new and interesting stuff.

Of all the Perl events I’ve been to, YAPC::Asia is by a long way the biggest. This year, once you counted speakers in, there was over 800 people there! That’s incredible for a community-organized conference, and the organizers did a great job of making it happen. Also impressive is that they got the videos of the talks up really soon afterwards – so you can even watch my talk! I suggest having the PDF of the slides too for easier following.

Anyway, I’ll now get on with enjoying the Japanese countryside for a few days before returning to Sweden and resuming life as normal. Maybe I’ll even be lucky enough to catch a glimpse of Fuji, :-)

Posted in Uncategorized | 2 Comments

The Moving to Moose Hackathon, and NQP on QAST

It’s already been almost a week and a half since I returned from the Moving To Moose Hackathon. Annoyingly, the first thing that happened on my return is that I got sick. This has combined badly with a very busy work schedule, meaning I’ve struggled to get the rest I need to really recover. As a result, I’ve mostly fallen off the Perl 6 grid over the last week. Anyway, I’ll be fine, though it may take another week before I’m really in decent shape and able to be productive on Rakudo and NQP again.

Anyway, let me talk about the days I spent at the hackathon. It was held at the Preikestolen Mountain Lodge, which gives it the title of “most beautiful hackathon venue I’ve ever been at”, by some margin. Getting to such a remote place with public transport would not have been entirely easy; thankfully, Oslo.pm did an incredible job of organizing their own transport, hiring a minibus that was even adorned with the lovely Oslo.pm logo. The lodge itself was by a lake surrounded by hills, and one of the hackrooms had delightful views. The lodge restaurant also had beautiful views, and it was lovely to be up there at dinner and watch the sunset. Talk about an inspiring location…

I’ve not really been that involved with Moose, or the Perl 5 MOP project, so you may wonder how I ended up at the hackathon. The reason was that I was invited to share my experiences working on 6model, and thus had various interesting discussions about that. I designed and built 6model in relative isolation (though taking many existing meta-object systems as inspiration), thus it was nice to be sat in a room with interesting discussions about MOPs going on. Overall I feel good about what is happening with the Perl 5 MOP work, and having witnessed the work on it in progress I think it’s going good places. I think we’ll be able to get good interop between Perl 5 and Perl 6 objects also.

So what did I hack on? Having a good chunk of time where I could focus entirely on Perl related things meant it was a good chance to get one of the Big Tricky Things on the Rakudo and NQP roadmap taken care of: the moving of NQP to use our new backend, QAST. I’d already got Rakudo largely moved over to using this. In some senses, you’d think getting Rakudo using QAST would be a more difficult task; in reality, Rakudo actually is much more loosely coupled to the VM than NQP has traditionally been. Mostly that’s because a bit more design effort has gone into Rakudo, since it’s the more important product. NQP has more filled the role of “the thing we use to build the product”, and thus has mostly been improved on an as-needed basis. (Both pmichaud and I think NQP can and should be an interesting product in its own right, as a toolchain for compiler construction. But Rakudo has priority.)

Anyway, the Big Result is that NQP is now moved over to the new backend toolchain. Just by doing that, we’ve managed to get some performance improvements thanks to better code generation. Thus you can expect the next release of Rakudo to compile your programs a little faster (we’re talking 5% – 10%, which is welcome even if not huge). This has in turn meant much other code can be eliminated, NQP needs less memory to build, and various hacks have gone away. For example, in the course of the work I re-unified multi-method and multi-sub dispatch in NQP, which had diverged a little some time ago.

This also means the last large dependency that NQP had on code written in Parrot’s Intermediate Language is now gone. This is not only good for maintainability (it’s much better to maintain code in a high level language), but also removes the final big blocker for porting NQP to other VMs. Having spoken to plenty of people on this matter during events over the summer, I sense there’s a lot of enthusiasm for it – especially from those whose views I consider most worth listening to – and I look forward to digging into it.

Other things that this has unlocked is working on giving NQP further native type support and an optimizer. These together will also enable us to improve the performance of NQP execution, in turn facilitating further improvements to the compilation performance of Rakudo. I’m hopeful of getting to those in the course of the coming month; they won’t land for the September release, but having them for the October one seems quite comfortably achievable. We’ll see. :-)

Anyway, I’d like to finish up by thanking Oslo.pm for organizing yet another awesome hackathon. It was very well organized and run, and I’m really grateful for all the time, effort and money they put in to it. Not to mention that they’ve a really nice bunch of individuals also. Great work! :-)

Posted in Uncategorized | Leave a comment

A Rakudo Debugger

I’m currently in the midst of a 2-week chunk of time focused on Perl stuff. It started with the Perl Reunification Summit, which I’ll write more about in a future post, and was followed by YAPC::Europe in Frankfurt. After about 36 hours at home after YAPC, I was on my way to Norway for the Moving To Moose Hackathon, organized by the industrious Oslo.pm. And that’s where I am now!

At YAPC::Europe I presented a project I’d been quietly working on in the couple of weeks running up to YAPC: building a debugger for Rakudo. Nobody there knew I’d been working on it, so it was a nice little surprise for everyone. :-) Happily, it got a very positive reception, and Pm is currently working on the next Rakudo Star build, which will include it.

Let’s take a very quick look at it. When you enter the debugger, it takes you to the first non-declarative line of code that is to be executed (which may be in a BEGIN block or a macro body). It highlights it in yellow and shows some context. In the screenshot below, we’re some way into a program, about to execute the first line of a subroutine. We can enter the name of a variable to introspect its content.

It is also possible to eval code in the current context, or evaluate complex expressions. Another use of the debugger is to understand what went wrong when an exception occurs.

Here, we can see what went wrong, and have been able to introspect the variables at the point where the exception was thrown.

The debugger is also able to single-step through regexes. It also recognizes when it is in a regex (it works for rules and tokens in grammars too) and shows what part of the string has been matched so far.

Here, the yellow indicates the next regex atom to try and match, and the green shows the portion of the string matched so far.

Other features include debugging inside of eval’d code, breakpoints and breaking whenever an exception is thrown. Last bot not least, there’s a REPL mode that works much like the normal Rakudo REPL, apart from it lets you step through the expression you entered:

I have plans for more features in the future, including trace points (so you can log the result of evaluating an expression every time a certain place in the code is hit). If you find bugs or have ideas, just file a GitHub issue…or submit a patch. Much of the code is, after all, written in Perl 6 (the rest is in NQP).

Enjoy!

Posted in Uncategorized | 4 Comments

Rakudo QAST switch brings memory reductions

In my previous post I mentioned that the work to get Rakudo using QAST was going nicely. QAST is a refreshed abstract syntax tree design and implementation, written in NQP (the Perl 6 subset we write much of Rakudo in) instead of PIR. I’m happy to say that this work has now been merged into the mainline Rakudo and NQP branches, and thus will be included in the next release. There were no spectest regressions, and just one module needed updating (and that’s because it was using a Parrot-specific construct that has got a bit stricter; all of the other pure Perl 6 modules were fine) As part of this work, POST – a much simpler and more low level tree that gets transformed into Parrot Intermediate Code – has also been replaced.

One of the goals of this work was to reduce memory consumption during compilation, especially of CORE.setting. The results on this front look pretty good. Exact numbers vary depending on 32-bit vs 64-bit and other details. I’ve seen various numbers flying around on channel that all suggest a notable improvement; better still, nwc10++ took a moment to take some decent measurements before and after and from them it seems we’re looking at something around a 37% reduction in memory consumption during compilation of CORE.setting compared to the previous release. Since we use the standard Perl 6 compiler to compile the setting, this means the improvement applies to compilation of scripts and modules in general.

While I was fairly comfortable I could get memory usage down some amount, I was a bit concerned that we may take a speed hit. I mean, this work has involved re-writing a bunch of code written in low-level PIR into rather higher level NQP. That carries obvious risks of a slow down – especially since NQP currently lacks an optimizer. Happily, though, we’ve actually got a little bit faster at compilation. This is thanks to more efficient data structures (of note, using natively typed attributes), the AST being a little better angled towards the needs of Perl 6 compilation, and through an implementation that learned from PAST compilation profiles. The most dramatic speedup was to the optimization phase, which now completes in a third of the time it used to! The gains in other phases are much, much more modest, but it’s still movement in the right direction in a project that carried a real risk of a regression speed wise.

While this work has been merged, this isn’t the end of the road for QAST-related tasks. Here’s what I will be looking at from here, both in time for the August release and probably a bit beyond that:

  • Getting inlining of routines back in place; this is a re-implementation rather than an update. Happily, the re-implementation will be much, much cleaner and should allow for better inlining support. This is currently top of my todo list, so should be in place in the next couple of days.
  • Some more optimizations here and there, which should get us some more (albeit probably relatively smaller) wins in memory usage and speed.
  • Getting NQP switched over to using QAST.
  • Giving NQP natively typed variables, which should let us write a bunch of code more efficiently in the compiler.
  • Giving NQP an optimizer, which should hopefully get us some more speed wins.

In other exciting news, the arrival on QAST means masak++ is digging into taking macros some steps further. Maybe there’ll be something nice there for the next release too. :-)

Posted in Uncategorized | Leave a comment