Meta-programming slides, and some Rakudo news

I’m just back from a trip to the German Perl Workshop. Of course, the food was great and the beer was greater. :-) I was also very happy to see various Rakudo hackers in person; moritz++ was one of the organizers, and tadzik++ attended and gave a talk also. Much fun was had, and some hacking happened too. :-)

You may be interested to check out the slides from my talk on meta-programming in Perl 6. It covers some of the things we’ve been able to do for a while, as well as having a new example that shows off an interesting use of bounded serialization in combination with meta-programming. Also, I have now submitted this talk to the French Perl Workshop, which will take place in June. :-)

As for Rakudo progress, here’s some of the things that have been going on.

  • The bounded serialization branch has landed (and thus will be in the next release)! This has, as hoped, greatly decreased Rakudo’s startup time – an improvement that has been welcomed by a bunch of folks on #perl6. Module and setting pre-compilation is now also faster, along with loading of pre-compiled modules. Win all around.
  • Thanks to the serialization, various things have been unblocked. Amongst them, the constant declarator now takes non-literals on its RHS, and the BEGIN and CHECK phasers now work in r-value context. In both cases, the values are serialized if these things are done in a pre-compiled module.
  • masak++ merged his initial work on macros. While there’s much more to do in this area yet, it’s great that we now have the basics in place, enabling declarations of macros and quasis. I’m sure masak will blog about this soon – keep a lookout for it.
  • tadzik++ has got us some support for the Set, Bag, KeySet and KeyBag types.
  • moritz++ cleaned up match handling, and also implemented the :exhaustive and :nth match adverbs. So, that’s two out of three branches mentioned in my last post landed. :-)
  • Stunning progress on phasers! We now support ENTER, LEAVE, KEEP, UNDO and START. The LEAVE, KEEP and UNDO drew very heavily on work by mls++. Additionally, FIRST, NEXT and LAST are supported inside for loops (though not yet available in other loops). I’m fairly fired up to do PRE and POST too, though need some spec clarifications first.

moritz++ has also been continuing his excellent typed exceptions work; we discussed how to handle typed exceptions in the metamodel during the workshop, which should extend their reach to cover even more of the errors that can crop up.

So, all in all, lots of progress, and we’ve still just under two weeks to go until the March release. Having landed the bounded serialization work, I’ve been enjoying doing some feature development; right now, I have a branch that is muchly cleaning up name handling, improving name interpolation and getting various of the missing pseudo-packages in place. I expect to have this work completed in time to go in to the release also.

Beyond that, it’ll be time for me to dig back into some of the fundamentals again. Most immediately, that will be getting QRegex to be the engine we use to parse NQP and Perl 6 code (currently, we use QRegex for compiling the regexes and grammars you write in your Perl 6 programs, but still use the previous generation engine for parsing Perl 6 source). This will allow us to eliminate a vast swathe of Parrot PIR code that we depend on (the new regex compiler is written in NQP), aiding portability. I aim to land this in time for the April release; I’ve already done significant pieces of the work towards it.

Probably somewhat in parallel with that (though set to land after it), I’ll also be taking on our AST and AST compiler. This work will see it being ported from PIR to NQP, switched over to use 6model objects (which means we can use natively typed attributes to make the nodes more memory efficient, and also serialize the nodes as macros require) and given much better handling of native types. Generally I’m hoping for compilation performance improvements and better code generation. This will also eliminate our last large PIR dependency, thus clearing the final hurdle for making a real stab at targeting an extra backend.

One slight slowdown will be that I’m taking a vacation during the second half of March. I’m going to a land far away, and plan to spend most of my time seeing it, and checking out their food and drink. The laptop is coming, though, so it won’t be a complete stop; my stress levels are happily in fairly good shape right now, so I don’t need my vacation to be a complete break with normality, just a refreshing change of scene. :-)

Posted in Uncategorized | 2 Comments

Rakudo Star 2012.02 out – and the focus for 2012.03

I’ve just cut the February 2012 release of Rakudo Star. The good news is that we got a bunch of nice improvements into the release.

  • More typed exceptions, and better backtraces. This work has been done by moritz++; I’ll simply point you to his excellent blog post on that topic today. I think everyone working with Rakudo will appreciate the new backtraces, which cut out a lot of the uninformative details and are thus much more focused.
  • FatRat is now implemented. This type gives you rational number support with big integer semantics for both the numerator and denominator. The Rat type now correctly degrades to a Num if the denominator gets too large, as per spec (this avoids performance issues in certain cases). Again, we’ve moritz++ to thank for this.
  • We have object hashes! You can now make a declaration like:
    my %hash{Any};
    Which will have keys of type Any. You can constrain this as you wish. You can also still constrain the value in the usual way:
    my Int %hash{Any};
    This latter example gives you a hash with Int values and object keys, meaning that keys are not coerced to strings, as is the default. Thus, using %hash.keys will give you back the original objects. Identify is done based on .WHICH and thus ObjAt.
  • The coercion syntax, Int($foo), is now implemented. This gives you another – perhaps neater – way to write $foo.Int, but types may also implemented the method postcircumfix:<( )> if the target type is the one that knows how to do the coercion.
  • The reduction meta-operator used to have crazy bad performance, and parsed dodgy in some cases. Both of these issues are now resolved; it’s an order of magnitude faster and parses more correctly.
  • Various regex improvements and more problems caught at compile time, which I discussed in my previous post.

As always, there’s a bunch of bug fixes and other tweaks – see the Rakudo ChangeLog for more details.

Naturally, one release done simply means another one to work towards. :-) If you’ve been following along with my posts, you’ll have noticed that the bounded serialization work I’ve been doing didn’t make it in to the 2012.02 release; it just wasn’t done in time. The good news is that it’s well on course to land in very good time for the 2012.03 release.

Today I got to the milestone of having a Rakudo binary, with a serialized CORE.setting, that will compile and attempt the spectests. There’s a bunch of failures (which I expected), but a lot more passes than fails. Anyway, that means things are essentially working and I’m into triage mode. I’ve also got some very preliminary data on what kind of improvements it brings.

Just from trying it out, I could feel that startup was faster, as hoped. tadzik++ built the bounded serialization branch and the current main development branch and compared the time they took for a simple “hello world” (such a simple program that startup cost will dominate). The result: the bounded serialization branch ran it in around 25% of the time. I think we can squeeze a bit more out yet, but achieving startup in a quarter of the time we do today is a big improvement. Loading time for pre-compiled modules will see a similar improvement.

I also checked memory consumption when compiling CORE.setting. The bounded serialization branch uses 60% of the memory that the main development branch uses. I’d been hoping that this work would cut memory usage by around a third, and it’s done at least that. I didn’t measure the speedup for this task yet; while the other bits are fair comparisons, this one would not be yet even if I had a figure, since the optimizer is currently disabled (one of the transformations causes issues; I’m not expecting it to be a huge pain to resolve, but was more focused on the critical path to running spectests again so I could understand the fallout better).

Once the busted spectests are running again, I’ll get this merged, then dig in to exploiting some of the opportunities it makes available. I’m hoping we can get some more nice features in place for the next release also; moritz++ has already got branches for sink context and the :exhaustive modifier in regexes in progress, and I’ve got some ideas of things to hack on… :-)

Posted in Uncategorized | 1 Comment

Bounded serialization, better regexes and better errors

Here’s a quick roundup of some of the things that have been going on in Rakudo since the January release.

Bounded Serialization

This is where I’ve been focusing the majority of my efforts. At the moment, when we pre-compile Perl 6 code – such as the CORE setting with all the built-ins – we also build up a bunch of code that reconstructs the various objects that are made as a result of compile time declarations. This is done by recording an “event” for each thing that happens at compile time, then replaying them when the module/setting is loaded.

We’ve done things this way throughout every generation of Rakudo to date, but as I worked on the “nom” development branch and 6model, I built things so that we could later switch over to a different model; one where all of the objects created as a result of compile-time declarations are serialized. Then, when we need to load the module, we deserialize this blob of data back into the required objects, rather than doing all of the method calls and data shuffling to build them up again.

I’ve been working on this since we got the last release out. So far I’ve got the serialization and deserialization engine to a reasonably capable state; it can happily handle references between compilation units, has no problems with circular references between objects, and today I got basic support for serializing closures in place also. At this point, I’ve just got it being exercised by a bunch of tests; the next step will be to integrate it into NQP’s compilation process. I’m hoping I can get through that at the weekend, and after that it’ll be time to try and get Rakudo using it. Whether I get this landed for the February release or if we have to wait until the March one, I’m not yet sure. I know for sure I don’t want a half-baked version of it going out, so I’ll hold it for the March release if I can’t get it working as reliably as I want for the February one.

Why am I working on this? Here’s some of the things that I’m aiming at as a result of this work.

  • Improved startup time (the deserialization should be less work than rebuilding everything up from scratch)
  • Reduced memory during pre-compilation of modules. Most notably, I’m hoping to make a notable reduction in the memory (and time) required to build CORE.setting, which will most certainly be welcomed by anyone trying to build in a lower memory environment. The faster build will also be helpful for Rakudo developers, and enable us to be more productive.
  • The restrictions on the “constant” declarator can be lifted, enabling use of non-literal values.
  • Phasers as r-values can be implemented.
  • We can implement constant folding much, much more easily, as well as build other immutables at compile time.
  • Other nice things, no doubt. :-)

Completing this will also be one prerequisite down for much better inlining optimization support.

More Regex Bits

We now support the use of <x> in a regex to also call any predeclared lexical regex “x”. The <Foo::Bar::baz> syntax for calling a rule from another grammar is also now supported. A nasty bug that caused <!> not to work has been fixed. Finally, <prior> – which matches the same string that the last successful match did – is now implemented.

Better Errors and Exceptions

moritz++ has continued this typed exception work. We’ve also been improving various bits of error reporting to be more informative, and catching a few more things at compile time. moritz++ has also ported STD’s $*HAS_SELF handling, which gives us better handling of knowing when operations needing an invocant can take place. We currently have this work in a branch (it depends on some other work I was doing to align us with changes to how STD parses initializers, and there’s one last bug that prevents us merging it just yet; it’ll be sorted out in the coming days, I’m sure).

Pod Bits

tadzik++ dropped by with a couple of patches, one a bug fix, the other pretty cute: it makes the auto-generated usage message for MAIN subs include the declarator documentation.


Thanks to moritz++, we now support copy and rename functions. There’s also been the usual range of bug fixes that we get into every release, steadily chipping away at making Rakudo better.

Posted in Uncategorized | 4 Comments

This month’s Rakudo Star release – and what’s coming next

So, we made it – a Rakudo Star release based on the “nom” development branch has landed. It’s based on the compiler release moritz++ cut earlier this week, and pmichaud++, masak++ and myself have been involved in getting the Star-specific build and installation scripts in shape.

Getting to this release has been a lot of work; in many sense this is a revolution in Rakudo’s development rather than a small evolutionary step over what we had before. It’s taken a while (indeed, longer than I’d first hoped) and has been a lot of work – but it was worth it. Not just because of the many improvements that are in this release, but because of the enormous future potential that we now have.

Here’s some of the things I’m happiest about in the release.

  • The performance improvements in many areas. Yes, we’ve plenty of work to do here – but this is a solid step forward for a wide range of scripts, and in some cases an order of magnitude improvement.
  • That 6model – something I started designing a year and a half ago – has not only cleanly supported all of the things we needed to do in Rakudo, but also opened up so many other doors. For example, the new NativeCall module uses its representation polymorphism support to great effect.
  • Protoregexes doing real NFA-driven Longest Token Matching rather than the cheating version we had before that only operated on literals.
  • The optimizer, along with the various extra compile time error reporting it gives. This will be an important future area for Rakudo.
  • Initial native type support, and bigint semantics for the Int type.
  • The POD6 support, thanks to tadzik++’s Google Summer of Code grant in summer.

So, what’s next? Currently I’m working hard on getting true bounded serialization support in place. This should further improve BEGIN time support (including constructs that depend on BEGIN time), greatly cut down resource consumption during CORE.setting compilation (both time and memory) and give us faster startup. It’s hard to guess at figures for the improvement, but I’m expecting it to be a noticeable improvement in all of these areas. I’m aiming at getting this landed for the next Rakudo compiler release (which I expect us to do a Star release based on too), though largely it depends on whether I can get it working and stable enough in time; while some parts are a simple matter or programming, other parts are tricky.

That aside, we’ve already got various other new features in the pipeline; even since last weekend’s compiler release, there are multiple new regex-related things in place, moritz++ has continued with his typed exceptions work, we’re catching a couple more errors at compile time rather than letting them slip through until runtime, and there’s various other miscellaneous bug fixes. Also, masak++ is working on macros in a branch, and I’m optimistic that we’ll have some initial macro support in place by the next release also. Busy times! :-)

Posted in Uncategorized | 6 Comments

Looking back, looking forward

So, 2012 is here, and here’s my first Perl 6 post of the year. Welcome! :-)

Looking Back

2011 brought us a faster Rakudo with vastly improved meta-programming capabilities, the first work on exploring native types in Perl 6, the start of a powerful type-driven optimizer and many other bits. It also took me to various conferences and workshops, which I greatly enjoyed. I’d like to take a moment to thank everyone involved in all of this!

This was all great, but slightly tainted by not managing to get a Rakudo Star distribution release out based on a compiler release with all of these improvements. I’d really hoped to get one out in December. So what happened? Simply, there were a certain set of things I wanted to get in place, and while many of them got done, they didn’t all happen. While the compiler releases are time based – we do one every month – the distribution releases are more about stability and continuity. By the time I headed back to the UK to spend Christmas with family, we were still missing a some things I really wanted before a Star release was done. Given the first thing that happened when I started relaxing a little was that I immediately got unwell, I figured I should actually use my break as, well, a break – and come back recharged. So, I did that.

So, let’s try again

So, the new goal is this month. I’m happy to report that in the week since I’ve got back to things, one of the things I really wanted to sort out is now done: Zavolaj, the native calling library, now does everything the pre-6model version of it did. In fact, it does a heck of a lot more. It’s even documented now! It’s also far cleaner; the original implementation was built in the space of a couple of days with mberends++ while I was moving apartment, and was decidedly hacky in places. The missing bits of the NativeCall library were important because they are depended on by MiniDBI, and I really didn’t want to ship a Rakudo Star that can’t connect to a database. So, next up is to make sure that is in working order. I’m not expecting that to be difficult.

That aside, there were some things to worry about in Rakudo itself. I’ve dealt with some of those things in the last week, and perhaps the one important remaining thing I want to deal with before Star is a nasty regex engine backtracking related bug (I’ve been hoping pmichaud++, the regex engine guru, might appear and magic it away, but it seems it’s going to fall on my plate). But overall, we’re well on track to cut the Star release this month.

What’s the direction for the year ahead?

During 2011, Rakudo underwent a significant overhaul. It was somewhat painful, at times decidedly not much fun, but ultimately has been very much worth it: many long standing issues have been put to rest, performance has been improved and many things that were once hard to do are now relatively easy or at least accessible.

I think it goes without saying that we won’t be doing any such wide-ranging overhaul in 2012. :-) The work in 2011 has opened many doors that we have yet to walk through, and 2012 will see us doing that.

At a high level, here’s what I want:

  • Less bugs, more stability: mostly this is about continuing to work through the ticket queue and fix things, adding tests as bugs are fixed to ensure they stay fixed.
  • Better error reporting: there are things in STD, the Perl 6 standard grammar, that allow it to give much more informative error reports on syntax errors than we often can in Rakudo today. I want us to bring these things into Rakudo. Additionally, there’s plenty of improvements to be made in runtime errors. Furthermore, I want to expand the various bits of static analysis that I have started doing in the optimizer to catch a much wider range of errors at compile time.
  • Run programs faster: this process is helped by having decent profiling support these days. There’s a lot more to be done here; the optimizer will help, as will code generation improvements.
  • Compile programs faster: this will come from more efficient parsing and greatly improving the quality of the code NQP generates (NQP is the language we write much of the compiler in)
  • Shorter startup time: this mostly involves finishing the bounded serialization work up. I think the best way to describe this stuff is “fiddly”.
  • Use less memory: Rakudo has got faster in no small part thanks to being able to understand its performance through profiling. Our understanding of its memory consumption is much more limited, which makes it harder to target improvements. That said, I’ve some good guesses, and some ideas for analyzing the situation.
  • More features: while being able to do the things Rakudo can do today faster and with less bugs would give a very usable language for quite a lot of tasks, there’s still various features to come. In particular, Rakudo’s support for S05 and S09 needs work.
  • VM Portability: we’ve made good progress towards being able to make a serious stab at this over the last year, while at the same time also managing to perform vastly better on Parrot too. With help from kshannon++, I’m currently working on completing the switch to QRegex (a regex engine with NFA-powered LTM, and written in NQP rather than PIR), which should carry on a pattern of simultaneously increasing performance and portability. Beyond that will be an overhaul of our AST and code generation, with the same goal in mind.

So, lot’s of exciting things coming up, and I look forward to blogging about it here. :-)

A way to help

There are many ways to get involved, depending on what you’re interested in. One way is to take a look at our fixed bugs that need tests writing. At the time of writing, just short of 100 tickets are in this state. No guts-y knowledge needed for this – you just need to understand enough Perl 6 (or be willing to learn enough) to know what the ticket is about and how to write a test for it. Drop by #perl6 for hints. :-)

Posted in Uncategorized | 3 Comments

Rakudo: this week’s release, and the next Rakudo Star

On Thursday, tadzik++ cut release #46 of Rakudo. This time, we named it for the London Perl Mongers, organizers of this year’s outstanding and very well attended London Perl Workshop. I’d not been to one for a couple of years, and I’d forgotten what a fun event it is. This one felt better than ever. So, thanks folks! :-)

So, what was in the Thursday release?

  • Big Integer Support: the Int and Rat type now support very large values without loss of precision/coercing to floating point. This means you can do such things as compute factorial to 1000 (which stringifies to 2568 characters, by the way :-)). You don’t need to get any libraries set up for this – we bundle a slightly extended libtommath with NQP, and it’s exposed by some ops and an extra 6model representation, which the Int type inlines into its body (so Int is still a single garbage collectable object at the VM level). Thanks go to moritz++ for doing much of the work on this.
  • Protoregexes with LTM: it’s taken us a lot longer to get protoregexes back into the nom development branch than we’d initially hoped. Here’s the story. A while ago, pmichaud++ worked on real Longest Token Matching support – driven by an NFA – in a re-development of the regex engine. We’d only ever had “cheating” protoregexes in Rakudo before – ones that could work on literal prefixes, but nothing more. The improved engine is a really nice piece of work – amazingly little code for what it does, and very elegant. Sadly, pmichaud has not been able to work so much on Rakudo of late, so the work lay not-quite-done and unmerged for a while. Just ahead of the release, I picked it up, and found it was far enough along and sufficiently easy to get in to that I could get a first cut integrated in time for this month’s release. Since the release, along with diakopter++, I’ve been hacking away at regexy bits, so there’ll be many improvements in this area in next month’s release. It was nice to get something in place for this month’s, though.
  • CATCH improvements: mls++ did a bunch of work that made our handling of CATCH blocks far, far more in line with the Perl 6 specification. Naturally, this makes them much more useful; you now can write when blocks in your CATCH, and any exceptions you don’t match will get re-thrown to the next handler. Write a default block to swallow everything. We also do the stack unwinding at the correct point now. Great work!
  • Improved MAIN argument parsing: japhb++ has been leading the way on improving our support for the MAIN subroutine, including nice usage message generation. This is a really nice Perl 6 feature for writing command line applications, so it’s good to see work here. :-)
  • 6model REPR API 2: this is very much a behind the scenes thing, and deserves a post of its own for those who follow the blog in order to know more about metamodel design. However, in short: I re-worked the REPR API in 6model somewhat, and updated the various representations to use it. The immediate result was that we could efficiently have Perl 6’s Int type do big integer stuff. However, it’s also the foundation for some forthcoming work on compact structs and packed arrays. Hopefully at a user level, though, you’ll notice nothing at all about this change; it’s all guts. :-)
  • Various other fixes and improvements: bugs fixed, performance improved, missing stuff implemented…we had quite a few smaller, but very worthwhile things done in this release too.

So, this is great, but the next question on everybody’s mind is: when is a nom-based Rakudo Star release coming out? The answer – epic disasters aside – is “December”. Yes, next month. Two big things have happened in the last week: grammar support in the nom development branch drastically improved, and this unblocked tadzik++ to get our ecosystem tools (for module build and installation) working with it. The focus from here is…

  • Native call support: get our support for loading and calling into C libraries re-established. The NativeCall module basically worked…apart from it made a load of guts-y assumptions that don’t hold true any more, and was somewhat restricted in what it could handle (for multiple reasons). Now we have 6model and native type support, we can do far better. My task for over the next week is to get our native call support back in shape, all being well with a bunch more features.
  • More regex work: while we’ve come a really long way in recent days – we have many things working that never worked in the previous development branch – there are still some issues to address, including one nasty backtracking issue.
  • Fixing Modules: quite a few modules already work just fine on nom. Some block on native call support, others on the remaining regex stuff. Others will point to bugs or will be using outdated semantics or relying on previous bugs. tadzik++ has figured out how to automatically run through the module ecosystem, build modules and run their tests, so we can get a good sense of what needs doing.

And, of course, the usual round of feature addition, bug fixing and performance work.

And, with that, it’s back to the hacking. :-)

Posted in Uncategorized | 2 Comments

Slides from my Optimizing Rakudo Perl 6 talk

Over the weekend, I visited to Bratislava for a few days, the beautiful city I once called home. It felt oddly familiar, and I found myself noticing all kinds of little changes here and there – where one shop had given way to another, or a statue had appeared or changed. Happily, my favorite eating and watering holes were still there, and my sadly somewhat rusted Slovak language skills were still up to decoding menus and ordering tasty beer, and I did plenty of both. :-)

I was there was to attend the Twin City Perl Workshop. I repeated my Perl 6 grammars talk, and gave a new one about optimizing Rakudo. This included both the optimization work myself and others have been doing, but also some details about the optimizer. I also made a couple of nice diagrams of Rakudo’s overall architecture and what it does with a program.

You can get the slides here, or if you’re heading to the London Perl Workshop this coming Saturday, I’ll be delivering it there too. Enjoy! :-)

Posted in Uncategorized | Leave a comment