Since the Hackathon…

Last time I got around to writing here was while I was at the Oslo Hackathon. It was a truly great event: hugely productive, a great deal of fun and a real motivation booster too. I’d like to again thank, and especially Salve, Rune and Jan, for thinking to organize this, and then making a superb job of doing so. Everything ran smoothly, there was lots of undistracted hacking time, and a couple of evening dinner and beer outings provided chance to enjoy time together as a team in real life, without the restricted bandwidth IRC normally enforces on us.

After the hackathon, it was right back to $dayjob, though in a fun way: teaching Git for a couple of days. Less fun was the couple of days after, which involved a lot of pain followed by a lot of noisy drilling at my local dental clinic. It took some days beyond that before I stopped feeling unusually tired… Anyways, things are back to normal again, or as normal as things ever tend to look for me… :-)

So, what’s been going on in Rakudo land since the hackathon? Lots and lots, as it turns out. Most public-facing, the April 2012 Rakudo Star release landed; moritz++ is to thank for getting the release out of the door this month. It’s the first distribution release to incorporate the bounded serialization work, which delivers the much-improved startup time. While I’ve talked about many of the compiler side improvements in previous posts, Rakudo Star also includes modules, and we had some exciting additions in this release too: Bailador (a Perl 6 port of Dancer), LWP::Simple and JSON::RPC.

Of course, one nice release out the door means time to start working on things to make the next one interesting. Here’s some of what to expect.

  • Following on from removing “lib” from the default @*INC, the current working directory (“.”) is also gone now. While they may be nice for development, they are not good defaults from a security perspective when you actually run stuff in a production environment. Making this a more comfortable removal, as well as supplying search paths through the PERL6LIB environment (as has been supported for a long while), the -I command line option and “use lib” have now been implemented, and will be in the next release. moritz++ is primarily to thank for this work.
  • I realized that now we have LEAVE and UNDO phasers, I could implement “temp” and “let”. “temp” saves away the current value of a variable, and arranges for that value to be restored when the block is left, so any changes made to it will be undone. “let” is a way of making hypothetical changes to a variable; if the block is left due to an exception or if it evaluates to an undefined value, the change will be rolled back just like “temp”, but a successful completion of the block will leave the value in place. Both are now in place.
  • Several days back, moritz++ asked me for some inspiration of what to hack on. Amongst my suggestions was working on tagged imports/exports. And sure enough, with me throwing the odd commit in now and then, we have ’em working. Actually the main thing I did was improve the trait handling to support passing the tags to export; some work had been left undone awaiting real serialization, which we now have, so happily it was fairly straightforward to sort things out there.
  • pmichaud++ has been working on making things better if you build Parrot and Rakudo without ICU. Previously, any case-insensitive string match – even if it didn’t use anything beyond ASCII – would immediately die over a missing ICU. Now you can get away with that.
  • A while back I did a first cut implementation of “ff”, the flip-flop operator. It sorta worked, but we missed the “fff” form and, true to form, masak++ managed to find a bug to submit too. :-) Today I tossed the previous approach and re-did it, using a state variable for the underlying storage (since I did the first cut at it, the spec was updated to explicitly say that it should be done with a state variable). I also got the “fff” form in place. We now pass the vast majority of spectests for this feature.
  • A few days back, tadzik++ showed up with a snippet of code that ran insanely slowly. It was doing regex interpolation: / <$search_term> / or the like. Here, if $search_term is a string, it will be compiled to a regex and evaluated. Turns out that we weren’t caching that compilation with the string, though. So, if it had to scan or if you backtracked over the <$search_term>, then tried it again, you’d end up compiling it every time. A little caching later and things went rather faster. I was happy to have a profiler to hand; the call counts into the compiler very quickly showed what was up.
  • Our END phasers didn’t always run when you’d hope they did. In the last few days, moritz++ ensured they run on exceptional exits, and I made sure they run when execution is terminated using the “exit” function.
  • This evening, kboga++ showed up with a patch to turn Real, which ended up as a class during the early days of the “nom” development branch, into a role as it should be. This enabled custom numeric types. It also enabled us to run real-bridge.t, which brought and extra 200 spectests, pushing us nicely over the 22,000 passing spectests mark.
  • In various other fixes: //=, ||= and &&= now short-circuit properly, reduction meta-operators on list-associative ops do the right thing (pmichaud++), %*ENV propagates changes properly (tadzik++), ms// now works (moritz++) and enumeration types can now turn themselves into roles and be composed or mixed in to things.

There’s still over a week before the next compiler release, and I’m sure there will be some more things beyond this. It’s already looking like a nice bunch of improvements, though.

In other news, the work on QAST has also been chugging along. QAST is a replacement for PAST, which is a set of AST nodes together with something that maps them down to POST (Parrot Opcode Syntax Tree), which then becomes PIR code (which Parrot then turns into bytecode and runs). In essence, it’s part of the compiler guts. The not-quite-gone-yet old regex engine aside, PAST is the only significant part of our codebase that is still written in PIR (an assembly-ish language for Parrot). It also predates 6model, bounded serialization and many years of learned lessons. All that said, it gets a lot right, so much will stay the same.

QAST really is a simultaneous port to NQP, leaving out things we came to realize were bad or unrequired, adding much better 6model integration, unifying the notion of “ops” somewhat, improving native type support and taking advantage of native types during its implementation to get the AST nodes more compact, so we can save on memory during the compilation. Additionally, it will unblock masak++’s work on quasi splicing in macros, and is a step towards Rakudo targeting other backends being a reality rather than a nice idea. So, lots of win lies ahead…after the hard work of getting it landed. I’m hoping to make some significant steps forward on it during May.

Last but not least, masak++ and I will be heading over to the Bristol IT Megameet the weekend after next, giving a 30 minute joint talk on Perl 6, followed by a couple of hour tutorial. Looking forward to it – and hopefully I’ll be able to sneak a British ale in while I’m over there too. :-)

Posted in Uncategorized | Leave a comment

Hackathoning in Oslo

I’m in Oslo with a bunch of Perl 6 folks. It’s great to see old friends and meet some new ones – and we’re having a highly productive time. After a wonderful evening of tasty food and lovely beer (amongst others, a delicious imperial stout) yesterday, today has been solidly focused on Getting Stuff Done.

I’m a little tired now, so here’s just a quick rundown of some of what I’ve been up to.

  • Since bounded serialization landed, we’ve had a few issues with pre-compilation of MiniDBI, the database access module. I’ve now tracked down all of the remaining issues there, fixed them. and happily moritz++ has been hacking lots on improving the module in other ways too. The MySQL and Postgres drivers now pass all the tests we have for them, which is some nice progress.
  • I’ve been answering a few questions for arnsholt++, who has picked up the Zavolaj (native calling) module where I left off, adding support for passing/returning arrays of structs, structs pointing to arrays and various other permutations. This will greatly improve the range of C libraries that can be used with it.
  • I had a design session with pmichaud++ on QAST, the successor to our current AST. The new nodes will integrate far better with 6model and bounded serialization, give us better native type handling and be much more memory efficient due to being able to use natively typed attributes in them. This is also a key part of our work towards getting Rakudo up and running on an extra back end.
  • After that, I got the nodes fleshed out somewhat, and have started a little work on QAST::Compiler too. It’s underway!
  • I also spent some time in the ticket queue and fixed a bunch of Rakudo bugs: constant initializers containing blocks now work out, state declarations together with list assignment work, $.foo/ now contextualize as they should, and :i now also applies to interpolated variables.

So, lots of stuff – and that’s just the things I’ve been directly involved with.  It’s nice to be a part of this hive of activity…and tomorrow there’s another day of this! Catch you then. :-)

Posted in Uncategorized | 1 Comment

Back from vacation, hackathon coming up!

So, I’m back from vacation. Turns out Argentina is a pretty awesome place to vacation in, too. As well as wonderful food and delicious imperial stout (amongst other good beers), there was walking like this…

…and other cool stuff, like glaciers…

…and so even though the laptop came with me, it was just too much fun to be outside, especially when the weather was good so much of the time. I did sneak in a few patches, though, most notably implementing PRE and POST phasers.

Anyway, I’m safely back, after an 8 hour flight delay from Buenos Aires and a small bus accident at Frankfurt airport. Yes, this airport fails SO hard they managed to screw up the 2 minute bus trip from the plane to the terminal…anyway, I got off with just a few small cuts. Suggest taking the train to YAPC::Europe this summer… :-)

So, what’s coming up? Well, this month brings a Perl 6 hackathon in Oslo, where I look forward to being together with a bunch of other Rakudo and Perl 6 folks. I’m sure we’ll get some nice stuff done, and some future directions worked out. I’m happy that one of the most industrious Perl Mongers groups I know when it comes to organizing such events is also set in a very pleasant city situated in a beautiful country. :-) By the way, it’s still very possible (and very encouraged) to sign up if you want to come along.

As moritz++ noted on, we’re skipping doing a distribution (Star) release based on the March compiler release since an unfortunate bug slipped in that busted precompilation of modules that used NativeCall. We hold ourselves to higher standards of stability in the distribution releases (which are user focused) than the compiler ones (which just ship at the same time each month), and this would have been too big a regression. The good news is that I’ve patched the bug today, so we’re now all clear for doing an April one – and what a nice release it should be.

Well, time for dinner here – which I’ll be having with masak++. No doubt macros will come up, and what’s needed to get us along to the next level with those. Stay tuned; the next month should be interesting in Rakudo land. :-)

Posted in Uncategorized | 2 Comments

Meta-programming slides, and some Rakudo news

I’m just back from a trip to the German Perl Workshop. Of course, the food was great and the beer was greater. :-) I was also very happy to see various Rakudo hackers in person; moritz++ was one of the organizers, and tadzik++ attended and gave a talk also. Much fun was had, and some hacking happened too. :-)

You may be interested to check out the slides from my talk on meta-programming in Perl 6. It covers some of the things we’ve been able to do for a while, as well as having a new example that shows off an interesting use of bounded serialization in combination with meta-programming. Also, I have now submitted this talk to the French Perl Workshop, which will take place in June. :-)

As for Rakudo progress, here’s some of the things that have been going on.

  • The bounded serialization branch has landed (and thus will be in the next release)! This has, as hoped, greatly decreased Rakudo’s startup time – an improvement that has been welcomed by a bunch of folks on #perl6. Module and setting pre-compilation is now also faster, along with loading of pre-compiled modules. Win all around.
  • Thanks to the serialization, various things have been unblocked. Amongst them, the constant declarator now takes non-literals on its RHS, and the BEGIN and CHECK phasers now work in r-value context. In both cases, the values are serialized if these things are done in a pre-compiled module.
  • masak++ merged his initial work on macros. While there’s much more to do in this area yet, it’s great that we now have the basics in place, enabling declarations of macros and quasis. I’m sure masak will blog about this soon – keep a lookout for it.
  • tadzik++ has got us some support for the Set, Bag, KeySet and KeyBag types.
  • moritz++ cleaned up match handling, and also implemented the :exhaustive and :nth match adverbs. So, that’s two out of three branches mentioned in my last post landed. :-)
  • Stunning progress on phasers! We now support ENTER, LEAVE, KEEP, UNDO and START. The LEAVE, KEEP and UNDO drew very heavily on work by mls++. Additionally, FIRST, NEXT and LAST are supported inside for loops (though not yet available in other loops). I’m fairly fired up to do PRE and POST too, though need some spec clarifications first.

moritz++ has also been continuing his excellent typed exceptions work; we discussed how to handle typed exceptions in the metamodel during the workshop, which should extend their reach to cover even more of the errors that can crop up.

So, all in all, lots of progress, and we’ve still just under two weeks to go until the March release. Having landed the bounded serialization work, I’ve been enjoying doing some feature development; right now, I have a branch that is muchly cleaning up name handling, improving name interpolation and getting various of the missing pseudo-packages in place. I expect to have this work completed in time to go in to the release also.

Beyond that, it’ll be time for me to dig back into some of the fundamentals again. Most immediately, that will be getting QRegex to be the engine we use to parse NQP and Perl 6 code (currently, we use QRegex for compiling the regexes and grammars you write in your Perl 6 programs, but still use the previous generation engine for parsing Perl 6 source). This will allow us to eliminate a vast swathe of Parrot PIR code that we depend on (the new regex compiler is written in NQP), aiding portability. I aim to land this in time for the April release; I’ve already done significant pieces of the work towards it.

Probably somewhat in parallel with that (though set to land after it), I’ll also be taking on our AST and AST compiler. This work will see it being ported from PIR to NQP, switched over to use 6model objects (which means we can use natively typed attributes to make the nodes more memory efficient, and also serialize the nodes as macros require) and given much better handling of native types. Generally I’m hoping for compilation performance improvements and better code generation. This will also eliminate our last large PIR dependency, thus clearing the final hurdle for making a real stab at targeting an extra backend.

One slight slowdown will be that I’m taking a vacation during the second half of March. I’m going to a land far away, and plan to spend most of my time seeing it, and checking out their food and drink. The laptop is coming, though, so it won’t be a complete stop; my stress levels are happily in fairly good shape right now, so I don’t need my vacation to be a complete break with normality, just a refreshing change of scene. :-)

Posted in Uncategorized | 2 Comments

Rakudo Star 2012.02 out – and the focus for 2012.03

I’ve just cut the February 2012 release of Rakudo Star. The good news is that we got a bunch of nice improvements into the release.

  • More typed exceptions, and better backtraces. This work has been done by moritz++; I’ll simply point you to his excellent blog post on that topic today. I think everyone working with Rakudo will appreciate the new backtraces, which cut out a lot of the uninformative details and are thus much more focused.
  • FatRat is now implemented. This type gives you rational number support with big integer semantics for both the numerator and denominator. The Rat type now correctly degrades to a Num if the denominator gets too large, as per spec (this avoids performance issues in certain cases). Again, we’ve moritz++ to thank for this.
  • We have object hashes! You can now make a declaration like:
    my %hash{Any};
    Which will have keys of type Any. You can constrain this as you wish. You can also still constrain the value in the usual way:
    my Int %hash{Any};
    This latter example gives you a hash with Int values and object keys, meaning that keys are not coerced to strings, as is the default. Thus, using %hash.keys will give you back the original objects. Identify is done based on .WHICH and thus ObjAt.
  • The coercion syntax, Int($foo), is now implemented. This gives you another – perhaps neater – way to write $foo.Int, but types may also implemented the method postcircumfix:<( )> if the target type is the one that knows how to do the coercion.
  • The reduction meta-operator used to have crazy bad performance, and parsed dodgy in some cases. Both of these issues are now resolved; it’s an order of magnitude faster and parses more correctly.
  • Various regex improvements and more problems caught at compile time, which I discussed in my previous post.

As always, there’s a bunch of bug fixes and other tweaks – see the Rakudo ChangeLog for more details.

Naturally, one release done simply means another one to work towards. :-) If you’ve been following along with my posts, you’ll have noticed that the bounded serialization work I’ve been doing didn’t make it in to the 2012.02 release; it just wasn’t done in time. The good news is that it’s well on course to land in very good time for the 2012.03 release.

Today I got to the milestone of having a Rakudo binary, with a serialized CORE.setting, that will compile and attempt the spectests. There’s a bunch of failures (which I expected), but a lot more passes than fails. Anyway, that means things are essentially working and I’m into triage mode. I’ve also got some very preliminary data on what kind of improvements it brings.

Just from trying it out, I could feel that startup was faster, as hoped. tadzik++ built the bounded serialization branch and the current main development branch and compared the time they took for a simple “hello world” (such a simple program that startup cost will dominate). The result: the bounded serialization branch ran it in around 25% of the time. I think we can squeeze a bit more out yet, but achieving startup in a quarter of the time we do today is a big improvement. Loading time for pre-compiled modules will see a similar improvement.

I also checked memory consumption when compiling CORE.setting. The bounded serialization branch uses 60% of the memory that the main development branch uses. I’d been hoping that this work would cut memory usage by around a third, and it’s done at least that. I didn’t measure the speedup for this task yet; while the other bits are fair comparisons, this one would not be yet even if I had a figure, since the optimizer is currently disabled (one of the transformations causes issues; I’m not expecting it to be a huge pain to resolve, but was more focused on the critical path to running spectests again so I could understand the fallout better).

Once the busted spectests are running again, I’ll get this merged, then dig in to exploiting some of the opportunities it makes available. I’m hoping we can get some more nice features in place for the next release also; moritz++ has already got branches for sink context and the :exhaustive modifier in regexes in progress, and I’ve got some ideas of things to hack on… :-)

Posted in Uncategorized | 1 Comment

Bounded serialization, better regexes and better errors

Here’s a quick roundup of some of the things that have been going on in Rakudo since the January release.

Bounded Serialization

This is where I’ve been focusing the majority of my efforts. At the moment, when we pre-compile Perl 6 code – such as the CORE setting with all the built-ins – we also build up a bunch of code that reconstructs the various objects that are made as a result of compile time declarations. This is done by recording an “event” for each thing that happens at compile time, then replaying them when the module/setting is loaded.

We’ve done things this way throughout every generation of Rakudo to date, but as I worked on the “nom” development branch and 6model, I built things so that we could later switch over to a different model; one where all of the objects created as a result of compile-time declarations are serialized. Then, when we need to load the module, we deserialize this blob of data back into the required objects, rather than doing all of the method calls and data shuffling to build them up again.

I’ve been working on this since we got the last release out. So far I’ve got the serialization and deserialization engine to a reasonably capable state; it can happily handle references between compilation units, has no problems with circular references between objects, and today I got basic support for serializing closures in place also. At this point, I’ve just got it being exercised by a bunch of tests; the next step will be to integrate it into NQP’s compilation process. I’m hoping I can get through that at the weekend, and after that it’ll be time to try and get Rakudo using it. Whether I get this landed for the February release or if we have to wait until the March one, I’m not yet sure. I know for sure I don’t want a half-baked version of it going out, so I’ll hold it for the March release if I can’t get it working as reliably as I want for the February one.

Why am I working on this? Here’s some of the things that I’m aiming at as a result of this work.

  • Improved startup time (the deserialization should be less work than rebuilding everything up from scratch)
  • Reduced memory during pre-compilation of modules. Most notably, I’m hoping to make a notable reduction in the memory (and time) required to build CORE.setting, which will most certainly be welcomed by anyone trying to build in a lower memory environment. The faster build will also be helpful for Rakudo developers, and enable us to be more productive.
  • The restrictions on the “constant” declarator can be lifted, enabling use of non-literal values.
  • Phasers as r-values can be implemented.
  • We can implement constant folding much, much more easily, as well as build other immutables at compile time.
  • Other nice things, no doubt. :-)

Completing this will also be one prerequisite down for much better inlining optimization support.

More Regex Bits

We now support the use of <x> in a regex to also call any predeclared lexical regex “x”. The <Foo::Bar::baz> syntax for calling a rule from another grammar is also now supported. A nasty bug that caused <!> not to work has been fixed. Finally, <prior> – which matches the same string that the last successful match did – is now implemented.

Better Errors and Exceptions

moritz++ has continued this typed exception work. We’ve also been improving various bits of error reporting to be more informative, and catching a few more things at compile time. moritz++ has also ported STD’s $*HAS_SELF handling, which gives us better handling of knowing when operations needing an invocant can take place. We currently have this work in a branch (it depends on some other work I was doing to align us with changes to how STD parses initializers, and there’s one last bug that prevents us merging it just yet; it’ll be sorted out in the coming days, I’m sure).

Pod Bits

tadzik++ dropped by with a couple of patches, one a bug fix, the other pretty cute: it makes the auto-generated usage message for MAIN subs include the declarator documentation.


Thanks to moritz++, we now support copy and rename functions. There’s also been the usual range of bug fixes that we get into every release, steadily chipping away at making Rakudo better.

Posted in Uncategorized | 4 Comments

This month’s Rakudo Star release – and what’s coming next

So, we made it – a Rakudo Star release based on the “nom” development branch has landed. It’s based on the compiler release moritz++ cut earlier this week, and pmichaud++, masak++ and myself have been involved in getting the Star-specific build and installation scripts in shape.

Getting to this release has been a lot of work; in many sense this is a revolution in Rakudo’s development rather than a small evolutionary step over what we had before. It’s taken a while (indeed, longer than I’d first hoped) and has been a lot of work – but it was worth it. Not just because of the many improvements that are in this release, but because of the enormous future potential that we now have.

Here’s some of the things I’m happiest about in the release.

  • The performance improvements in many areas. Yes, we’ve plenty of work to do here – but this is a solid step forward for a wide range of scripts, and in some cases an order of magnitude improvement.
  • That 6model – something I started designing a year and a half ago – has not only cleanly supported all of the things we needed to do in Rakudo, but also opened up so many other doors. For example, the new NativeCall module uses its representation polymorphism support to great effect.
  • Protoregexes doing real NFA-driven Longest Token Matching rather than the cheating version we had before that only operated on literals.
  • The optimizer, along with the various extra compile time error reporting it gives. This will be an important future area for Rakudo.
  • Initial native type support, and bigint semantics for the Int type.
  • The POD6 support, thanks to tadzik++’s Google Summer of Code grant in summer.

So, what’s next? Currently I’m working hard on getting true bounded serialization support in place. This should further improve BEGIN time support (including constructs that depend on BEGIN time), greatly cut down resource consumption during CORE.setting compilation (both time and memory) and give us faster startup. It’s hard to guess at figures for the improvement, but I’m expecting it to be a noticeable improvement in all of these areas. I’m aiming at getting this landed for the next Rakudo compiler release (which I expect us to do a Star release based on too), though largely it depends on whether I can get it working and stable enough in time; while some parts are a simple matter or programming, other parts are tricky.

That aside, we’ve already got various other new features in the pipeline; even since last weekend’s compiler release, there are multiple new regex-related things in place, moritz++ has continued with his typed exceptions work, we’re catching a couple more errors at compile time rather than letting them slip through until runtime, and there’s various other miscellaneous bug fixes. Also, masak++ is working on macros in a branch, and I’m optimistic that we’ll have some initial macro support in place by the next release also. Busy times! :-)

Posted in Uncategorized | 6 Comments