YAPC::Russia

OK, so let me get the obligatory meme out of the way first: “In Soviet Russia, the YAPC enjoys you!” Well, I hope that my talks were enjoyed anyway. :-)

Despite having been at every Perl Mova event in Ukraine and even having made it to the Vladivostok Perl Workshop, this was the first time I managed to make it to YAPC::Russia. That said, I have been in Moscow before, and it was nice to be back again. It’s a vast city with an almost equally vast metro. The marble stations, with their epic decorations, never fail to impress me. For the first time, I was also impressed by my airport experience: I arrived in a shiny new terminal with efficient immigration (parallel processing for the win). The girl who inspected and stamped my passport almost looked like she was capable of smiling! :-) Getting the visa to go to Russia was the usual hassle, but an hour into my time there – surrounded by the Cyrillic script, grand buildings and the sounds of a Slavic language (my favorite language family) – it all felt worth it. :-)

The first day was a Perl 6 hackathon. Many people had never installed a Perl 6 implementation before, so masak and I helped people through that. Along the way, somebody’s spectest run found a platform-specific unicode related bug and with masak’s help they filed it. It was nice to see the hackathon quickly turning up results. Some people played with Perl 6 and modules; there was some work on Perl 6′s LWP module. We got to the bottom of why Zavolaj broke recently and I was able to write to the Parrot list about the issue. With a little guidance, a performance-boosting patch for NQP, which I’ve now also been able to use in the nom branch of Rakudo, was also developed at the hackathon. I also had chance to write a couple of patches myself.

Days two and three were “normal” conference days. After Andrew Shitov gave the opening, Carl and I took to the stage to talk about what had been done at the previous day’s hackathon. I decided to try and introduce myself in Russian. I’m normally not great at it, and was very nervous about speaking in front of a bunch of people in a language I’m not especially good at. Afterwards I was told that while my pronunciation was quite accented, I spoke pretty much correctly and it was easy to understand. Phew! :-) Some day, I hope to do a lightning talk in Russian…at this rate, it may be a few years yet though…

Later that day I had my first real talk, which was a re-run of “Inside a Compiler”, but with a few updates given that NQP has been tweaked a bit since I gave it in Taiwan (back then, I was still deep in the middle of a bunch of changes). My second talk was on the final day of the conference, and was essentially a Rakudo progress report, very much like the one I gave at the Netherlands Perl Workshop. The difference this time was that I’m actually actively working on a bunch of the optimizations and improvements that the talk discusses, whereas back at NLPW it was more forward-looking. Now I’ve given it in the future and present tense, I look forward to giving it in the past tense, once all of the things it describes are completed. :-) You can find the slides and code here. By the way, masak’s talk – on the value of compiler-ish techniques in everyday programming – was excellent; I’m sure he’ll post slide links in his blog soon.

After YAPC, masak and I headed off to Saint Petersburg for a few days of vacation with a little hacking here and there. Due to Saint Petersburg being full of many wonderful things to look at, along with the discovery of a “Beer Restaurant” (yes, that’s how it actually described itself…even better, it was called “Pivorama”), not a great deal of hacking got done. Well, I don’t get to be in Russia all that often, so I was all for enjoying seeing it. :-)

Anyway, time to get back to the Rakudo “nom” branch hacking. It’s going pretty well, though there’s nothing hugely exciting to report just yet; I’m still busily re-building a bunch of the primitives. It’s feeling good, both in terms of the immediate benefits we’ll have, but also in the number of things it will open up for the future.

Rakudo on 6model gets underway!

Recently I finished up the long-running compile time meta-object branch in NQP, and got the final installation issues sorted out (there were issues having it and nqp-rx installed, which are now resolved). Many thanks go to moritz++ for various contributions to NQP over the recent month’s development, and to others for testing and feedback. This doesn’t mean I don’t intend to do much more with NQP – I do, and I’ll return to it in a month or two. However, now Rakudo is taking center stage.

So far, I’ve mostly just broken everything and then started putting it back together again. :-) At this point, anyone who was around when we did the alpha => ng transition (the last big Rakudo internals upgrade) will probably be thinking, “oh no, not again!” While this is a serious upgrade in some ways, it is also one that is somewhat less user facing. Yes, there will be fallout. But…

• The grammar is largely staying the same in terms of what it parses; we started over with that in the alpha => ng transition.
• The list implementation is largely staying the same – at least, semantically. Various details of how the implementation is factored will inevitably change, but in terms of semantics it shouldn’t be a user-facing change.
• A lot of the setting is written in Perl 6, and so won’t be greatly affected in the process.
• The meta-object API will be implemented totally differently underneath, but many aspects of the API will remain the same.

Some things will change drastically:

• Most immediately user facing, multiple dispatch is being massively overhauled to bring it in line with the latest specification. Much of the design work for that has been done already in NQP implementation of it. You may actually not notice the fallout from this in many cases, apart from if you were using proto subs or proto methods, in which case you’re in for some changes. Thankfully, this is relatively unusual. Multi-method semantics will also be different, but the amount of code affected in actuality will probably be relatively small.
• Separate compilation is another big deal, but most cases I’ve hit so far boil down to “add a missing use statement”.
• Many things that the compiler used to generate code to build or do will actually be built or done by the compiler at compile time. This includes meta-objects, code objects, constants and packages. Traits will be called during compile time. This is a massive advance that will allow us to implement a wide range of optimizations (v-table method calls, offset-based attribute access, compile-time multi selection, sub inlining) and do a bunch of static analysis. It only really affects you if you’re writing traits (unlikely) or meta-objects (so nobody yet).
• Package handling is changing a lot, but mostly this will just enable things that never worked before. It’ll be a bit stricter in some ways too. That bit me somewhat in NQP, but Rakudo has already been stricter than NQP, so again it’s probably not a big issue.

So, drastic under-the-hood changes that will deal with some long-standing issues and enable better code analysis and – critically – optimizations, as well as our building blocks being a bunch more optimal both in terms of runtime and memory. But I’m aiming to keep regressions low.

Nice goals, but what’s actually done so far? One big change is to the build procedure. Previously we did this:

1. Compile various bits (grammar, actions, other bits of the compiler) down to PIR.
2. Bundle those along with some PIR-based built-ins into a “stage 1″ compiler.
3. Use the stage 1 compiler to compile the setting to PIR.
4. Re-build what we did in step 2 but with the setting included this time.
5. Bundle it all into a single perl6 executable.

Now the process is as follows:

1. Build the various bits of the compiler (grammar, actions, symbol table handling, module loader, other such bits) into various bytecode files (via PIR). These are all written in NQP and handled by the NQP compiler. At most, there may be some embedded blocks of PIR here and there, but they’re rare and on the chopping block.
2. Compile a rather small perl6.p6 file, which is NQP, into the perl6 executable (via PIR and PBC). It has “use” statements that load the various other bits (which are pre-compiled to bytecode). End result: the Perl 6 executable is just a little frontend, without any setting or meta-objects in there.
3. Compile the meta-objects library. This is written in NQP.
4. Compile the setting, written in Perl 6, using the perl6 executable with –setting=NULL. It gets compiled into CORE.setting.pbc, which is what the perl6 executable goes looking for if you don’t use –setting=… to tell it you want otherwise. The first thing it does is “use Perl6::Metamodel”, to pull in the meta-objects.

And that’s it. No re-building the Perl 6 executable just for a change in the setting. Another result of this is that we can probably make it so Perl 6 programs compiled to bytecode will just load the setting, and it will only load the compiler if you actually need it (that is, if you do an eval). So, more efficient for development, more modular and more what you’d expect a Perl program to look like (that is, no magic PIR-driven bundling of stuff together, but instead use statements).

Symbol handling is being extensively refactored at the moment. I’m a lot of the way with that because it’s largely copy-paste-twiddle from NQP. Many of the meta-objects are fleshed out. Using custom meta-objects for built-in package declarators are supported, for the most part because there’s no built-in defaults any more (Perl6::Metamodel supplies a mapping, and the default setting just imports it). I’ve started on the bootstrap of the class hierarchy. I’ve got meta-objects being created and composed at compile time, but there’s no way to add attributes/methods yet. I’ve got initial native type bits in so we can do natively typed attributes (really soon) and other bits (later). You can’t actually *do* anything with it just yet, but things so far have come together fast.

The next big steps will be getting the new multiple dispatch in place, along with refactors to code objects. After that it’ll be trait handlers, methods and attributes. I suspect I’ll need a bit of scalar container refactoring along the way too.

After that, it’ll largely be “just” a process of gradually bringing the setting back into being, and writing meta-objects for stuff like roles, subsets, enumerations and so forth. No small task, but I think things will go much faster once I get to the point where it’s easy for people to jump in and help.

More soon – though I’ll soon also have the nice distraction of YAPC::Russia and then several days vacation after it. :-)

Separate compilation, package refactors and gradual typing: oh boy, what a mix!

First of all, I’m sorry that it’s been so long since I posted here. I’m happy to say that my NQP and Perl 6 involvement has been much higher than my post rate here. In fact, I managed to make it to the ever-wonderful Netherlands Perl Workshop and, excitingly, also made my first visit to OSDC.TW. I gave a few talks, of course; if you like the things I post here, they’ll probably be of interest.

• Rakud’oh: Making Our Compiler Smarter covered many of the current things my work is focusing on – essentially, work to make Rakudo a faster and more helpful Perl 6 implementation
• Implementing Classes in 15 Minutes is a crash-course on how to write a meta-object
• Inside A Compiler walks you through building a compiler using NQP and PCT. It touched on 6model too, as well as my goals for the future of these tools. I also published the code from the talk.

I also enjoyed a hackathon arranged my mberends++ after the NLPW. It was wonderful to meet tadzik for the first time, and masak came along also. And yes, there was beer. Good beer. :-)

So, what have I been hacking on? Last time I wrote here, I was digging into a set of thorny issues surrounding gradual typing, and grappling with the relationship between compile time and runtime. As I did so, a few other things came into the picture. It turns out that the issues I mentioned with regard to compilation unit purity were only scratching the surface. Following things along further, I realized that I was going to have to deal with the issue of separate compilation, and that this would require the package refactor that I’d once hoped could be done independently of the metamodel one.

Let me unpack these a little bit. The notion of separate compilation is that every time you start compiling a module, it gets a “clean slate”. No matter what other things the code that is loading the module has done – be it grammar tweaks or loading other modules – the module we’re compiling now should be isolated from it. This is critical if we intend to pre-compile modules. Obviously, this means that you kind of need to have parallel realities: the module’s view of GLOBAL is decidedly not going to contain whatever happens to be in GLOBAL by whatever does the “use” statement to load the module. Of course, they must be reconciled at some point, but not until that module’s compilation is completed.

In some languages, this may be relatively easy. For example, if you have completely distinct compile time and runtime phases and no custom meta-programming capabilities that the compiler may need to interact with, then you can maintain a symbol table in the compiler, do your code generation and be done. This is basically where Rakudo – and NQP – started out. We’ve come a long way with that approach, but getting to the next level needs a different approach.

To date, in NQP and Rakudo we’ve only had the capability to have one runtime reality. When I came to start bringing compile time and runtime into a much closer relationship, I quickly discovered that this was going to be deeply problematic. In fact, in NQP I had the acid test for these issues: a bootstrapping compiler. When a program is compiling itself, then there is 100% overlap between the symbols in the running program, and the program we’re compiling. Recall that the compilation may need to do a bit of runtime too; whenever you load a module, that module’s mainline is executed, for example. And if we’re going to be dealing with meta-object construction at compile time, then we’re actually going to have the compiler running code that’s part of the program that it is compiling. This doesn’t even factor in things Rakudo has to deal with that NQP does not, such as BEGIN blocks.

So, I’ve spent much of the time since I last wrote here working on these issues. Since this involved re-working packages, I’ve also dealt with a lot of the issues there. We can now have support for lexically scoped packages, for example.

my module Secret {
    our class Beer {
        method drink() { say("mmm...pivo!") }
    }
}
Secret::Beer.drink();

Here, the class Beer is decidedly installed in the package Secret, but the Secret package it is installed into is lexical. This means that while we can see that package inside the current lexical scope (say, the main lexical scope of our module), that package will not be visible outside of the module at all – unless you explicitly export it.

This is just one small example of things that were once scary hard features to try and implement, but are now relatively straightforward. I’ve also done a bunch of work on the concept of static lexpads, so we can get a bunch more symbol installation stuff correct.  This will also make things that were once hard much easier to support.

The really critical thing, however, is that the notion of separate compilation is now strongly supported. NQP can bootstrap itself, and it knows the difference between the HLL.pm it’s loaded and is currently running and the HLL.pm that it’s currently compiling. It also locates its current meta-object set via the lexical setting, importing them from it, so we get the correct incarnations of the meta-objects too, and can start making instances of them during the compile. In fact, a bunch of meta-object operations are now switched to being invoked at compile-time, and the rest will be coming along soon.

What does this mean? It means that the door is open to a whole bunch more compile-time cleverness. It means the compiler will work with the real meta-object, not just some simulation of it that it builds to try and catch some errors.  Oh, and I realized that it also means that I’ve opened the door to writing your own custom class meta-object in a module and then – on using that module – have it replace what the class keyword means by default.

Overall, this has been a more challenging piece of work than I had expected to face, but coming out of it, it feels like it’s been decidedly worth it. I’m aware that it looks like I’m spending a lot of time in NQP land and one might wonder how long it’ll take to get Rakudo using all of this stuff. Really, the bulk of the time has gone on solving hard problems and building solutions that will directly apply to Rakudo. All of this work – separate compilation, improved package handling, compile time meta-object handling and the starting point for lots of optimizations – will solve real issues Rakudo users are running into today, and also provide a framework to solve many more of them into the future.

My plan from here is to tie up the remaining work in the area of compile time meta-objects – there are a couple of hard problems remaining, but they’re at least quite isolated in scope – then return to the nom branch in Rakudo.

Smudging the compile-time/run-time boundary

I’m steadily plodding along with the NQP and Rakudo refactors. Things have slowed down a little since I last posted here, in part because work involved quite a bit of travel for a couple of weeks in a row, but also because I’ve needed some thinking time (actually, they went together well – sitting on a train for some hours provides good thinking time). The big challenges in the whole 6model journey so far have very rarely been writing the code; they’ve been in working out how to do things. And – with NQP all but switched over to 6model – I’d pretty much exhausted my stash of stuff I’d already worked out how to do.

After a while, things fell into place enough that I was able to write a roadmap for the nom branch in Rakudo – that is, the one where I switch it over to using 6model. I also set about starting to sketch out how the Rakudo metamodel will be factored, in code. In summary, most meta-objects are mainly made up of functionality composed in from roles. For example, both classes and roles will have methods, so it makes sense that the implementation of “having methods” is shared. These roles are implemented in NQP, and thus use the NQP role meta-objects; they provide a somewhat simplified form of Perl 6 roles, but plenty good enough for this task. For more on factoring a meta-model in terms of roles, see this excellent paper on the topic.

While it would be nice to now go full speed ahead and start re-working Rakudo to use these shiny new meta-objects, I’ve decided to hold off and work on some other issues first. The thing is, for months now I’ve been saying that part of the goal of 6model is to have a unified compile-time and runtime MOP, and at the moment I’ve got some serious blockers in the way of doing so.

One of those blockers is the notion of static lexpads. At the moment, the way we look at lexpads at compile time and runtime is completely separate. We can’t really go on that way, though, and need to start walking the path towards unification. The immediate place I run into this is with settings. At the moment, Rakudo has no clue what’s in the setting – that is, the outer lexical scope of the code it is compiling. Here are some examples of how this causes problems.

• When we are parsing, we need to know what is a valid type name. At the moment, we only look in the package, so things that should be “my” scoped in the setting currently have to be “our” scoped.
• When compiling a multi, we need to know if a proto is in the setting or if we need to auto-generate one. Without being able to know this, we can’t do operator overloading. Again, we’d have to paper over this by our-scoping the operators. That’s wrong.
• We can’t do any of the funky gradual typing based optimizations on method dispatch without having access to the meta-object for a type, and all the meta-objects for the built-in types have to be located through the setting.
• To have has int $!x result in natively typed storage, we need the attribute meta-object to hold a reference to the int type object, so the REPR can compute the storage allocation.

Some of these start to cross over into the second blocker: we need to be creating type objects and meta-objects during the compile, rather than just generating code to build them. Well, kinda. There are two situations.

1. We are compiling code and immediately running it. In this case, we create the meta-objects during the compile, then just need to do some fixups and run.
2. We are compiling the code to some intermediate language, or bytecode, then storing that so we can run it later on. In this case, the objects created at compile time need to be re-created or deserialized (where re-creation ends and deserialization starts is a blurry line, though the latter tends to imply a more data-driven approach).

This is another case where compile time and runtime need to start having a closer relationship than they currently do. Dealing with this is hard, and what I’m in digging in to at the moment. Or at least, trying to – but I’ve actually had to step away from it and worry about some other things first.

Today’s pain point is that we’ve been living with two definitions of “compilation unit” for a while now. We’ve relied on being able to feed multiple files to the NQP or Rakudo compiler, get some PIR files out and concatenate them. The immediate clue this is bad is “Perl 6 has no such mechanism”. We’ve essentially been epicly cheating on importation, and had a situation where what the compiler thinks there are a few separate compilation units, but when we go from PIR to bytecode it’s treated as one. Today I hit a point where that discrepancy bit hard. Given it too relies on things being our-scoped rather than lexically scoped, I think it’s time to kill the PIR concatenation off once and for all, and do things more properly.

A compiler that’s going to be smart about the code it’s compiling and allow more optimization and static checking – not to mention one that follows the spec much more closely – gives a lot less wiggle room to cheat. We already passed the “make a feature-rich Perl 6 compiler people can play with and use for some stuff” goal with Rakudo Star; the next step is to make a Perl 6 implementation that is faster, more helpful, implements many of the missing features and has a much better handle on Perl 6′s language extensibility aims. The nom refactor of Rakudo is not only going to deliver some of those things, but is also a critical enabling step for achieving the rest of them.

NQP changes getting there; Rakudo next!

This week wasn’t quite as productive in terms of getting code written as the last few, though what did happen is noteworthy: native attribute support came along far enough that it is now used by Cursor to store its $!from and $!pos attributes. While the functionality isn’t yet exposed at NQP level, I’ve worked out with pmichaud++ how that can be arranged (there were a few possible ways to factor it) and it’ll land in the near future.

I also spent some time worrying exactly how to factor roles atop of 6model, and how to avoid certain problems in roles in Rakudo today – particularly with regard to parametricity (which is supported pretty well but has some icky rough edges in various cases). By this point I have a good idea of how to move forward and hope to have an implementation done in the next few days.

It will very soon be time to dig into the Rakudo changes. Here is a rough idea of how I think this will play out.

• Write new meta-objects against 6model that will cover the Perl 6 semantics of classes, attributes, parametric roles, concrete roles, role composition, role object mixins, subset types and native types.
• Get definitions of base types in the stage 1 compiler reworked to use the new meta-objects.
• Update multi-dispatch core to use the proto semantics.
• Get grammar and actions happy again, and update them to work with compiling towards the new metamodel. At this point, everything up to the setting should work.
• Get setting to build and work again.
• Fix up test regressions or tests based on bad assumptions.

That said, I expect bits of these will happen in parallel. It’s hard to put times against them, but my aim is to be there – or close – by the time I go to OSDC.TW at the end of March.

Well that was hard work: getting nqp-rx grammars using 6model

Another Sunday, another “what happened this week” post.

The big meaty task this week was to get grammars switched over to using 6model. This – as I had feared – proved rather trickier than doing the same thing for classes. My initial approach was to port some of Cursor to NQP and have the rest in NQP but with PIR method bodies. This turned out to be biting off a bit more than I was really up for chewing, and with a growing queue of yaks awaiting shaving I put down the razor and decided to take a more modest approach: keep the PIR code and switch it to use 6model, even if this was going to look a bit, well, ugly, and would mean that some 6model optimizations were not going to be so readily applicable.

That itself still proved a fairly tall order, mostly because unlike with classes the code I needed to change wasn’t neatly reified into bootstrap stages. Thus changing it to what it needed to be in order to work with 6model would break earlier stages in the bootstrap that shared the code. In the end, I realized I’d have to make copies of various things and install them with a different name.

The steps between there and something that works were many, and in the end all a bit of a blur. Amongst them, I had to modify various bits in Cursor to handle working with the new meta-model (especially the way the proto-regex implementation did introspection), deal with proto-regex prefix introspection methods not getting registered with the new meta-model, get some bits from the HLL::Grammar cheats moved into HLL::Grammar itself, re-write a couple of them into NQP, and hunt down and fix a few mis-translations as they manifested themselves.

As I worked my way from “hopeless” to “wow, it parses stuff again”, something very worrying started to become emerge. Things had got enormously slower as a result of the transition. We’re not talking a little slower: the nqp-rx test suite ran 12 times slower than it had before.

I at first wondered if the wrapping of various things that were on the “hot path” in nqp-rx method bodies could have done it. This does result in an exception handler to catch return exceptions being set up. I did a micro-benchmark in PIR and found that the extra setup work this caused led to the overhead of a sub invocations being 3 to 4 times greater. (This is a bad result from a Parrot perspective; since the routine has the same handler needed in every invocation then the setup work should really only need doing once ever and apply to all invocations of the sub, or otherwise be very cheap per invocation to the point it’s noise in a benchmark).

I figured this couldn’t be the whole story, but put in the obvious, easy to implement optimization: only add a return exception handler if return is used anywhere in the routine. It made a noticeable difference…but was hardly the real issue: now things were just 11 times slower than they had been rather than 12.

I recalled that when classes were switched to 6model I hadn’t noticed a slowdown. My attention turned to attribute access: cursor objects work a lot with attributes, whereas the compiler object and action classes do little. Could something be adrift there? I reviewed the P6opaque code and hit upon a problem fairly quickly. Something that I’d translated fairly directly from the C# implementation turned out to have been a really bad idea. While on .Net you can create a hash table that is type parameterized and should perform quite well, Parrot’s hashes aren’t type parametric. They want the keys to be strings. So if using an integer as a key – as I was – it first puts that into a PMC (creating a GCable) and then coerces that into a string (another GCable) to do the hash lookup. In other words, every attribute get and bind – which I’d intended to be cheap – was actually going and allocating two GC-able objects. “Oops.” I fixed things.

So was that it? Well, no. Still 9 times slower. Both of the above changes were worth doing, but something else was still dominating me. Out of curiosity, I wrote a micro-benchmark of Parrot method dispatch performance vs 6model method dispatch performance. I was expecting them to be pretty much neck and neck; none of the gradual typing optimizations that 6model can support are in place yet, so much faster would have been surprising. As soon as I saw the numbers, I knew I’d found the real problem. 6model was dispatching methods 50 times slower! When a grammar isn’t allocating and populating a gazillion Cursor objects, it’ll be busying itself calling methods. Lots of them. This was almost certainly to blame.

So how does method dispatch work in 6model? Well, in theory finding a method is an operation on the meta-object. So you call a method on the meta-object to find the method you want. Of course, this bottoms out somewhere with a primitive (that’s the KnowHOW meta-object). In reality doing that every dispatch is far too slow, so meta-objects that know ahead of time what code object a given method name will map to can construct and publish a cache: essentially just a hash mapping names to invokable things. (A v-table is similar, but it’s an array, mapping indexes to invokable things, saving having to do a hash lookup). Well, that should have been the reality, but it turns out the code that populated the cache had a really silly bug in it. A bug this silly.

With that fixed, the test run was consistently beating the time it used to take before I switched grammars and cursors to 6model. That’s more like it! And that’s before any of the changes go in that really start taking advantage of 6model’s strengths, plus clawing back some of the losses I know have been incurred by wrapping various things in HLL::Grammar in NQP bodies, which has introduced some wasteful boxing.

The side show to all of that work was that I finally managed to spend some real time on natively typed attributes this week. I’m certainly not all the way there yet, but I completed the initial enabling refactor and the allocation and REPR side of the work. The enabling refactor itself has delivered some immediate benefits: it means that there is now one less memory allocation per object. Compared to Parrot objects, 6model objects now result in a half the number of allocations and are malloc-less (object bodies use the Parrot fixed size allocator, which tends to be more efficient).

I’m currently at the point of needing to work out some of the remaining design issues for natively typed attributes before I can actually have something working and ready to show. Trying to think that through will be one of my main tasks in the week to come. I expect that once I hit upon the design, the implementation will be relatively little effort.

Overall, things are going well. My aim was to have everything in place to be ready to branch Rakudo and re-build its OO bits atop of 6model by early February, and I seem to be on target with that. At a push it may happen next weekend (that is, just within January), but mostly it just boils down to “when I feel I’m ready”. The main tasks to do in order to get there are:

• Resolve remaining design issues and finish getting a working implementation of natively typed attributes
• Write a basic roles implementation for nqp-rx
• Mix-in support (got most of the design in my head, just need to see if it can be implemented and work as easily as hoped)

Which is a pretty short list of “hard dependencies”.  There are, of course, other bits that are desirable, such as getting the new multi-dispatch work neatened up where it has rough edges, getting Parrot v-table integration implemented and dealing with the fact that match objects are still not using 6model (something that will become a problem later on). Those will all happen anyway over the next few weeks, but needn’t block the start on the Rakudo changes.

NQP and 6model: big steps forward on Parrot and JVM

Despite spending a chunk of the last week fighting off a pesky throat infection, it’s been pretty productive. Here’s a quick update on where things have got to.

Mostly, I’ve been working on getting nqp-rx (the Parrot implementation of the NQP language, which is the base that Rakudo is built on today) to use 6model. In my last post I set a goal that by the end of the month, all uses of the “class” keyword would be switched over to using 6model. Since NQP is bootstrapped, this is kind of a baptism by fire: to switch the class keyword to use 6model, it needs to be stable and complete enough to handle the use that the NQP compiler makes of classes. Happily, I’ve managed to pull this off during the last week – far sooner than I had expected. So if you build nqp-rx/nom today and run the tests, then you’re using 6model not only in the tests, but during the building of NQP itself.

That would be an exciting week in itself, but wait – there’s more! I also managed to port much of the multi-dispatch support introduced in the .Net CLR version of NQP/6model over to nqp-rx on Parrot too. The caching is missing, errors need improving and there’s the odd rough edge but overall it works. Both for multi subs and multi methods, with the correct proto semantics and including dispatch by arity, by type and using the :U/:D definedness constraints. This is a big step forward not only for nqp-rx, but also because it lays the foundation for introducing the new proto semantics and dispatch-by-definedness into Rakudo too.

This weekend, mberends++ came to visit and hack with me. Shortly after I announced my intention to work on 6model and an NQP implementation on the .Net CLR, he expressed an interest in porting that work to the JVM. While there’s been ported code showing up in the repo for a while, there’s never been anything actually runnable…until this weekend. Today, for the first time, 6model on JVM managed to run the meta-model bootstrap, a cut-down version of the NQP setting successfully initialized itself…and say(“hello world”) ran. \o/ Happily, a few test files run and pass already too.

Now that something actually runs, it’ll be a lot easier – and more rewarding – to hack on. A lot of the work is initially getting it up to the level of NQP on the .Net CLR – which of course will be a moving target. ;-) If you are interested in contributing to a Perl 6 on JVM implementation, this is a good time to jump on board; getting NQP running and bootstrapped on the JVM is the first step on the path to Rakudo on the JVM. Get in touch with mberends if you want to help.

On the subject of helping, if NQP and Rakudo on the .Net CLR is more (or also :-)) your thing, then see my file of low hanging fruit – tasks that should be relatively accessible and self-contained.

Finally, as planned, I wrote an overview of 6model. It discusses the motivation and scope, design principles and – at a high level – the architecture of it.

So, not a bad week. And what do I plan to do in the week to come? Mostly, work more on nqp-rx/nom:

• Tie up some of the multi-dispatch loose ends
• Get NQPClassHOW to publish a method cache, which should make things a bunch faster
• Get nqp-rx’s grammar keyword switched over to using 6model
• Start the planning for natively typed attributes

I’m also planning to do some more bits in NQP on the .Net CLR (of note, use the type context bits I added to avoid some more needless boxing/unboxing). Fun times! :-)