I’m steadily plodding along with the NQP and Rakudo refactors. Things have slowed down a little since I last posted here, in part because work involved quite a bit of travel for a couple of weeks in a row, but also because I’ve needed some thinking time (actually, they went together well – sitting on a train for some hours provides good thinking time). The big challenges in the whole 6model journey so far have very rarely been writing the code; they’ve been in working out how to do things. And – with NQP all but switched over to 6model – I’d pretty much exhausted my stash of stuff I’d already worked out how to do.
After a while, things fell into place enough that I was able to write a roadmap for the nom branch in Rakudo – that is, the one where I switch it over to using 6model. I also set about starting to sketch out how the Rakudo metamodel will be factored, in code. In summary, most meta-objects are mainly made up of functionality composed in from roles. For example, both classes and roles will have methods, so it makes sense that the implementation of “having methods” is shared. These roles are implemented in NQP, and thus use the NQP role meta-objects; they provide a somewhat simplified form of Perl 6 roles, but plenty good enough for this task. For more on factoring a meta-model in terms of roles, see this excellent paper on the topic.
While it would be nice to now go full speed ahead and start re-working Rakudo to use these shiny new meta-objects, I’ve decided to hold off and work on some other issues first. The thing is, for months now I’ve been saying that part of the goal of 6model is to have a unified compile-time and runtime MOP, and at the moment I’ve got some serious blockers in the way of doing so.
One of those blockers is the notion of static lexpads. At the moment, the way we look at lexpads at compile time and runtime is completely separate. We can’t really go on that way, though, and need to start walking the path towards unification. The immediate place I run into this is with settings. At the moment, Rakudo has no clue what’s in the setting – that is, the outer lexical scope of the code it is compiling. Here are some examples of how this causes problems.
- When we are parsing, we need to know what is a valid type name. At the moment, we only look in the package, so things that should be “my” scoped in the setting currently have to be “our” scoped.
- When compiling a multi, we need to know if a proto is in the setting or if we need to auto-generate one. Without being able to know this, we can’t do operator overloading. Again, we’d have to paper over this by our-scoping the operators. That’s wrong.
- We can’t do any of the funky gradual typing based optimizations on method dispatch without having access to the meta-object for a type, and all the meta-objects for the built-in types have to be located through the setting.
- To have has int $!x result in natively typed storage, we need the attribute meta-object to hold a reference to the int type object, so the REPR can compute the storage allocation.
Some of these start to cross over into the second blocker: we need to be creating type objects and meta-objects during the compile, rather than just generating code to build them. Well, kinda. There are two situations.
- We are compiling code and immediately running it. In this case, we create the meta-objects during the compile, then just need to do some fixups and run.
- We are compiling the code to some intermediate language, or bytecode, then storing that so we can run it later on. In this case, the objects created at compile time need to be re-created or deserialized (where re-creation ends and deserialization starts is a blurry line, though the latter tends to imply a more data-driven approach).
This is another case where compile time and runtime need to start having a closer relationship than they currently do. Dealing with this is hard, and what I’m in digging in to at the moment. Or at least, trying to – but I’ve actually had to step away from it and worry about some other things first.
Today’s pain point is that we’ve been living with two definitions of “compilation unit” for a while now. We’ve relied on being able to feed multiple files to the NQP or Rakudo compiler, get some PIR files out and concatenate them. The immediate clue this is bad is “Perl 6 has no such mechanism”. We’ve essentially been epicly cheating on importation, and had a situation where what the compiler thinks there are a few separate compilation units, but when we go from PIR to bytecode it’s treated as one. Today I hit a point where that discrepancy bit hard. Given it too relies on things being our-scoped rather than lexically scoped, I think it’s time to kill the PIR concatenation off once and for all, and do things more properly.
A compiler that’s going to be smart about the code it’s compiling and allow more optimization and static checking – not to mention one that follows the spec much more closely – gives a lot less wiggle room to cheat. We already passed the “make a feature-rich Perl 6 compiler people can play with and use for some stuff” goal with Rakudo Star; the next step is to make a Perl 6 implementation that is faster, more helpful, implements many of the missing features and has a much better handle on Perl 6’s language extensibility aims. The nom refactor of Rakudo is not only going to deliver some of those things, but is also a critical enabling step for achieving the rest of them.