Those weeks: much progress!

This post is an attempt to summarize the things I worked on since the last weekly report up until the end of last week (so, through to Friday 25th). Then I’ll get back to weekly. :-)


The Great List Refactor accounted for a large amount of the time I spent. Last time I wrote here, I was still working on my prototype. That included exploring the hyper/race design space for providing for data parallel operations. I gave an example of this in my recent concurrency talk. I got the prototype far enough along to be comfortable with the API and to have a basic example working and showing a wallclock time win when scaling over multiple cores – and that was as far as I went. Getting the GLR from prototype mode to something others could help with was far more pressing.

Next came the long, hard, slog of getting the new List and Array types integrated into Rakudo proper. It was pleasant to start out by tossing a bunch of VM-specific things that were there “for performance” that were no longer going to be needed under the new design – reducing the amount of VM-specific code in Rakudo (we already don’t have much of it anyway, but further reductions are always nice). Next, I cleared the previous list implementation from CORE.setting and put the new stuff in place, including all of the new iterator API. And then it was “just” a case of getting the setting to compile again. You may wonder why getting it to even compile is challenging. The answer is that the setting runs parts of itself while it compiles (such is the nature of bootstrapping).

Once I got things to a point where Rakudo actually could build and run its tests again, others felt confident to jump in. And jump in they did – especially nine++. That many people were able to dive in and make improvements was a great relief – not only because it meant we could get the job done sooner, but because it told me that the new list and iterator APIs were easy for people to understand and work with. Seeing other people pick up my design, get stuff done with it, and make the right decisions without needing much guidance, is a key indicator for me as an architect that I got things sufficiently right.

Being the person who initially created the GLR branch, it was nice to be the person who merged it too. Other folks seemed to fear that task a bit; thankfully, it was in the end straightforward. On the other hand, I’ve probably taught 50 Git courses over the last several years, so one’d hope I’d be comfortable with such things.

After the merge, of course, came dealing with various bits of fallout that were discovered. Some showed up holes in the test suite, which were nice to fill. I also did some of the work on getting the JVM backend back up and running; again, once I got it over a certain hump, others eagerly jumped in to take on the rest.

GLR performance work

After landing the GLR came getting some of the potential optimizations in place. I wanted a real world codebase to profile to see how it handled things under the GLR, rather than just looking at synthetic benchmarks. I turned to Text::CSV, and managed a whole bunch of improvements from what I found. The improvements came from many areas: speeding up iterating lines read from a file, fixing performance issues with flattening, improving our code-gen in a number of places… There’s plenty more to be had when I decide it’s time for some more performance work; in the meantime, things are already faster and more memory efficient.


I also did some work on S07, the Perl 6 design document for lists and iteration. I ended up starting over, now I knew how the GLR had worked out. So far I’ve got most of the user-facing bits documented in there; in the coming weeks I’ll get the sections on the iterator API and the parallel iteration design fleshed out.

Syntactic support for supplies

At YAPC::Asia I had the pleasure of introducing the new supply/react/whenever syntax in my presentation. It’s something of a game-changer, making working with streams of asynchronous data a lot more accessible and pleasant. Once I’d had the idea of how it should all work, getting to an initial working implementation wasn’t actually much effort. Anyway, that’s the biggest item ticked off my S17 changes list.

Other concurrency improvements

A few other concurrency bits got fixed. RT #125977 complained that if you sat in a tight loop starting and joining threads that themselves did nothing, you could eat rather a lot of memory up. It wasn’t a memory leak – the memory was recovered – just a result of allocating GC semispaces for each of the threads, and not deallocating them until a GC run occurred. The easy fix was to make joining a thread trigger a GC run – a “free” fix for typical Perl 6 programs which never touch threads directly, but just have a pool of them that are cleaned up by the OS at process end.

The second issue I hunted down was a subtle data race involving closure semantics and invocation. The symptoms were a “frame got wrong outer” on a good day, but usually a segfault. Anyway, it’s gone now.

Last but not least, I finally tracked down an issue that’s been occasionally reported over the last couple of months, but had proved hard to recreate reliably. Once I found it, I understood why: it would only show up in programs that both used threads and dynamically created types and left them to be GC’d. (For the curious: our GC is parallel during its stop the world phase, but lets threads do the finalization step concurrently so they can go their own way as soon as they get done finalizing. Unfortunately, the finalization of type tables could happen too soon, leaving another thread finalizing objects of that type with a dangling pointer. These things always sound like dumb bugs in hindsight…)

Fixed size/shaped arrays

Work on fixed size arrays and shaped arrays continued, helped along by the GLR. By now, you can say things like:

my @a :=,3));
@a[1;1] = 42;

Next up will be turning that into just:

my @a[3;3];
@a[1;1] = 42;

Preparing for Christmas

With the release of the Perl 6 language getting closer, I decided to put on a project manager hat for a couple of hours and get a few things into, and out of, focus. First of all, I wrote up a list of things that will explicitly not be included in, and so deferred to a future Perl 6 language release.

And on the implementation side, I collected together tickets that I really want to have addressed in the Rakudo we ship as the Christmas release. Most of them relate to small bits of semantic indecision that we should really get cleaned up, so we don’t end up having to maintain (so many…) semantics we didn’t quite want for years and years to come. Having compiler crashes and fixing them up in the next release is far more forgivable than breaking people’s working code when they upgrade to the next release, so I’m worrying about loose semantic ends much more than “I can trigger a weird internal compiler error”.

The `is rw` cleanup

One of the issues on my Christmas list was getting the “is rw” semantics tightened up. We’ve not been properly treating it as a constraint, as the design docs wish, meaning that you could pass in a value rather than an assignable container and not get an error until you tried to assign to it. Now the error comes at signature binding time, so this program now gives an error:

sub foo($x is rw) { }
 foo(42); # the 42 fails to bind to $x

Error reporting improvements

I improved a couple of poor failure modes:

  • Fix RT #125812 (error reporting of with/without syntax issues didn’t match if/unless)
  • Finish fixing RT #125745 (add hint to implement ACCEPTS in error about ~~ being a special form)
  • Remove leftover debugging info in an error produced by MoarVM

Other bits

Finally, the usual collection of bits and pieces I did that don’t fit anywhere else.

  • Test and look over a MoarVM patch to fix VS 2015 build
  • Reject RT #125963 with an explanation
  • Write response to RT #126000 and reject it (operator lexicality semantics)
  • Start looking into RT #125985, note findings on ticket
  • Fix RT #126018 (crash in optimizer when analysing attribute with subset type as an argument)
  • Fix RT #126033 (hang when result of a match assigned to variable holding target)
  • Reviewing the gmr (“Great Map Refactor”) branch
  • Fix crash that sometimes happened when assembing programs containing labeled exception handlers
  • Review RT #125705, check it’s fixed, add a test to cover it. Same for RT #125161.
  • Cut September MoarVM release
  • Hunt JIT devirtualization bug on platforms using x64 POSIX ABI and fix it
  • Tests for RT #126089
  • Fix RT #126104 (the `is default` type check was inverted)
  • Investigate RT #126029, which someone fixed concurrently; added a test to cover it
  • Fix RT #125876 (redeclaring $_ inside of various constructs, such as a loop, caused a compiler crash)
  • Fix RT #126110 and RT #126115.
  • Fixed a POST regression
  • Fix passing allomorphs to native parameters and add tests; also clear up accidental int/num argument coercion and add tests
  • Fix RT #124887 and RT #124888 (implement missing <.print> and <.graph> subrules in regexes)
  • Fix RT #118467 (multi-dispatch sorting bug when required named came after positional slurpy)
  • Looking into RT #75638 (auto-export of multis), decided we’d rather not have that feature; updated design docs and closed ticket
  • Investigate weird return compilation bug with JIT/inline enabled; get it narrowed down to resolution of dynamics in JIT inlines
  • Fix RT #113884 (constant scalars interpolated in regexes should participate in LTM)
  • Investigate RT #76278, determine already fixed, ensure there is test coverage
Posted in Uncategorized | Leave a comment


It’s amazing just how long has gone by since I last wrote something here. I intend to get back to writing weekly posts about my Perl 6 work again starting around now, but here’s a few big-picture updates of what I’ve been up to.

First came YAPC::Asia. I gave a talk on Perl 6’s concurrent, parallel, and asynchronous features; the slides as well as a video are available. The talk was well received, and I was happy to have some interesting questions at the end. I was also happy to find myself talking with some Go compiler folks during the conference dinner; of course, we talked about concurrency models, and plenty besides. While the conference was great, and I’m glad I took up the invitation to go and speak there, my body basically refused to sleep at local night time. Given I’d only just managed to get shake off the exhaustion that had been troubling me for a while before the trip, this was rather sub-optimal – and meant I was pretty much beat by the time I got back home. Just in time for…the Swiss Perl Workshop.

Switzerland is, thankfully, in the same timezone as Prague, so no trouble with being able to sleep on a night! And the hotel was very comfortable too. Sadly, I still arrived to the workshop really quite exhausted – no thanks to needing to make the trip from Prague to Zurich via a night in Stockholm for annoying immigration-related reasons (I’m generally appreciative of the time the Schengen agreement saves me, but there’s certainly some improvement needed). Despite my tiredness, it was still a useful time. Day 1 was a hackathon, and lots of GLR-related decisions got made – much thanks to having the right people in the same room. On day 2, I gave my concurrency talk again, along with a talk on NFG. And on day 3, I taught a day-long Perl 6 introduction course – to a totally full room.

Despite feeling dizzy and exhausted in some periods, SPW was a great time. But by the time it was over, I really felt my body telling me I was pushing it way too far. The mere thought of getting on another flight gave me a headache. So, sadly I ended up cancelling my trip to YAPC::Europe this year. After attending 10 years worth of YAPC::Europe conferences in a row, I was sad to miss it – and it was a pity to have to cancel my session there too. Happily, lizmat++ stepped up to take my slot and deliver my concurrency talk – which she’d already seen in Japan and Switzerland. I spent several days offline, working my way back home overland with a couple of stops in Austria, thankfully with my wife to take good care of me along the way.

In terms of big Perl 6 implementation news: in the days between my last post here and YAPC::Asia, I got the GLR (Great List Refactor) to a point where others could contribute. And contribute they did! A special mention goes to nine++, who put in an incredible amount of work, meaning that I could merge the GLR work in early September. Since then we’ve been working through various details, but things are looking really good there.

Obviously, I’ve been having to take it somewhat easy in the last weeks. It’s going to be a while before I’m back to full health, and critically I need to avoid doing too much. I’ve been finding a decent amount of time to do Perl 6 things, but with limited energy have prioritized doing things (and helping others do things) rather than writing about doing things. By this point, I think I’ll be able to get back to regular progress reports again, so you’ll have those to look forward to. I’ll hopefully have one out for the last week or so sometime near the weekend.

Posted in Uncategorized | 2 Comments

This week: fixing lots of RTs, digging into the GLR

This report covers the week starting 27th July. The first half was spent teaching the course that I was busy finishing up writing last week; happily, it went extremely well and was a lot of fun to deliver. And since then, I’ve been back to Perl 6 things pretty much exclusively. Anyway, here’s what I got up to.

Fixing the &?ROUTINE pessimization

I noted in a recent episode that I fixed &?ROUTINE’s semantics with regards to closures. And indeed I had. Unfortunately, in doing so, I managed to make a whole bunch of things impossible for the MoarVM inliner to handle. Given how much fiddly work it was to implement multi-level inlining and deoptimization, I guess I should be happy that ending up with a lot less things being inlined would lead to a very measurable slowdown in real-world code – in this case, Text::CSV. This week I took another crack at &?ROUTINE, retaining the correct semantics, but making it so you only pay the cost of having it if you actually use it. And with that, the inlining started working much more effectively again (and Test::CSV regained some speed – and likely other code too).

The status of our-scoped things inside roles

There were four RT tickets on the subject of our-scoped things inside of roles. They all hinged on a seemingly easy question. If this works:

class C {
    our sub s() { }

Then why doesn’t this:

role R {
    our sub s() { }

There’s actually not one, but two reasons why. The first is that a given role is actually rather like a template, generic on the type of the class that it actually ends up being composed into (think about how ::?CLASS is generic inside the body of a role, then realize you could mention it in the our sub; you’ll note the our sub you’re referencing – if you could – would be ambiguous). The second is that you could later define:

role R[::T] {

That’s fine, this role and the previous one are disambiguated by their parameter lists. But it means that the symbol R that we install actually refers to a role disambiguator. (If you’re thinking this sounds like proto and multi subs all over again – you’re spot on. In fact, it’s implemented using the very same mechanism.)

Once you know these two things, it makes sense that you’re not going to reach an our-scoped thing burried two levels deep in genericity. Of course, when you start out with Perl 6 you don’t know those two things, you’ll more likely flail around frustratedly. Now we reject our-scoped declarations inside of roles at compile time, with an explanation of why.


The Great List Refactor, or Great List Re-implementation, or Great List Re-design, was identifed as one of the Big Three Tasks for Perl 6 ahead of the release we’re working towards later this year. In summary, the goal is to take on the semantic, speed, and memory issues with the current list design and implementation. I wasn’t expecting to be the person who led implementation work on this, but in the end it has fallen to me. Thankfully, a lot of the design thinking has already taken place, so it really is a case of focusing on the lower-level data structure design and implementation. Anyway, it takes a while to get from no code to something that’s ready for the wider community to have a conversation around, and so my GLR time in the week covered by this report started out with isolated contemplation and fleshing out code. Spoiler: I actually released it this Monday, and have continued evolving it since; you can find the latest in a Gist I’m keeping updated (though I suspect in the next days I’ll be moving over to working in a Rakudo branch).

Other fixes

I fixed quite a few other tickets too:

  • Fix RT #125675 (tighten up various signatures so we get bind failures, not .count/.arity dispatch failures)
  • Fix RT #125670 (rx{foo} as a parameter default caused compiler crash when it tried to do some static analysis)
  • Fix RT #125715 (problems using EXPORT-d type as a type constraint on an attribute)
  • Fix RT #125694 (private method calls in roles bound too tightly to role type)
  • Fix getting ugly low-level backtrace when sinking last statement in a program
  • Verify RT #125346 is fixed, write a test for it
  • Fix a MoarVM crash involving lexotic (return) handlers and a race condition in frame validation
  • Fix RT #125480 (program counter corruption due to bad interaction of LEAVE/return/closures)
  • Trying to hunt down a MoarVM parallel GC bug. Found one issue and patched it, but it’s seemingly not The One…

See you next week!

Posted in Uncategorized | 1 Comment

This week: too little sleep and too little Perl 6

This report covers the week starting the 20th July – which turned out to be the week I had to finish preparing course material and labs for one of Edument’s new courses. That managed to swallow most of the week, so only a handful of Perl 6 things got done. Happily, August is free of any teaching and authoring responsibilities, and will be dominated with Perl 6 work.

Digging into multi-dimensional arrays in Perl 6

I started a Rakudo branch to work on the multi-dimensional array support, and made various decisions about how things will work. I got some way into the changes to Array itself (and had it basically working provided you worked in terms of the *-POS API directly), and started to look at the ways the slicing implementation will need to change to pass multiple dimensions along (so you’ll actually be able to do the access with […] as you’d expect).

Banning confusion

I took on one of the oldest RTs in the queue, which wished for there to be a compile time error on:

say $*a; my $*a;

There’s long been one on:

say $a; my $a;

Basically, it catches use-before-declaration confusion (since the declaration happens at compile time). The second case is clear cut; the first is less so as it involves a dynamically scoped variable and so we’re naturally a bit looser about those. But after a little discussion, Larry said he’d like to outlaw the first just like the second if it wasn’t too hard to implement. I took a look at the code, decided it wasn’t too bad at all, and fixed it.

Unblocking the release

During the preparation for the 2015.07 release, somebody noticed a regression in reporting “return outside of routine” errors, that the tests had missed. I jumped in to get it fixed up, and added a better test so we don’t bust it again.

This week’s token regex engine patches

I fixed RT #125648 (no syntax error for /00:11:22/), as well as looking into RT #77524 (Rakudo treated /a:/ as legal syntax and STD did not; it turns out Rakudo was right on this one).

Other bits

  • Fix RT #125642 and RT #121308 (traits expecting types didn’t report bad type or make suggestions)
  • Analyze RT #125634 and get it down to a much smaller example of what’s wrong; no fix yet
Posted in Uncategorized | 1 Comment

This week: concurrency stuff, multi-dimensional stuff, stuff stuff…

Finally, I got a week of peaceful hacking time at home and not-too-bad health, and so Stuff Got Done. Here’s what.

Progress on multi-dimensional arrays

Last time, I’d gotten decent support into MoarVM for multi-dimensional arrays (including packing natively typed values into a single piece of memory), and had started on porting this functionality to the JVM. This week, I finished that JVM porting work. Doing something for the second time is often pretty good at showing up things you didn’t think through well enough the first time. In doing the port, I thought up a couple of tests I’d not written on MoarVM that would be explodey, or faily, or otherwise unhappy – and found one that I’d written that really didn’t make a lot of sense. So I wrote some extra tests, fixed things, and we now have MoarVM and JVM equipped with the guts needed to dig into implementing multi-dimensional arrays at the Perl 6 level.

Concurrency thinking and tooling

I did two interesting things relating to Perl 6’s concurrency support, both of which are interesting to discuss a little.

For a while, I’ve known that our syntactic relief for concurrent programming needed some attention. For one, our await keyword – so far implemented as the simplest possible thing – does not offer the semantics that would make it genuinely powerful. If you await a Promise today, you block the waiting thread until it’s ready. Sure, it’s a kernel-supported, OS-scheduler-efficient blocking, not busy waiting, but it still swallows one of the thread pool’s threads – which are real OS threads. And that makes it impossible to have many thousands of start blocks “active” and awaiting something to happen in order to make progress. What we want is for an await in a start block scheduled on the thread pool to return control to the thread pool, so the OS thread can be used for something else. Then, once the Promise being waited on reaches some conclusion, the rest of the start block is scheduled for resumption. That was always my vision for it, but until now I never got around to defining the API through which that happens, let alone implementing it. This week I tackled the API design part of the job. I’ll work on the implementation in the coming weeks.

Next in line were supplies. I like where we’ve gone with them so far, but working with them is very much an exercise in functional programming. It’s a bit like not having for loops and if statements, and having to write everything with map and grep. You can certainly do it, but plenty of normal people find code written that way harder to follow. Heck, some of the less-normal folks like me recognize that some solutions just read better when they’re more imperatively specified. I’ve been pondering this for a while. I really want the asynchronous aspects of Perl 6 to be accessible, and I really want people to be able to write operations that combine many asynchronous data sources – including time – without epic functional contortions. Having done my share of teaching Rx.Net, I’ve had plenty of chance to see people grapple with asynchronous data using an API a lot like we have for supplies. When there’s a nice built-in that does Just What You Want, it works out great. But sometimes there’s not, and you have to get creative, and then the result tends to feel clever rather than clear.

So, I also proposed some syntactic relief for working with supplies. It comes in two parts: supply { … } blocks for constructing supplies, and an asynchronous looping construct called whenever that works with it. So far, feedback has been positive; Larry said it looked sane, and other responses ranged from approving up to excited. So, it’s looking promising.

I didn’t update S17 yet, but rather wrote my proposals in a gist, which has all of the details.

The second big thing I did this week was work on a MoarVM bytecode instrumentation that can identify when one thread writes to an object that was created by another thread while not holding a lock. While there are of course patterns where you can legitimately do such a thing, they are the exception rather than the norm – and so having a tool that tells you when such things happen can help identify bugs. I wrote it to help me get a better insight into some of the threading bugs we have in the RT queue. It’s turned on by setting an environment variable, and it instruments bytecode (using the same approach the profiler does) to detect and record such cross-thread writes. It was also not a lot of code to implement, which I guess is good news on MoarVM’s architecture. And yes, there are sophisticated data race detection algorithms out there, but they’d all take a good bit more work to get in place (maybe some day in the future, I’ll take this one). For now, this first, simplistic, approach should help us hunt down a bunch of issues.

Meanwhile, in regex land…

I was active in the regex engine again.

  • Fix RT #125608 (Longest Token Matching did not factor in the first branch of a sequential alternation)
  • Verify RT #125391 (order of zero-width delimiters in .caps) already fixed and write a test for it
  • Fix RT #117955 (quantified captures only captured last items when used in a conjunction)
  • Investigate ways to deal with RT #67128 (calling another grammar); discussion, prototype a fix, find it needs lang design input

Better errors

This week had its share of improved failure modes and better feedback, to enhance Perl 6 user experience.

  • Fix RT #125595 (improve error reporting on bad loop specification, in line with STD)
  • Fix RT #125600 (good error message for running a directory, plus make sure we report such issues on STDERR)
  • Fix RT #115398 and RT #115400 (give good error with location info on trying to parameterize a non-parametric type)
  • Fix RT #125591 (failed to detect various mis-uses of $.x and $!x in signatures at compile time)
  • Fix RT #125625 (misleading/malformed error for useless accessor method generation with my $.a and our $.a)
  • Fix RT #125620 (gist method on custom exceptions with no message method would crash)

Other assorted bits

As usual, there are a few other small things I did that are worth a quick mention:

  • Fixing MSVC MoarVM build after an otherwise-good patch busted it
  • Fix a Proc::Async test file for Windows and add it to those we run
  • Fix RT #124121 (using “but” for role mixins with Array literal did the wrong thing, plus bad behavior with Parcel)
  • Implement does trait on variables (resolves RT #124747)
Posted in Uncategorized | 1 Comment

This week: less than hoped, but still good stuff

This week actually means “the week starting 6th July”, which is around the time much of Europe was being unreasonably hot. I spent the week in lovely Kyiv with my wife – where the weather was, predictably, also hot. I’d hoped for a nice mix of hacking, sight-seeing, and nice food. Well, I got a decent amount of nice food. Unfortunately, I also had a pretty bad time with the pesky hayfever thing that’s been bothering me this year (mostly due to bad sleep and hot weather), got frustrated enough that I decided to try a different anti-allergy medication, reacted less than awesomely to it, and generally spent plenty of time feeling crap. Happily, things are improving a good bit now that I’m back home, and ahead of me are five weeks with only 3 nights that I need to be away from home. After two and a half months in which I was never in the same place for much more than a week, you can’t imagine how happy I am of that – I can only imagine it’ll do my productivity wonders, and I’m hopeful for my health too. Anyway, let’s take a look through the few bits I did manage to get done.

Multi-dimensional array progress

In the last report, I mentioned I’d got most of the way with the new MultiDimArray representation I was implementing in MoarVM, along with various new ops. This week I got the last loose ends tied up, adding support for cloning, serialization, and deserialization.

With that done, it was time to move on to porting the work over to the JVM. I stubbed in all the new op mappings, so the NQP test file I established while working on the MoarVM implementation would compile on the JVM backend also (the strategy of writing tests at NQP level to exercise backend-level stuff continues to serve us very well). Then I set about working to make them pass. I got up to 71 out of 188 tests passing, which is a decent start – especially given there’s various bits of setup work to do early on.


The &?BLOCK symbol refers to the current block of code we’re in (which amongst other things provides a way to write recursive, yet anonymous, blocks). &?ROUTINE is the same, but for the current enclosing routine (sub, method, token, etc.) We’ve had &?ROUTINE for a while, but not &?BLOCK. I set out to implement it, and noted that one had to be careful that it refers to the current closure. Glancing at &?ROUTINE, I noticed it didn’t take sufficient care over closure semantics, and soon had a failing test case exposing the issue. So, I fixed the &?ROUTINE bug, wrote tests for &?BLOCK, and got that in place too. So, one missing feature added and one potential nasty bug down.

when/default semantics

All the way back in April, I tried to deal with RT #71368, which noted that our when/default semantics were out of line with the design in S04. Trouble is, when I tried to bring us in line with them, I found that I would break a bunch of folk’s code. Of note, this pattern would not be allowed:

sub foo($n) {
    $_ = bar($n);
    # use when/default here against $_

That is, setting $_ – which you get fresh per block anyway – for the sake of using when/default. Nobody really felt this should be outlawed. I agreed and put aside my changes.

This week I finally got around to returning to the issue to try and bring it to some kind of conclusion. The result was a commit to the design docs to bring them in line with the semantics that folks seem to prefer (which was a nice simplification also), along with adding some more tests to give us better coverage.

One more regex engine bug down

A while back, we got dynamic quantifiers, so you can use a variable (or any expression, really) to decide how many times to match something:

/'x' ** {$n}/

RT #125521 pointed out an icky bug that showed up when you tried to mix this feature with captures. Thankfully, this turned out to be one of the easier kind of regex engine bugs to figure out: some capture-related code paths simply hadn’t been updated to understand dynamic quantifiers.

And the usual other little bits

Here are a few other assorted small things that I dealt with:

  • Fix RT #125537 (type variable resolution failed to look in outer scopes) and add test
  • Fix RT #124940 (for type variable T, my T $x = … could fail to assign)
  • Review test mentioned in RT #125003, correct it, resolve ticket.
  • Fix RT #125574 (missing error on too-late application of ‘is repr(…)’ trait)
  • Fix RT #125513 (could auto-gen a %_ when there already was one in some unusual cases)
  • Review RT #80694, observe weird .^can behavior is gone, add a test for that, suggest a good solution to problem and test it too

Stay tuned for next week’s report, which already has as much to talk about as this one – and we’re only half way through the week!

Posted in Uncategorized | 2 Comments

This week: digging into multi-dimensional arrays – and plenty more

This report covers what I got up to during the closing days of June and the opening days of July.

Multi-dimensional array support in MoarVM

I’ve been pondering how to approach the multi-dimensional array aspects of the S09 Perl 6 design document for a while. I started out the implementation phase by taking another pass through the document, with an eye for things that were likely to hurt, or simply that did not fit with the current Perl 6 language as we have it today. That resulted in a gist that I tossed in Larry’s direction. Thankfully, nothing in that list was a huge blocker for getting the majority of the work done. With the top-down bit of out the way, it was time to move on to the bottom-up. MoarVM’s opset wanted a few additions, the representation API wanted a few extensions, and a new representation (named MultiDimArray) was needed. To recap, a representation is a memory allocation, layout, and access strategy, and is one half of what makes up the Perl 6 notion of “type” (the other half being the meta-object, which cares for all the high-level, VM-independent bits like method dispatch and type relations). Gradually, I test-drove my way through implementing the new multi-dimensonal APIs on the existing 1D dynamic array representation, then started to flesh out the new multi-dimensional representation. By the end of the week, I had the majority of it in place. The new representation can happily, for example, store a 10x10x10 array of 8-bit integers in a single 1000-byte blob. To come is filling out a few more missing pieces, the JVM port of these new guts (thankfully with a nice set of tests to guide the way), and then onwards and upwards to making use of it all at Rakudo level, so we can have multi-dimensional array support in Perl 6.

Another pre-compilation bug nailed

One of the most annoying kinds of bugs people run into are pre-compilation bugs: ones where your modules work fine, but a version of the module pre-compiled to bytecode breaks in some way. While that’s not always the compiler’s fault (for example, if you monkey-patch recklessly, or meta-program carelessly), most of the time it is. This week I hunted down a bug involving variables typed with subset types running into pre-compilation issues. Thankfully, it wasn’t overly difficult to fix once I worked out what was going on – but most happily, it was also the pre-compilation bug afflicting Text::CSV. It now works just fine pre-compiled.

A few tasks down in the regex engine

For a while, there’s been some debate of the failure semantics of the goal-matching syntax. That is, should:

/'(' ~ ')' \d+/

Backtrack on not finding the closing parentheses, as if you’d just written:

/'(' \d+ ')'/

Or should it throw an exception? Now, all of the uses of this construct in the Perl 6 grammar want the exception semantics. So, that’s the behavior we’ve had. However, it was argued (on a few occasions over the years) that this was not a desirable behavior for using the construct in normal regexes. My argument was always, “so just write it the other way” – but after enough tickets on the issue it was time for a review. Patrick Michaud wrote up a possible way forward a while ago, and this week I ran that by Larry, who agreed we’d change things. So, I set about putting the change into effect. Here, the design of the goal matching error mechanism came in handy. Actually, the syntax:

/'(' ~ ')' \d+/

Desugars to:

/'(' \d+ [ ')' || <.FAILGOAL(')')>]/

And the FAILGOAL method threw the exception. So, the behavior change simply meant adding:

token FAILGOAL($missing) { <!> }

The compiler toolchain already overrode FAILGOAL to throw a more helpful exception, so things continued to work for the Perl 6 grammar’s own needs. The only folks left in the middle are those using the goal matching syntax who wanted the exception. Thankfully, that’s easy to get back in your own grammars:

method FAILGOAL($missing) {
    die "Oh noes, I needed a '$missing'";

I also fixed another obscure, but potentially infuriating bug involving a mis-guided optimization. I’ll just reference RT #72440 and the patch that fixed it.

More failure mode improvements

Here’s a selection of things I did related to improving error reporting, to improve overall user experience:

  • Fix RT #125120 (bad error reporting if you declared a type X then made a syntax error)
  • Fix RT #108462 (missing redeclaration checks)
  • Fix RT #125335 (lack of escaping in error message about illegal numification)
  • Fix RT #125227 (trait warnings pointed to useless internals line, not relevant position in the source code)
  • Implement RT #112922 (catch impossible default values on parameters at compile time)
  • Add test for and resolve RT #123897 (bad error reporting, improved by implementing RT #112922)

And finally…

There’s the usual collection of things not worth a headline mention, that that are gladly dealt with.

  • Fix RT #125505 (getting Capture elements stripped away Scalar containers)
  • Working on RT #125110 (leading combining characters mis-handed in Str.perl) and unfudge tests, plus further fixes to combining chars and Str.perl
  • Fix RT #125509 (=== didn’t work on Complex), plus a few other issues observed with ===
  • Add test coverage for RT #115868, plus improve the two errors that are produced
  • Fix RT #116102 (ENTER phaser did not work as an r-value)
  • Add test coverage for RT #125483 (the “;;” syntax actually influences mutli-dispatch when placed prior to the first parameter)
  • Update test now “my ${a}” is parsed as legal Perl 6 (allows anonymous hash with key type declarations)
Posted in Uncategorized | Leave a comment