Not guts, but 6: part 5

It’s time for me to start building a simple Stomp::Server class, test-driven. I’ll need to extend my Test::IO::Socket::Async to make this possible, as it currently doesn’t handle listening sockets.

The simplest start

I’ll start out by stubbing an almost empty Stomp::Server class, which goes in lib/Stomp/Server.pm6:

class Stomp::Server {
    method socket-provider() {
        IO::Socket::Async
    }
}

Once again, I’ll give it a socket-provider method so I can inject a socket test double. Then, it’s time for a new test file, server.t:

use Test;
use Test::IO::Socket::Async;
use Stomp::Server;

constant $test-socket = Test::IO::Socket::Async.new;
my \TestableServer = Stomp::Server but role {
    method socket-provider() {
        $test-socket
    }
}

So, where to begin? To set me off in a consistent direction, I take a look at Stomp::Client and notice it expects to be constructed with a host and port. That seems like a good starting point. So, some tests:

constant $test-host = 'localhost';
constant $test-port = 1234;
dies-ok { TestableServer.new }, "Must provide host and port to new (1)";
dies-ok { TestableServer.new(host => $test-host) }, "Must provide host and port to new (2)";
dies-ok { TestableServer.new(port => $test-port) }, "Must provide host and port to new (3)";

These are easily passed, by adding to Stomp::Server:

has Str $.host is required;
has Int $.port is required;

A typical pattern for asynchronous server-like things in Perl 6 is to expose a supply of incoming connections. I may as well call that listen. Here’s the simplest test I can write for that:

my $test-server = TestableServer.new(host => $test-host, port => $test-port);
my $listen-supply = $test-server.listen();
isa-ok $listen-supply, Supply, "Stomp::Server listen method returns a Supply";

It explodes since there’s no listen method. Stubbing one in that contains a supply block gets me a pass:

method listen() {
    supply {
    }
}

So far, so easy.

The audience is listening

Now for something a little more involved. I want to make sure that a listening socket is only opened once I tap the supply that comes back from Stomp::Server’s listen method:

my $socket-listening = $test-socket.start-listening;
nok $socket-listening, "Not listening before supply is tapped";
my $listen-tap = $listen-supply.tap(-> $incoming { });
my $socket-listener = await $socket-listening;
ok $socket-listener, "Listening once supply is tapped";

I’d also like to make sure it listens on the correct host/port:

is $socket-listener.host, $test-host, "Listening on correct host";
is $socket-listener.port, $test-port, "Listening on correct port";

And that closing the tap on the supply Stomp::Server gives me back will also close the listen supply from the socket:

$listen-tap.close;
ok (await $socket-listener.is-closed), "Closing supply tap also closes socket";

So, that’s the tests, but Test::IO::Socket::Async isn’t up to the job yet – so that’s first in line. I’ll want an object that represents a listening socket, and I can see that at the very least it’ll need a host and a port. I’ll also need to deal with the same race between tests and code under test that I had with connect, meaning that Listener itself should hold the Supply that I will use to simulate incoming connections. Finally, I need to provide a way to test that at some point it stops listening, exposing a Promise that is kept when that happens. That’s quite a few things that need wiring together. Happily, it falls out really quite easily, by wiring things up at construction time:

class Listener {
    has $.host;
    has $.port;
    has $.is-closed = Promise.new;
    has $!is-closed-vow = $!is-closed.vow;
    has $!connection-supplier = Supplier.new;
    has $.connection-supply = $!connection-supplier
        .Supply
        .on-close({ $!is-closed-vow.keep(True) });
}

That’s really quite pretty. Perl 6 promises that attribute initializers run in order, so I can safely rely on $!is-closed containing the Promise I next take a vow from – and also keep that vow private. I also keep the ability to inject new connections private, and then tweak the Supply with on-close, which lets me run some logic to keep the is-closed promise when a tap on the supply is closed. Since everything exposed is either immutable or concurrent, there’s no need for this to be a monitor rather than a class.

That just leaves me to write a couple of methods in Test::IO::Socket::Async. One is listen, which should match the IO::Socket::Async method. The other is start-listening, which returns a Promise that tests can await to get the Listener instance. As with testing incoming connections, I’ll need a couple of attributes too.

has @!waiting-listens;
has @!waiting-start-listening-vows;

method listen(Str() $host, Int() $port) {
    my $listener = Listener.new(:$host, :$port);
    with @!waiting-start-listening-vows.shift {
        .keep($listener);
    }
    else {
        @!waiting-listens.push($listener);
    }
    $listener.connection-supply
}

method start-listening() {
    my $p = Promise.new;
    with @!waiting-listens.shift {
        $p.keep($_);
    }
    else {
        @!waiting-start-listening-vows.push($p.vow);
    }
    $p
}

Recall that Test::IO::Socket::Async is a monitor, meaning there are no data races here. Installing these updates, and running my tests, things get a bit further, then hang here:

my $socket-listener = await $socket-listening;

That’s not surprising, because the code under test never actually starts to listen. Let me make that happen with the simplest possible addition to Stomp::Server’s listen method:

method listen() {
    supply {
        whenever self.socket-provider.listen($!host, $!port) {

        }
    }
}

With that, all the tests pass. But wait…how did the socket closed test pass? That’s thanks to supply blocks being smart enough to keep track of all the things tapped by a whenever inside of them, and closing them automatically when the corresponding tap on the supply block itself is closed. Resource management is one of the things that supply blocks quietly take care of, avoiding all kinds of potential resource leaks.

Incoming connections

Next, I want to test and implement the server side of the incoming connection handshake. This will need me to finish up Test::IO::Socket::Async. As usual, I’ll start by writing the tests I want to have:

constant $test-login = 'user';
constant $test-password = 'correcthorsebatterystaple';

my $test-server = TestableServer.new(host => $test-host, port => $test-port);
my $listen-tap = $test-server.listen().tap(-> $conn { });
my $socket-listener = await $test-socket.start-listening;
my $test-conn = $socket-listener.incoming-connection;

$test-conn.receive-data: Stomp::Message.new(
    command => 'CONNECT',
    headers => (
        login => $test-login,
        passcode => $test-password,
        accept-version => '1.2'
    ));

my $message-text = await $test-conn.sent-data;
my $parsed-message = Stomp::Parser.parse($message-text);
ok $parsed-message, "Server responded to CONNECT with valid message";
my $message = $parsed-message.made;
is $message.command, "CONNECTED", "Server sent CONNECTED command";
ok $message.headers<accept-version>:exists, "Server sent accept-version header";
is $message.body, "", "Server sent no message body";

The key new piece is the incoming-connection method. The API for simulating received data and obtaining sent data works just like in the client sockets testing – suggesting they’ll want to share a lot of the same code. But can they share all of the same code?

Glancing at the Connection monitor I already wrote, it seems the answer is no. The first four attributes:

has $.host;
has $.port;
has $.connection-promise = Promise.new;
has $!connection-vow = $!connection-promise.vow;

Aren’t really interesting for an incoming connection. The rest of the code, which deals purely with sending and receiving data, seems relevant, however. So, I’ll rename Connection to ClientConnection. I’ll then add a role called Connection, and factor the common bits out there. This gives me:

role Connection {
    has @!sent;
    has @!waiting-sent-vows;
    has $!received = Supplier.new;

    # print, write, sent-data, Supply, receive-data...
    ...
}

monitor ClientConnection does Connection {
    has $.host;
    has $.port;
    has $.connection-promise = Promise.new;
    has $!connection-vow = $!connection-promise.vow;

    method accept-connection() {
        $!connection-vow.keep(self);
    }

    method deny-connection($exception = "Connection refused") {
        $!connection-vow.break($exception);
    }
}

I’ll then define a ServerConnection monitor, which for now simply composes the Connection role:

monitor ServerConnection does Connection {
}

I note how nice it is that code factored out to a role can happily be composed into classes and monitors, and will automatically get the desired mutual exclusion behaviour when composed into a monitor. Now, I can implement the incoming-connection method in Listener:

method incoming-connection() {
    my $conn = ServerConnection.new;
    $!connection-supplier.emit($conn);
    $conn
}

And I’m done extending Test::IO::Socket::Async to support testing server sockets. Committed! And yes, I owe that module some tests and a README some time soon…maybe on the plane tomorrow. For now, the STOMP must go on!

So, how can I make my test pass? By doing the absolute easiest thing possible, of course. Here it is:

method listen() {
    supply {
        whenever self.socket-provider.listen($!host, $!port) -> $conn {
            whenever $conn {
                await $conn.print: Stomp::Message.new:
                    command => 'CONNECTED',
                    headers => ( accept-version => '1.2' );
            }
        }
    }
}

This is a fairly epic cheat. It just responds with a CONNECTED message when it receives anything on the socket! It passes the test, though. I’ve learned – mostly when doing ping-pong pair programming – that being willing to “cheat” my way to passing simple tests is actually a good thing. It makes it clear what tests I need to write next.

Sharing the parsing

To check I do treat CONNECT messages correctly, I’ll now write a test case where I sent complete junk to the server:

my $test-server = TestableServer.new(host => $test-host, port => $test-port);
my $listen-tap = $test-server.listen().tap(-> $conn { });
my $socket-listener = await $test-socket.start-listening;
my $test-conn = $socket-listener.incoming-connection;

$test-conn.receive-data: "EPIC FAIL!";

my $message-text = await $test-conn.sent-data;
my $parsed-message = Stomp::Parser.parse($message-text);
ok $parsed-message, "Server responded to invalid message with valid message";
is $parsed-message.made.command, "ERROR", "Server sent ERROR command";

Obviously, it fails, sending back a CONNECTED message instead of an ERROR. So what to do about it? Clearly, I’ll need to start paying a bit more attention to the messages coming it. That is, I need to turn a sequence of packets into a sequence of Stomp::Message objects. Hmm, that sounds familiar! Glancing in Stomp::Client, I see this:

method !process-messages($incoming) {
    supply {
        my $buffer = '';
        whenever $incoming -> $data {
            $buffer ~= $data;
            while Stomp::Parser::ServerCommands.subparse($buffer) -> $/ {
                given $/.made -> $message {
                    die $message.body if $message.command eq 'ERROR';
                    emit $message;
                }
                $buffer .= substr($/.chars);
            }
        }
    }
}

Well, that’s precisely what I want – except it’s parsing server commands, and I need to parse client commands. Factoring a method out feels like a job for a role – and needing to make the factored out code be parametric on something makes a parametric role just the ticket. So, I’ll add a Stomp::MessageStream role:

role Stomp::MessageStream[::MessageGrammar] {
    method !process-messages($incoming) {
        supply {
            my $buffer = '';
            whenever $incoming -> $data {
                $buffer ~= $data;
                while MessageGrammar.subparse($buffer) -> $/ {
                    given $/.made -> $message {
                        die $message.body if $message.command eq 'ERROR';
                        emit $message;
                    }
                    $buffer .= substr($/.chars);
                }
            }
        }
    }
}

And use it in Stomp::Client:

class Stomp::Client does Stomp::MessageStream[Stomp::Parser::ServerCommands] {
    ...
}

The client.t tests still happily pass, so it goes in as a commit. Now I can also compose the role into my Stomp::Server and use it:

class Stomp::Server does Stomp::MessageStream[Stomp::Parser::ClientCommands] {
    has Str $.host is required;
    has Int $.port is required;

    method listen() {
        supply {
            whenever self.socket-provider.listen($!host, $!port) -> $conn {
                whenever self!process-messages($conn) {
                    await $conn.print: Stomp::Message.new:
                        command => 'CONNECTED',
                        headers => ( accept-version => '1.2' );
                }
            }
        }
    }

    method socket-provider() {
        IO::Socket::Async
    }
}

That turns my test fail into…a hang. Why? Because the process-messages private method simply assumes that if it didn’t manage to parse a message, the reason must be that it’s incomplete – not that it’s broken. It will therefore just accumulate broken data in its buffer. Clearly, the parser needs to fail more violently if the incoming message is utterly bogus.

So, over in lib/Stomp/Parser.pm6, I’ll add an exception:

class X::Stomp::MalformedMessage is Exception {
    has $.reason;
    method message() {
        "Malformed STOMP message: $!reason"
    }
}

Then, I’ll tweak the TOP token to:

token TOP {
    [ <command> \n || <.maybe-command> ]
    [<header> \n]*
    \n
    <body>
    \n*
}

The addition here is the sequential alternation, calling maybe-command (the dot indicates to not capture the result). The idea of maybe-command is to check if the data so far might viably parse as a command if some more data were to arrive. It can look like this:

token maybe-command {
    <[A..Z]>**0..15 $ || <.malformed('invalid command')>
}
method malformed($reason) {
    die X::Stomp::MalformedMessage.new(:$reason);
}

With that, I’ve got from a hanging test to an exploding test. Happily, all of the client.t tests that also use the parser still work, though, so it can go in as a commit of its own.

It’s at this point that I finally ran into my first Rakudo bug of this series. The code I want to now write looks like this:

method listen() {
    supply {
        whenever self.socket-provider.listen($!host, $!port) -> $conn {
            whenever self!process-messages($conn) {
                await $conn.print: Stomp::Message.new:
                    command => 'CONNECTED',
                    headers => ( accept-version => '1.2' );

                QUIT {
                    when X::Stomp::MalformedMessage {
                        await $conn.print: Stomp::Message.new:
                            command => 'ERROR',
                            body => .message;
                    }
                }
            }
        }
    }
}

Unfortunately, QUIT phasers suffer a scoping bug if there is an exception before any other messages have been received. Thankfully, it’s easy enough to drop the whenever sugar in this case and fall back on the tap method:

method listen() {
    supply {
        whenever self.socket-provider.listen($!host, $!port) -> $conn {
            self!process-messages($conn).tap:
                {
                    await $conn.print: Stomp::Message.new:
                        command => 'CONNECTED',
                        headers => ( accept-version => '1.2' );
                },
                quit => {
                    when X::Stomp::MalformedMessage {
                        await $conn.print: Stomp::Message.new:
                            command => 'ERROR',
                            body => .message;
                    }
                };
        }
    }
}

And with that, the test passes.

And that’s Stomp::Server’s first steps

There’s still a bit more to do to ensure the first message really is a CONNECT, and to provide a hook for authentication. And then I’ll need to work out how the API for incoming messages is going to look, along with subscription/unsubscription requests. But that’ll be for next time. In the meantime, here’s a commit of the server work so far.

Posted in Uncategorized | 3 Comments

Not guts, but 6: part 4

I’ve managed to marry myself into getting two Christmases a year. The Orthodox one takes place on the 7th of January, so I’ve been celebrating that. And now the trek back home is underway, stopping off to enjoy the snow and nice mood in Kiev for a couple of nights before returning to Prague and normal life and work. (And, if you’re wondering, yes, I shall eat a Chicken Kiev while here.)

In today’s post, I’ll be keeping it simple: improving my test coverage, fixing a couple of small design issues, supporting unsubscription, and using a new little module I wrote to deal with a pesky data race.

Tweaking send

The next easy thing to write tests for is the send method, so I’ll start there. Here’s the tests:

constant $test-destination = "/queue/shopping";
constant $test-body = "Buy a karahi!";
my $send-promise = $client.send($test-destination, $test-body);
$message-text = await $test-conn.sent-data;
$parsed-message = Stomp::Parser.parse($message-text);
ok $parsed-message, "send method sent well-formed message";
$message = $parsed-message.made;
is $message.command, "SEND", "message has SEND command";
is $message.headers<destination>, $test-destination, "destination header correct";
is $message.headers<content-type>, "text/plain", "has default content-type header";
is $message.body, $test-body, "message had expected body";
is $send-promise.status, Kept, "Promise retunred by send was kept";

A little wordy, but there’s nothing new going on. One of them fails, though:

not ok 13 - destination header correct
# Failed test 'destination header correct'
# at t\client.t line 75
# expected: '/queue/shopping'
#      got: '/queue//queue/shopping'

Hmm. Let me look at send:

method send($topic, $body) {
    self!ensure-connected;
    $!connection.print: Stomp::Message.new:
        command => 'SEND',
        headers => (
            destination  => "/queue/$topic",
            content-type => "text/plain"
        ),
        body => $body;
}

Ah, there it is. My advent post hard-coded the RabbitMQ queue path, but the module really should allow full control over the destination. That’s easily fixed, and I’ll take the time to do a little rename also:

method send($destination, $body) {
    self!ensure-connected;
    $!connection.print: Stomp::Message.new:
        command => 'SEND',
        headers => (
            destination  => $destination,
            content-type => "text/plain"
        ),
        body => $body;
}

It’s easy to under-value simple things like renaming variables to keep up with the evolving language of a design, but I’ve found it to be really worthwhile. I tend to call such refactors “domain refactors”. They are often small and subtle, but together they help make the code easier to follow, improve consistency, and so ease future development. Anyway, committed!

There’s one other thing that stands out to me here, which is that it’d be good to be able to choose the content type also. First, a test:

constant $test-type = "text/html";
$send-promise = $client.send($test-destination, $test-body,
    content-type => $test-type);
$message = Stomp::Parser.parse(await $test-conn.sent-data).made;
is $message.headers<content-type>, $test-type, "can set content-type header";

It’s easily implemented, adding an optional named parameter that defaults to the text/plain content type. With the variable names perfectly matching the header names, this means I can get some repetition out of the code with the variable colon pair syntax:

method send($destination, $body, :$content-type = "text/plain") {
    self!ensure-connected;
    $!connection.print: Stomp::Message.new:
        command => 'SEND',
        headers => ( :$destination, :$content-type ),
        body => $body;
}

And there’s my second commit.

Subscription and unsubscription

Now I’ll turn to receiving messages. Once again, the tests aren’t too difficult to write, and follow a sufficiently common pattern I’m already starting to ponder whether it’s time to factor things out a bit:

my $sub-supply = $client.subscribe($test-destination);
isa-ok $sub-supply, Supply, "subscribe returns a Supply";
my $sent-data-promise = $test-conn.sent-data;
is $sent-data-promise.status, Planned, "did not yet send subscription request";
my @messages;
my $sub-tap = $sub-supply.tap({ @messages.push($_) });
$message-text = await $sent-data-promise;
$parsed-message = Stomp::Parser.parse($message-text);
ok $parsed-message, "subscribe method sent well-formed message";
$message = $parsed-message.made;
is $message.command, "SUBSCRIBE", "message has SUBSCRIBE command";
is $message.headers<destination>, $test-destination, "destination header correct";
ok $message.headers<id>:exists, "had an id header";

One fails. Once again, it’s the destination header. Here’s how my subscribe method looks:

method subscribe($topic) {
    self!ensure-connected;
    state $next-id = 0;
    supply {
        my $id = $next-id++;

        $!connection.print: Stomp::Message.new:
            command => 'SUBSCRIBE',
            headers => (
                destination => "/queue/$topic",
                id => $id
            );

        whenever $!incoming {
            if .command eq 'MESSAGE' && .headers<subscription> == $id {
                emit .body;
            }
        }
    }
}

Ah, yes, it’s the topic/destination discrepancy again. And, given I have a $id variable, I’ll be able to use the colon pair variable form again. Here goes:

method subscribe($destination) {
    self!ensure-connected;
    state $next-id = 0;
    supply {
        my $id = $next-id++;

        $!connection.print: Stomp::Message.new:
            command => 'SUBSCRIBE',
            headers => ( :$destination, :$id );

        ...
    }
}

That’s better but…something is not quite right still. I cheated a bit when I wrote this for the advent post, and nobody was observant enough to call me out on it – so I guess I’ll just have to out myself. There’s a data race on $next-id, should two threads end up making subscriptions at the same time. It’s not likely to crop up, but it still wants dealing with. I’ll do that in a moment.

Before that, I’d like to get unsubscription handled. Closing the tap should do an unsubscribe. First, some tests:

my $expected-id = $message.headers<id>;
$sub-tap.close;
$message-text = await $test-conn.sent-data;
$parsed-message = Stomp::Parser.parse($message-text);
ok $parsed-message, "unsubscribing sent well-formed message";
$message = $parsed-message.made;
is $message.command, "UNSUBSCRIBE", "message has UNSUBSCRIBE command";
is $message.headers<id>, $expected-id, "id matched the subscription";

This hangs on the await, because at present nothing is sent when the tap on the supply is closed. Happily, the CLOSE phaser makes it easy to write logic that will be run on tap close:

method subscribe($destination) {
    self!ensure-connected;
    state $next-id = 0;
    supply {
        my $id = $next-id++;

        $!connection.print: Stomp::Message.new:
            command => 'SUBSCRIBE',
            headers => ( :$destination, :$id );
        CLOSE {
            $!connection.print: Stomp::Message.new:
                command => 'UNSUBSCRIBE',
                headers => ( :$id );
        }

        ...
    }
}

I could write the CLOSE phaser wherever I wanted inside of the supply block, and so chose to put it near the logic to send a SUBSCRIBE message. Phasers are often handy in that way: they specify code that runs at certain phases in the program, and so free me to place that code in the most helpful place for the reader. And with that, the tests pass. Commit!

Dealing with that data race

So, how to deal with the getting ascending IDs in the safe way? There are a couple of options that come to mind:

  • Make Stomp::Client a monitor. That’s probably overkill, however. It’s quite capable of otherwise having methods invoked on it concurrently, since it has no state beyond that set up in connect.
  • Use Lock. But using Lock is generally a last resort, not a first one.

What I really want is a mechanism that can just give me ascending integers. If I generalize that thought a little, I want a safe way to grab the next value available from some sequence. And sequences of values in Perl 6 are typically handled by iterators. However, an Iterator is only safe for consumption from one thread at a time.

So, I wrote another little module: Concurrent::Iterator. It’s weighs in at well under 50 lines of code, and does a bit more than I need for this use case. Using it, I can just ask for a concurrent iterator over the range of integers from 1 up to infinity, and keep it around in an attribute:

has $!ids = concurrent-iterator(1..Inf);

And then use it in subscribe:

method subscribe($destination) {
    self!ensure-connected;
    supply {
        my $id = $!ids.pull-one;
        ...
    }
}

Message arrival

I’m almost up to having tests covering all the stuff that matters in Stomp::Client, but there’s one glaring exception: receiving messages from a subscription. I already set up an array that such messages can be pushed to:

my $sub-tap = $sub-supply.tap({ @messages.push($_) });

So, I’ll now sneak some extra tests in between the subscription and unsubscription tests:

my $expected-id = $message.headers<id>;
is @messages.elems, 0, "no messages received yet";
$test-conn.receive-data: Stomp::Message.new(
    command => 'MESSAGE',
    headers => ( subscription => $expected-id ),
    body    => $test-body
);
is @messages.elems, 1, "one message now received";
isa-ok @messages[0], Stomp::Message, "it's a Stomp::Message";
is @messages[0].command, "MESSAGE", "has the command MESSAGE";
is @messages[0].body, $test-body, "has the correct body";

And…epic fail!

not ok 26 - it's a Stomp::Message
# Failed test 'it's a Stomp::Message'
# at t\client.t line 108
# Actual type: Str

Since the Stomp::Message headers may well contain relevant information for processing of the message – such as a content-type header, it would be a good idea to pass those along to the consumer. Thankfully, that’s an easy change to the whenever block, to emit the Stomp::Message itself rather than its body:

whenever $!incoming {
    if .command eq 'MESSAGE' && .headers<subscription> == $id {
        emit $_;
    }
}

And that’ll be the final commit for this time.

I live to server

Next time, I’ll add support to Test::IO::Socket::Async for testing listening sockets, and then use it to start implementing a Stomp::Server class.

Posted in Uncategorized | Leave a comment

Not guts, but 6: part 3

To me, one of the most important things about the asynchronous programming support in Perl 6 is the uniform interfaces the language provides. Promises represent asynchronous operations that will produce a single result, while supplies represent asynchronous operations that may produce a stream of values (which we might find more natural to call “events”) over time.

Perl may well embrace There’s More Than One Way To Do It. However, being able to quickly put together programs that combine our selection of preferred modules still hinges on there being things they do all agree to use. It’s typically the little, unspoken things: the basic data structures (scalars, arrays, hashes – and in Perl 6 lazy iterators too), and that method calls look the same no matter what magic may lie behind their dispatch.

Promises and supplies are the basic asynchronous data structures. Whether we are working against sockets, message queues, time, domain events, or GUI events, we can talk about these sources of asynchronous values or value streams using the Promise and Supply types. And, since it’s easy to create a Promise or Supply and back it with whatever data we feel like, we can use them in writing tests for our asynchronous code too. Which brings me nicely to the next steps for my Stomp::Client.

Sketching out the double I want

I tend to find that the point I actually have a concrete need for something is a good time to design and build it. It gives me a use case, or use cases, to check the design against. To move ahead with testing Stomp::Client – something I wish to do before evolving it further – I need a test double for IO::Socket::Async. Just as a stunt double stands in for a real actor for the purpose of doing dangerous things in a film, a test double stands in for a real object for the purpose of testing code that uses it. Stub objects and mock objects are common examples of test doubles.

I’m going to use my need to test Stomp::Client to drive out the design and implementation of a Test::IO::Socket::Async. I’ll just write tests as I’d like them to look, and then do what’s needed to make things work. First, I’ll add a few constants providing some test data, so I won’t have to repeat it:

constant $test-host = 'localhost';
constant $test-port = 1234;
constant $test-login = 'user';
constant $test-password = 'correcthorsebatterystaple';

Then, pretending I have a Test::IO::Socket::Async already, I’ll take the Stomp::Client type and derive an anonymous type from it that overrides the socket-provider method I added back on day 1. I’ll arrange for the method to return my test socket instance.

constant $test-socket = Test::IO::Socket::Async.new;
my \TestableClient = Stomp::Client but role {
    method socket-provider() {
        $test-socket
    }
}

With the setup out of the way, it’s time to sketch out the first couple of tests. Checking that Stomp::Client connects to the host and port it was constructed with seems like a good start. So, here goes:

my $client = TestableClient.new(
    host => $test-host, port => $test-port,
    login => $test-login, password => $test-password
);
my $connect-promise = $client.connect();
my $test-conn = await $test-socket.connection-made;
is $test-conn.host, $test-host, "Connected to the correct host";
is $test-conn.port, $test-port, "Connected to the correct port";

The first two statements look just like a normal usage of Stomp::Client. The connect method gives back a Promise, which for now I’ll just stick in a variable and worry about later. The third statement is where the test double is used. Since IO::Socket::Async is asynchronous, interactions with its test double also should be. An asynchronous socket may be used by multiple threads, and the code under test may end up interacting with it or creating it on a different thread than our test is running on. Therefore, the test double has a connection-made method that returns a Promise that will be kept when a connect call is made on the test double. The Promise will be kept with some object that represents a test connection, and provides the host and port that were supplied to connect. These are examined in the final two statements.

Implementing the test double

First, the easy part. I’m not yet sure what the thing representing a test connection is going to look like when it’s completed, but I know it must have both a host and a port. So, I’ll just declare a simple class for it inside of Test::IO::Socket::Async:

class Test::IO::Socket::Async {
    class Connection {
        has $.host;
        has $.port;
    }
    ...
}

Next, I’ll do the connect and connection-made methods. Some care is needed here, because there’s a race condition just waiting to happen. Two orderings of events are possible. Either:

  1. The connect call is made on the test double
  2. The connection-made call is made on the test double

Or:

  1. The connection-made call is made on the test double
  2. The connect call is made on the test double

It doesn’t matter which happens, but it does matter that the behaviour is the same. Also, while I don’t immediately have a use case for it, it’s clear that other users of such a test double may wish to test code that connects to many things. Therefore, I’ll add to Test::IO::Socket::Async a pair of attributes:

has @!waiting-connects;
has @!waiting-connection-made-vows;

The first will hold Connection objects for any connect calls that were made, but that are not yet matched up with a connection-made call from the test code. The second plays the opposite role: it holds vows (the thing that is used to keep or break a Promise) for promises returned by connection-made that are not yet matched up with a connect call. The two methods will have a similar kind of symmetry:

method connect(Str() $host, Int() $port) {
    my $conn = Connection.new(:$host, :$port);
    with @!waiting-connection-made-vows.shift {
        .keep($conn);
    }
    else {
        @!waiting-connects.push($conn);
    }
    my $p = Promise.new;
    $p.keep($conn);
    $p
}

method connection-made() {
    my $p = Promise.new;
    with @!waiting-connects.shift {
        $p.keep($_);
    }
    else {
        @!waiting-connection-made-vows.push($p.vow);
    }
    $p
}

Note that a with block is like an if block, but it tests for definedness instead of truth, and sets $_ to the tested object.

Coping with concurrency

There’s one more important thing I need to take care of. Running the tests at this point reveals it. They hang. I try again. Ooh, a pass. Third time? Hang. So, what’s going on? Well, I only told a half-truth earlier when discussing the ordering between connect and connection-made. It’s also possible for the two to be called at the same time! Thankfully, that’s easily fixed. My class needs to become a monitor, which enforces one-at-a-time semantics on the methods of a particular instance. So, it’s off to the ecosystem:

panda install OO::Monitors

OO::Monitors uses Perl 6’s meta-programming features to good effect. All that is needed to make a class into a monitor is to replace the class declarator with a monitor declarator, which is provided by OO::Monitors. Here’s how my test double ends up looking:

use OO::Monitors;

monitor Test::IO::Socket::Async {
    class Connection {
        has $.host;
        has $.port;
    }

    has @!waiting-connects;
    has @!waiting-connection-made-vows;

    method connect(Str() $host, Int() $port) {
        my $conn = Connection.new(:$host, :$port);
        with @!waiting-connection-made-vows.shift {
            .keep($conn);
        }
        else {
            @!waiting-connects.push($conn);
        }
    }

    method connection-made() {
        my $p = Promise.new;
        with @!waiting-connects.shift {
            $p.keep($_);
        }
        else {
            @!waiting-connection-made-vows.push($p.vow);
        }
        $p
    }
}

Not bad.

In denial

This is a promising start. That’s the great thing about Perl 6: every start { … } is Promise-ing. But something is just a little off. While it would be easy to plough ahead and write the next test on the happy path – where the connection to a STOMP server is successful – an even easier one to write is the case where the socket connection fails, and Stomp::Client never gets so far as doing the handshake. But right now, there’s no way to write such a test. The connect Promise is immediately kept when a connect call is made in the test.

Once again, I’ll write the test as I’d like to express it:

my $client = TestableClient.new(
    host => $test-host, port => $test-port,
    login => $test-login, password => $test-password
);
my $connect-promise = $client.connect();
my $test-conn = await $test-socket.connection-made;
$test-conn.deny-connection();
dies-ok { await $connect-promise },
    "Failed STOMP server connection breaks connect Promise";

Now for the changes. First, I’ll extend Connection a bit. It will hold the Promise that will be returned by the connect method. Then, the accept-connection and deny-connection methods will use the vow on that Promise.

class Connection {
    has $.host;
    has $.port;
    has $.connection-promise = Promise.new;
    has $!connection-vow = $!connection-promise.vow;

    method accept-connection() {
        $!connection-vow.keep(self);
    }

    method deny-connection($exception = "Connection refused") {
        $!connection-vow.break($exception);
    }
}

Finally, back in Test::IO::Socket::Async, I’ll update the connect method to just return this Promise:

method connect(Str() $host, Int() $port) {
    my $conn = Connection.new(:$host, :$port);
    with @!waiting-connection-made-vows.shift {
        .keep($conn);
    }
    else {
        @!waiting-connects.push($conn);
    }
    $conn.connection-promise
}

And the test passes. Hurrah. This means Stomp::Client isn’t failing to pass on socket connect errors to its consumer, which is certainly a test worth having.

Testing what was sent

Now I’d like to start filling out a test case for the CONNECT handshake that Stomp::Client should do with a STOMP server. Here’s the first bit, checking that a well-formed CONNECT message is sent with the correct information:

my $client = TestableClient.new(
    host => $test-host, port => $test-port,
    login => $test-login, password => $test-password
);
my $connect-promise = $client.connect();
my $test-conn = await $test-socket.connection-made;
$test-conn.accept-connection();

my $message-text = await $test-conn.sent-data;
my $parsed-message = Stomp::Parser.parse($message-text);
ok $parsed-message, "Client sent valid message to server";
my $message = $parsed-message.made;
is $message.command, "CONNECT", "Client sent a CONNECT command";
is $message.headers<login>, $test-login, "Client sent login";
is $message.headers<passcode>, $test-password, "Client sent password";
ok $message.headers<accept-version>:exists, 'Client sent accept-version header';
is $message.body, "", "Client sent no message body";

The only new thing here with regard to socket testing is the sent-data method on a test connection. It returns a Promise that will be kept when something is sent using the socket. It will be kept with what was sent. The code that follows checks that the message contains what was expected of it. Note that a few things are done here to avoid test fragility:

  • The headers are tested in a way that does not depend on their ordering, as any order is valid
  • The accept-version is not hard-coded, so the test will not break if the module is later updated to cope with newer protocol versions

I was able to use the previously factored out Stomp::Parser in order to avoid testing directly against the text of the message, which would result in an overly-specific test.

So, what happens in my test Connection class to support this? First of all, it’s time to make it a monitor, since I’m about to give it mutable state:

monitor Connection {
    ...
}

I’ll then do a somewhat similar thing as I did when testing connects: have an array of sent things and an array of vows to keep. Here’s the code that I add to the Connection class:

has @!sent;
has @!waiting-sent-vows;

method print(Str() $s) {
    @!sent.push($s);
    self!keep-sent-vows();
    self!kept-promise();
}

method write(Blob $b) {
    @!sent.push($b);
    self!keep-sent-vows();
    self!kept-promise();
}

method sent-data() {
    my $p = Promise.new;
    @!waiting-sent-vows.push($p.vow);
    self!keep-sent-vows();
    $p
}

method !keep-sent-vows() {
    while all(@!sent, @!waiting-sent-vows) {
        @!waiting-sent-vows.shift.keep(@!sent.shift);
    }
}

method !kept-promise() {
    my $p = Promise.new;
    $p.keep(True);
    $p
}

The switch to a monitor is critical to avoiding various races that could easily occur. The print and write methods return a kept Promise in order to match the API of IO::Socket::Async itself. And…the test is happy.

Testing what was received

My previous test wasn’t quite a complete test of the connection process, since the Promise returned by Stomp::Client’s connect method is not completed until a CONNECTED frame is received from the server. As usual, I’ll sketch out the test I want to have:

$test-conn.receive-data: Stomp::Message.new(
    command => 'CONNECTED',
    headers => ( version => '1.2' )
);
ok (await $connect-promise), "CONNECTED message completes connection";

IO::Socket::Async uses a Supply for incoming data received by the socket. That makes it rather straightforward to fake up. First, I’ll add an attribute to my test Connection monitor that holds a Supplier:

has $!received = Supplier.new;

The test version of IO::Socket::Async’s Supply method will simply return the Supply that goes with it:

method Supply() {
    $!received.Supply
}

And then I’ll use the Supplier to emit the data we want to fake the socket receiving, taking care to make sure I only pass along blobs of binary data or strings, conveniently stringifying any other type that isn’t already one:

multi method receive-data(Str() $data) {
    $!received.emit($data);
}
multi method receive-data(Blob $data) {
    $!received.emit($data);
}

And with that, I’ve a passing test.

A module is born

I developed Test::IO::Socket::Async at the top of the test file I was fleshing out the client tests in. However, it really wants to be a separate module, so others can use it in their own tests. So, I gave it its own git repo, with a META.info. Even before adding it to the module list, I could simply do:

panda install .

And use it from my client tests:

use Test::IO::Socket::Async;

Which I then committed.

And what next?

The tests only cover one method of Stomp::Client. However, it should be fairly easy to test the rest, now I’ve got a test double for IO::Socket::Async. This also means I can more confidently move on to implementing some further aspects of the client, working test-first to add features such as unsubscription, disconnecting, and transactions.

Posted in Uncategorized | 1 Comment

Not guts, but 6: part 2

It’s time for more hacking on my Perl 6 STOMP module. Today: parsing.

Pulling out the parser

Given my plans for adding a Stomp::Server to go with my Stomp::Client, I need to factor my STOMP message parser out so it can be used by both. That will be an easy refactor. First, the parser moves off into a file of its own and gets called Stomp::Parser:

grammar Stomp::Parser {
    token TOP {
        <command> \n
        [<header> \n]*
        \n
        <body>
        \n*
    }
    token command {
        < CONNECTED MESSAGE RECEIPT ERROR >
    }
    token header {
        <header-name> ":" <header-value>
    }
    token header-name {
        <-[:\r\n]>+
    }
    token header-value {
        <-[:\r\n]>*
    }
    token body {
        <-[\x0]>* )> \x0
    }
}

Then it’s just a use statement and a small tweak back in Stomp::Client. Done!

Testing parsing of commands – and a discovery

Perhaps the most basic test I should write is for being able to parse all of recognized commands, but not unrecognized ones. So, here goes:

use Test;
use Stomp::Parser;

plan 16;

my @commands = <
    SEND SUBSCRIBE UNSUBSCRIBE BEGIN COMMIT ABORT ACK NACK
    DISCONNECT CONNECT STOMP CONNECTED MESSAGE RECEIPT ERROR
>;

for @commands {
    ok Stomp::Parser.parse(qq:to/TEST/), "Can parse $_ command (no headers/body)";
        $_

        \0
        TEST
}

nok Stomp::Parser.parse(qq:to/TEST/), "Cannot parse unknown command FOO";
    FOO

    \0
    TEST

This doesn’t pass yet, because it turns out the grammar only supports the commands that a server may send, not those a client may send. That’s an easy fix:

token command {
    <
        SEND SUBSCRIBE UNSUBSCRIBE BEGIN COMMIT ABORT ACK NACK
        DISCONNECT CONNECT STOMP CONNECTED MESSAGE RECEIPT ERROR
    >
}

That makes me stop and think a bit, though. I just took a parser suitable for Stomp::Client and generalized it. But now it will also accept messages that a client should never expect to receive. That means I’ll have to add an extra error path for them, which feels suboptimal. Thankfully, since grammars are just funky classes, I can easily introduce variants of the parser that just accept client and server commands:

grammar Stomp::Parser::ClientCommands is Stomp::Parser {
    token command {
        <
            SEND SUBSCRIBE UNSUBSCRIBE BEGIN COMMIT ABORT ACK NACK
            DISCONNECT CONNECT STOMP
        >
    }
}

grammar Stomp::Parser::ServerCommands is Stomp::Parser {
    token command {
        < CONNECTED MESSAGE RECEIPT ERROR >
    }
}

And yes, I added tests to cover these too, in the resulting commit.

From parse tree to message

It’s fairly common in Perl 6 for a grammar to come paired with actions, which process the raw parse tree into a higher level data structure. I certainly have a desired data structure: Stomp::Message. So how is it being made today? Here is the code in question:

while Stomp::Parser::ServerCommands.subparse($buffer) -> $/ {
    $buffer .= substr($/.chars);
    if $<command> eq 'ERROR' {
        die ~$<body>;
    }
    else {
        emit Stomp::Message.new(
            command => ~$<command>,
            headers => $<header>
                .map({ ~.<header-name> => ~.<header-value> })
                .hash,
            body => ~$<body>
        );
    }
}

Clearly, part of this would end up getting duplicated in a Stomp::Server, so it’d be better pulled out, and stuck in an actions class. So, I’ll define an actions class nested inside my grammar, and put the logic there:

grammar Stomp::Parser {
    ...

    class Actions {
        method TOP($/) {
            make Stomp::Message.new(
                command => ~$<command>,
                headers => $<header>
                    .map({ ~.<header-name> => ~.<header-value> })
                    .hash,
                body => ~$<body>
            );
        }
    }
}

It’s nice to notice how this is basically a cut-paste refactor. Now for a test:

{
    my $parsed = Stomp::Parser.parse(qq:to/TEST/);
        SEND
        destination:/queue/stuff

        Much wow\0
        TEST
    ok $parsed, "Parsed message with header/body";

    my $msg = $parsed.made;
    isa-ok $msg, Stomp::Message, "Parser made a Stomp::Message";
    is $msg.command, "SEND", "Command is correct";
    is $msg.headers, { destination => "/queue/stuff" }, "Header is correct";
    is $msg.body, "Much wow", "Body is correct";
}

The test fails, because I forgot to set the actions class when calling parse. Hmm…I’d need to do that in Stomp::Client too…and in Stomp::Server. In fact, I don’t have an example off hand when I’d care to avoid producing a Stomp::Message. That probably means it wants to be the default. That’s easily taken care of by overriding parse and subparse to set the actions by default:

method parse(|c) { nextwith(actions => Actions, |c); }
method subparse(|c) { nextwith(actions => Actions, |c); }

I use |c to swallow up all the incoming arguments, and then pass them along. Notice how I take care to put my default first, and then splice in anything the caller specifies. This means there’s still a way to provide alternate actions, or to pass Nil to get none at all. Test passes. Commit. Yay.

Finally, I can go back and tidy up the code in the buffer processing some:

method !process-messages($incoming) {
    supply {
        my $buffer = '';
        whenever $incoming -> $data {
            $buffer ~= $data;
            while Stomp::Parser::ServerCommands.subparse($buffer) -> $/ {
                given $/.made -> $message {
                    die $message.body if $message.command eq 'ERROR';
                    emit $message;
                }
                $buffer .= substr($/.chars);
            }
        }
    }
}

It no longer needs to dig into the parse tree to find the command and body for the error handling. Generally, the code in this method is much more focused on doing a single thing: turning a stream of incoming characters into a stream of messages, coping with messages that fall over packet boundaries. Win!

Simplifying the actions

Refactoring feels nicer when there’s tests. So, is there anything of the code I now have nicely covered that I fancy cleaning up? Well, perhaps there is a little bit of simplification on offer in my small Actions class:

class Actions {
    method TOP($/) {
        make Stomp::Message.new(
            command => ~$<command>,
            headers => $<header>
                .map({ ~.<header-name> => ~.<header-value> })
                .hash,
            body => ~$<body>
        );
    }
}

For one, I don’t actually need to explicitly do the hash coercion there. The default semantics of construction perform assignment, not binding, and a list of pairs can happily be assigned to a hash. That map is digging into the parse tree too, and it’d probably be neater to do handle the pair construction in a second action method. So, here goes:

class Actions {
    method TOP($/) {
        make Stomp::Message.new(
            command => ~$<command>,
            headers => $<header>.map(*.made),
            body    => ~$<body>
        );
    }
    method header($/) {
        make ~$<header-name> => ~$<header-value>;
    }
}

I think I like that better. Not really any shorter, but breaks the work up into smaller chunks for easier digesting of the code. So, it’s in.

Pretty nice progress

That’ll do me for this time. By now, I’ve got the things I’d need to build my Stomp::Server module nicely factored out. Better still, they’re covered by some tests. Stomp::Client itself is now much more focused, and down to under a hundred lines of code.

Next, I’ll want to look into getting some testing in place for Stomp::Client. And that will mean taking a little diversion: there’s no test double in the ecosystem for IO::Socket::Async yet, so I’ll need to build one.

Posted in Uncategorized | Leave a comment

Not guts, but 6: part 1

After the Christmas release of Perl 6, I spent the better part of a week in bed, exhausted and somewhat sick. I’m on the mend, but I’m going to be taking it easy for the coming weeks. I suspect it’ll be around February before I’m feeling ready for my next big adventures in Perl 6 compiler/VM hacking. It’s not so much a matter of not having motivation to work on stuff; I’ve quite a lot that I want to do. But, having spent six months where I was never quite feeling well, just somewhere between not well and tolerably OK, I’m aware I need to give myself some real rest, and slowly ease myself back into things. I’ll also be keeping my travel schedule very light over the coming months. The Perl 6 Christmas preparations were intense and tiring, but certainly not the only thing to thank for my exhaustion. 3-4 years of always having a flight or long-distance train trip in the near future – and especially the rather intense previous 18 months – has certainly taken its toll. So, for the next while I’ll be enjoying some quality time at home in Prague, enjoying nice walks around this beautiful city and cooking plenty of tasty Indian dishes.

While I’m not ready to put compiler hat back on yet, I still fancied a little gentle programming to do in the next week or two. And, having put so much effort into Perl 6, it’s high time I got to have the fun of writing some comparatively normal code in it. :-) So, I decided to take the STOMP client I hacked up in the space of an hour for my Perl 6 advent post, and flesh it out into a full module. As I do so, I’m going to blog about it here, because I think in doing so I’ll be able to share some ideas and ways of doing things that will have wider applicability. It will probably also be a window into some of the design thinking behind various Perl 6 things.

Step 0: git repo

I took the code from the blog post, and dropped it into lib/Stomp/Client.pm6. Then it was git init, git add, git commit, and voila, I’m ready to get going. I also decided to use Atom to work on this, so I can enjoy the nice Perl 6 syntax highlighting plug-in.

Testing thinking

Since my demos for the blog post actually worked, it’s fairly clear that I at this point have “working code”. Unfortunately, it also has no tests whatsoever. That makes me uneasy. I’m not especially religious about automated testing, I just know there have been very few times where I wrote tests and regretted spending time doing so, but a good number of times when I “didn’t need to spend time doing that” and later made silly mistakes that I knew full well would have been found by a decent suite of tests.

More than that, I find that testable designs tend to also be extensible and loosely coupled designs. That partly falls out of my belief that tests should simply be another client of the code. Sometimes on #perl6, somebody asks how to test their private methods. Of course, since I designed large parts of the MOP, I can rattle off “use .^find_private_method(‘foo’) to get hold of it, then call it” without thinking. But the more thoughtful answer is that I don’t think you should be testing private methods. They’re private. They’re an implementation detail, like attributes. My expectation in Perl 6 is that I can perform a correct refactor involving private methods or attributes without having to be aware of anything textually outside the body of the class in question. This means that flexibility for the sake of testability will need to make it into the public interface of code – and that’s good, because it will make the code more open to non-testing extension too.

My current Stomp::Client is not really open to easy automated testing. There is one non-easy way that’d work, though: write a fake STOMP server to test it against. That’s probably not actually all that hard. After all, I already have a STOMP message parser. But wait…if my module already contains a good chunk of the work needed to offer server support, maybe I should build that too. And even if I don’t, I should think about how I can share my message parser so somebody else can. And that means that rather than being locked up in my Stomp::Client class it will need to become public API. And that in turn would mean a large, complex, part of the logic…just became easily testable!

I love these kinds of design explorations, and it’s surprising how often the relatively boring question of “how will I test this” sets me off in worthwhile directions. But wait…I shouldn’t just blindly go with the first idea I have for achieving testability, even if it is rather promising. I’ve learned (the hard way, usually) that it’s nearly always worth considering more than one way to do things. That’s often harder that it should be, because I find myself way too easily attached to ideas I’ve already had, and wanting to defend them way too early against other lines of thought. Apparently this is human nature, or something. Whatever it is, it’s not especially helpful for producing good software!

Having considered how I might test it as is, let me ponder the simplest change I could make that would make the code a lot easier to test. The reason I’d need a fake server is because the code tightly couples to IO::Socket::Async. It’s infatuated with it. It hard-codes its name, declaring that we shall have no socket implementation, but IO::Socket::Async!

my $conn = await IO::Socket::Async.connect($!host, $!port);

So, I’ll change that to:

my $conn = await self.socket-provider.connect($!host, $!port);

And then add this method:

method socket-provider() {
    IO::Socket::Async
}

And…it’s done! My tests will simply need to do something like:

my \TestClient = Stomp::Client but role {
    method socket-provider() {
        Fake::Client::Socket
    }
}

And, provided I have some stub/mock/fake implementation of the client bits of IO::Socket::Async, all will be good.

But wait, there’s more. It’s also often possible to connect to STOMP servers using TLS, for better security. Suppose I don’t support that in my module. Under the previous design, that would have been a blocker. Now, provided there’s some TLS module that provides the same interface as IO::Socket::Async, it’ll be trivial to use it together with my Stomp::Client. Once again, thinking about testability in terms of the public interface gives me an improvement that is entirely unrelated to testability.

I liked this change sufficiently I decided it was time to commit. Here it is.

Exposing Message

I’m a big fan of the aggregate pattern. Interesting objects often end up with interesting internal structure, which is best expressed in terms of further objects. Since classes, grammars, roles and the like can all be lexically scoped in Perl 6, keeping such things hidden away as implementation details is easy. It’s how I tend to start out. For example, my Message class, representing a parsed STOMP message, is lexical and nested inside of the Stomp::Client class:

class Stomp::Client {
    my class Message {
        has $.command;
        has %.headers;
        has $.body;
    }

    ...
}

The grammar for parsing messages is even lexically scoped inside of the one method that uses it! Lexical scoping is another of those things Perl 6 offers for keeping code refactorable. In fact, it’s an even stronger one than private attributes and methods offer. Those you can go and get at using the MOP if you really want. There’s no such trickery on offer with lexical scoping.

So, that’s how I started out. But, by now, I know that for both testing and later implementing a Stomp::Server module, I’d like to pull Message out. So, off to a Stomp/Message.pm6 it goes. Since it was lexical before, it’s easy to fix up the references. In fact, the Perl 6 compiler will happily tell me about them at compile time, so I can be happy I didn’t miss any. (It turns out there is only one). Another commit.

Oh, behave!

At the point I expose a class to the world, I find it useful to step back and ask myself what it’s responsibilities are. Right now, the answer seems to be, “not much!” It’s really just a data object. But generally, objects aren’t just data. They’re really about behaviour. So, are there any behaviours that maybe belong on a Message object?

Looking through the code, I see this:

await $conn.print: qq:to/FRAME/;
    CONNECT
    accept-version:1.2
    login:$!login
    passcode:$!password

    \0
    FRAME 

And, later, this:

$!connection.print: qq:to/FRAME/;
    SEND
    destination:/queue/$topic
    content-type:text/plain

    $body\0
    FRAME

There’s another such case too, for subscribe. It’s quite easy for a string with a bit of interpolation to masquerade as being too boring to care about. But what I really have here is knowledge about how is STOMP message formed scattered throughout my code. As this module matures from 1-hour hack to a real implementation of the STOMP spec, this is going to have to respect a number of encoding rules – or risk being vulnerable to injection attacks. (All injection attacks really come from failing to treat languages as languages, and instead just treating them as strings that can be stuck together.) And logic that will therefore even be security sensitive absolutely does not want scattering throughout my code.

So, I’ll move the logic to Stomp::Message. First, a failing test goes into t/message.t:

use Test;
use Stomp::Message;

plan 1;

my $msg = Stomp::Message.new(
    command => 'SEND',
    headers => ( destination => '/queue/stuff' ),
    body    => 'Much wow');
is $msg, qq:to/EXPECTED/, 'SEND message correctly formatted';
    SEND
    destination:/queue/stuff

    Much wow\0
    EXPECTED

I find it reassuring to see a test actually fail before I do the work to make it pass. It tells me I actually did something. Now for the implementation:

method Str() {
    qq:to/END/
        $!command
        %!headers.fmt('%s:%s')

        $!body\0
        END
}

The fmt method is one of those small, but valuable Perl 6 features. It’s really just a structure-aware sprintf. On hashes, it can be given a format string for each key and value, along with a separator. The default separator is \n, which is exactly what I need, so I don’t need to pass it. This neatly takes a loop out of my code, and means I can lay out my heredoc to look quite like the message I’m producing. Here’s the change.

Construction tweaks

With a passing test under my belt, I’d like to ponder whether there’s any more interesting tests I might like to write Right Now for Stomp::Message. I know I will need to make a pass through the spec for encoding rules, but that’s for later. Putting that aside, however, are there any other ways that I might end up with my Stomp::Message class producing malformed messages?

The obvious risk is that an instance may be constructed with no command. This can never be valid, so I’ll simply forbid it. A failing test is easy:

dies-ok
    { Stomp::Message.new( headers => (foo => 'bar'), body => 'Much wow' ) },
    'Stomp::Message must be constructed with a command';

So is this fix: just mark the attribute as required!

has $.command is required;

It is allowable to have an empty body in some messages. At present, it kind of supports that without having to pass it explicitly, but there will be a warning. The fix is 4 characters. It’s really rather borderline whether this is worth a test, for me. But I’ll write one anyway:

{
    my $msg = Stomp::Message.new(
        command => 'CONNECT',
        headers => ( accept-version => '1.2' ));
    is $msg, qq:to/EXPECTED/, 'CONNECT message with empty body correctly formatted';
        CONNECT
        accept-version:1.2

        \0
        EXPECTED
    CONTROL {
        when CX::Warn { flunk 'Should not warn over uninitialized body' }
    }
}

It fails. And then I do:

has $.body = '';

And it passes. The boilerplate there makes me thing there’s some market for an easier way to express “it doesn’t warn” in a test, but I’ll leave that yak for somebody else.

Those went in as two commits, because they’re two separate changes. I like to keep my commits nice and atomic that way.

Eliminating the duplication

Finally, I go and replace the various places that produced formatted STOMP messages with use of the Stomp::Message class:

$!connection.print: Stomp::Message.new:
    command => 'SUBSCRIBE',
    headers => (
        destination => "/queue/$topic",
        id => $id
    );

3 changes, 1 commit, done.

Enough for this time!

Next time, I’ll be taking a look at factoring out the parser, and writing some tests for it. Beyond that, there’ll be faking the async socket API, supporting unsubscription from topics, building STOMP server support, and more.

Posted in Uncategorized | 2 Comments

Reflecting, celebrating, and looking forward

As I write, the Perl 6 Christmas release is taking place. It goes without saying that it’s been a long journey to this point. I’ve walked less than half of it, joining the effort 7-8 years ago through. Back then, I was fresh from university, had enjoyed the courses on compiler construction, type systems, and formal semantics, and was looking for an open source project in one or more of these fields to contribute to.

For many years before I got involved with Perl 6, I’d been using Perl extensively for web application development. It helped me live more comfortably through my college and university years, and I had sufficient work coming in that I kept a few other part-timers in a job too. That would be reason enough for being fond of a language, but there was more. Perl is really the language where I “grew up” as a programmer. It’s the first language where I used regexes and closures, it helped develop my early understanding of OOP, and it introduced me to testing culture. It’s the first language where I went to a conference, and realized the value a language’s community can have. All of this has been foundational to what I’ve done in my career since then, which besides Perl 5/6 has seen me deliver code and training in a wide range of languages, including C, C#, Java, and Python.

Through using Perl I got to know about the Perl 6 project – and it seemed a good fit with my interest in languages, compilers, and types. I knew it was a long-running project with plenty of history, but looking at what it was trying to achieve, I was convinced it was worth trying to help out a little to make it happen. It’s surprising just how successful Perl 6 has been over the years at attracting really great people to work on it – knowing full well its long, and at times difficult, history.

From patcher to architect

As usual with anyone new to an open source project, my first contributions were small. Portability things, small fixes here and there, minor features, and the like. Over time, I found myself taking on increasingly large language features. By 2009 I was a regular contributor, and had a decent grasp of much of the Rakudo compiler code base. I caused Patrick Michaud, the lead developer of Rakudo at the time, a good number of “oh no!” moments, as I didn’t yet have a great picture of the overall design – but was putting in notable features anyway. He, and others, did a great job of steering me gently in the right kind of direction.

2010 probably goes down as the most difficult year of my involvement in the Perl 6 project. The first Rakudo Star, a “useful, usable, early adopter’s release of Perl 6”, was generally graded as “not good enough”. More problematic, as I took some steps back and reflected on how to rectify this, was that the issues went right to the very heart of the architecture of Rakudo itself. Rakudo in those days was a traditional compiler: you fed it code in Perl 6, it spat out code in an intermediate language, which was executed on the Parrot VM. Of course, this is what you learn a compiler does at university, and it’s all very clean and simple…and not at all what Perl 6 demands.

Compile time and runtime are not so cleanly distinct concepts in Perl. They never have been, thanks to things like BEGIN blocks. But in Perl 6, we really wanted to both handle BEGIN time – and have lots of meta-programming stuff going on at BEGIN time – and still be able to have separate compilation. This, it turns out, is a tricky problem. And, as I looked into how to solve it, it became clear that it was going require a deep, drawn out, overhaul. Further, it became clear that this was an effort I was going to need to play architect on and largely lead. History has shown it to be the right call; the architecture put in place then has largely survived intact to today’s release, and it enabled a lot of the great things we simply take for granted today. But at the time, it was a lonely path to walk.

In the years since then, I led the way on getting Rakudo ported to the JVM – and have been happy to see that work taken forward by others. I was also a founder of MoarVM, the VM that the Perl 6 Christmas release primarily targets. Most recently, I took up the long-neglected task of getting Perl 6’s concurrency design into decent shape, doing the initial implementations of the features. It goes without saying that none of this would have been possible without the incredible bunch of people in the Perl 6 community who not only contributed their great technical skills to these efforts, but also a good deal of encouragement and friendship along the way.

For 7 years I made a really great job of just being “this guy who hacks on stuff” while managing to disclaim wearing any particular hat – until Larry went and called me out as architect at the Swiss Perl Workshop this last summer. It’s a role I’m proud to hold for the moment, and I look forward to continuing to contribute to the Perl 6 project for some years to come.

So, about the release…

By this point into writing the post – the Christmas release of Perl 6 has already taken place! Hurrah! But…what does it mean?

First, it’s important to understand that we’ve actually released two things (and that there are a couple more to come in the next days).

The first of these is the specification of the Perl 6 language itself, which is expressed as a suite of over 120,000 tests. Versions of the Perl 6 language will be named after festivals or celebrations; our alpha was Advent, our beta was Birthday, and we’re now at Christmas. This is referred to as “Perl 6 Christmas”, or in short as Perl 6.c. The next major language release will most likely be called Diwali, though I’m not sure we’ve worked out how to spell it yet. :-)

The second is a release of the Rakudo Perl 6 compiler that complies with this specification. We don’t imagine we’ll manage to stop people blurring language specification and language implementation, and know full well that when most people say they want “Perl 6 Christmas” they actually want a compiler that implements that version of the language. All the same, it’s a valuable distinction, as it means we remain open to alternative implementations – something that may not be that important now, but may be in a decade or two.

In the coming days, we’ll also produce a Rakudo Star release – which consists of the compiler along with documentation and a selection of modules – and that will also have an MSI, to make life easier for Windows folks.

What happens next?

For me? Rest. A lot of rest. Really a lot of rest. It’s been an exhausting last few months in the run up to the release, chasing down lots of little semantic details that we really wanted to get straightened out ahead of the freezing of the Perl 6 Christmas language specification. It was worth it, and I’m really happy with the language we’ve ended up with. But now it’s time to take care of myself for a while.

Come 2016, the work will go on. However, the focus will shift. For compiler and runtime folks like me, the focus will be largely on performance and reliability engineering. Now we have a stable language definition, it makes much more sense to invest more heavily in optimizing things. That isn’t to say a great deal of optimization work hasn’t already taken place; many of us have worked hard on that. But there’s a lot more that can, and will, be done.

Even more important than that work, however, will be the work that takes place on the Perl 6 ecosystem in the year to come. Since we announced the Christmas target for a stable language/compiler release, a number of new faces showed up up to help with writing modules and building supporting infrastructure. Now, their work won’t have to contend with us compiler hackers breaking the world under them every week – and that hopefully will encourage more to dive into the ecosystem work also. Maturity here will take time, but there’s plenty of expertise and wisdom on these matters in the Perl community.

Rakudo will stick to a monthly release cycle. We’ll be making a number of process changes to help us deliver those monthlies at a higher quality, especially with regard to not regressing on the 6.c language test suite, key modules, and ecosystem tooling. These changes will also introduce a stability level that lies between bleeding edge commits and monthlies. We also expect the language specification itself to have a small number of minor versions between now and Diwali, and we will treat these a lot like we have the Christmas release, with some extra attention going into the release than normal monthlies will get. Those releases will tend to have seen a greater focus on semantics detail, so will for now serve as our “stable track” for those who want something more occasional than the monthlies. We’ll see how that serves us and our userbase, and adjust as needed. It’s all about keeping the ceremony of contributing and releasing low, while keeping the quality of releases up.

Last but not least…

…I’d like to say thank you. Thank you to all those who have been my fellow contributors on the Perl 6 project, for being among the best people I’ve ever worked with on anything. Thank you to all those who came to my Perl conference talks and read my ramblings here over the years, and provided feedback and encouragement. Thank you to those who donated financially to the Perl 6 project, and so enabled Perl 6 to be part of my day job. And last, but absolutely not least, thank you to those of you who have written and run Perl 6 programs over the years, and shared that you were doing so – because perhaps the greatest reward of all for a compiler/VM hacker is seeing others use their work to build their own great creations.

Together, we’ve breathed life into a new Perl. I’m damn proud of what we’ve built together – and I can’t wait to see how people put it to work.

Merry Christmas!

Posted in Uncategorized | 13 Comments

Getting closer to Christmas

The list of things we “really want to do” before the Christmas release is gradually shrinking. Last time I wrote here, the xmas RT list was around 40 tickets. Now it’s under 20. Here’s an overview of the bits of that I’ve been responsible for.

Supply API cleanup

I did the original implementation of supplies a couple of years back. I wasn’t sure how they would be received by the wider community, so focused on just getting something working. (I also didn’t pick the name Supply; Larry is to thank for that, and most of the other naming). Supplies were, it turns out, very well received and liked, and with time we fleshed out the available operations on supplies, and 6 months or so back I introduced the supply, whenever, and react syntactic sugar for supplies.

What never happened, however, was a cleanup of the code and model at the very heart of supplies. We’ve had to “build one to throw away” with nearly everything in Perl 6, because the first implementation tended to show up some issues that really wanted taking care of. So it was with supplies. Thankfully, since everything was built on a fairly small core, this was not to be an epic task. And, where the built-ins did need to be re-worked, it could be
done much more simply than before by using the new supply/whenever syntax.

While much of the cleanup was to the internals, there are some user-facing things. The most major one is a breaking change to code that was doing Supply.new to create a live supply. As I started cleaning up the code, and with experience from using related APIs in other languages, it became clear that making Supply be both the thing you tapped and the thing that was used to publish data was a design mistake. It not only would make it harder to trust Supply-based code and enforce the Supply protocol (that is, emit* [done | quit]),
but it also would make it hard to achieve the good performance by forcing extra sanity checks all over the place.

So, we split it up. You now use a Supplier in order to publish data, and obtain a Supply from it to expose to the world:

# Create a Supplier
my $supplier = Supplier.new;

# Get a Supply from it
my $supply = $supplier.Supply;
$supply.tap({ .say });

# Emit on it
$supplier.emit('oh');
$supplier.emit('hai');

This also means it’s easy to keep the ability to emit to yourself, and expose the ability to subscribe:

class FTP::Client {
    has $!log-supplier = Supplier.new;
    has $.log = $!log-supplier.Supply;
    ...
}

Since you can no longer call emit/done/quit on a Supply, you can be sure there won’t be values getting sneaked in unexpectedly.

The other change is that we now much more strongly enforce the supply protocol (that is, you’ll never see another emit after a done/quit unless you really go out of your way to
do so) and that only value will be pushed through a chain of supplies at a time (which prevents people from ending up with data races). Since we can ask supplies if they are
already sane (following protocol and serial (one at a time), we can avoid the cost of enforcing it at every step along the way, which makes things cheaper. This is just one of the
ways performance has been improved. We’ve some way to go, but you can now push into the hundreds of thousands of messages per second through a Supply.

Along the way, I fixed exceptions accidentally getting lost when unhandled in supplies in some cases, a data loss bug in Proc::Async and IO::Socket::Async, and could also resolve the RT complaining that the supply protocol was not enforced.

Preparing ourselves for stronger back-comparability

Once the Perl 6 Christmas release of the language is made, we’ll need to be a lot more careful about not breaking code that’s “out there”. This will be quite a change from
the last months, where we’ve been tweaking lots of things that bothered us. To help us with this change, I wrote up a proposal on how we’ll manage not accidentally changing tests that are part of the Perl 6 Christmas language definition, allow code to be marked with the language version it expects, and how we’ll tweak our process to give us a better chance of shopping solid releases that do not introduce regressions. Further feedback is still welcome; as with all development process things, I expect this to continue to evolve over the years.

I/O API cleanups

A few tickets complained about inconsistencies in a few bits of the I/O APIs, such as the differing ways of getting supplies of chars/bytes for async processes, sockets, and files. This has received a cleanup now. The synchronous and asynchronous socket APIs also got a little further alignment, such that the synchronous sockets now also have connect and listen factory methods.

Bool is now a real enum

This is a years old issue that we’ve finally taken care of in time for the release: Bool is now a real enum. It was mostly tricky because Bool needs setting up really quite early on in the language bootstrap. Thankfully, nine++ spent the time to figure out how do to this. His
patch nearly worked – but ran into an issue involving closure semantics with BEGIN and COMPOSE blocks. I fixed that, and was able to merge in his work.

Interaction of start and dynamic variables

A start block can now see the dynamic variables where that were available where it was started.

my $*TARGET_DIR = 'output/';
await start { say $*TARGET_DIR } # now works

Correcting an array indexing surprise

Indexing with a range would always auto-truncate to the number of elements in an array:

my @a = 1, 2, 3;
say @a[^4]; # (1 2 3)

While on the surface this might be useful, it was rather good at confusing people who expected this to work:

my @a;
@a[^2] = 1, 2;
say @a;

Since it auto-truncated to nothing, no assignment took place. We’ve now changed it so only ranges whose iterators are considered lazy will auto-truncate.

my @a = 1, 2, 3;
say @a[^4]; # (1 2 3 (Any)) since not lazy
say @a[0..Inf] # (1 2 3) since infinite is lazy
say @a[1..Inf] # (2 3) since infinite is lazy
say @a[lazy ^4] # (1 2 3) since marked lazy

Phaser fixes

I fixed a few weird bugs involving phasers.

  • RT #123732 noted that return inside of a NEXT phaser but outside of a routine would just cause iteration to go to the next value, rather than give an error (it now does, and a couple of similarly broken things also do)
  • RT #123731 complained that the use of last in a NEXT phaser did not correctly exit the loop; it now does
  • RT #121147 noted that FIRST only worked in for loops, but not other loops; now it does

Other smaller fixes

Here are a number of other less notable things I did.

  • Fix RT #74900 (candidate with zero parameters should defeat candidate with optional parameter in no-arg multi dispatch)
  • Tests covering RT #113892 and RT #115608 on call semantics (after getting confirmed that Rakudo already did the right thing)
  • Review RT #125689, solve the issue in a non-hacky way, and add a test to cover it
  • Fix RT #123757 (semantics of attribute initializer values passed to constructor and assignemnt was a tad off)
  • Hunt down a GC hang blocking module precomp branch merge; hopefully fix it
  • Review socket listen backlog patch; give feedback
  • Write up rejection of RT #125400 (behavior of unknown named parameters on methods)
Posted in Uncategorized | 1 Comment