Testing Protocols

In the chapter on checking outputs, we already have seen how to interact with external programs. In this chapter, we will extend this concept to full protocol testing across networks. This includes:

• Acting as a network client and interacting with network servers using generated inputs

• Acting as a network server and interacting with network clients using generated inputs.

Under Construction

Protocol testing is currently in beta. Check out the list of open issues.

Interacting with an SMTP server

The Simple Mail Transfer Protocol (SMTP) is, as the name suggests, a simple protocol through which mail clients can connect to a server to send mail to recipients. A typical interaction with an SMTP server smtp.example.com, sending a mail from bob@example.org to alice@example.com, is illustrated below:

        sequenceDiagram
    SMTP Client->>SMTP Server: (connect)
    SMTP Server->>SMTP Client: 220 smtp.example.com ESMTP Postfix
    SMTP Client->>SMTP Server: HELO relay.example.org
    SMTP Server->>SMTP Client: 250 Hello relay.example.org, glad to meet you
    SMTP Client->>SMTP Server: MAIL FROM:<bob@example.org>
    SMTP Server->>SMTP Client: 250 Ok
    SMTP Client->>SMTP Server: RCPT TO:<alice@example.com>
    SMTP Server->>SMTP Client: 250 Ok
    SMTP Client->>SMTP Server: DATA
    SMTP Server->>SMTP Client: 354 End data with <CR><LF>.<CR><LF>
    SMTP Client->>SMTP Server: From: "Bob Example" <bob@example.org>
    SMTP Client->>SMTP Server: To: "Alice Example" <alice@example.com>
    SMTP Client->>SMTP Server: Subject: Protocol Testing with I/O Grammars
    SMTP Client->>SMTP Server: (mail body)
    SMTP Client->>SMTP Server: .
    SMTP Server->>SMTP Client: 250 Ok: queued as 12345
    SMTP Client->>SMTP Server: QUIT
    SMTP Server->>SMTP Client: 221 Bye
    SMTP Server->>SMTP Client: (closes the connection)
    

Our job will be to automate this interaction using Fandango. For this, we need two things:

1. An SMTP server to send commands to

2. A .fan spec that encodes this interaction.

An SMTP server for experiments

For illustrating protocol testing, we need to run an SMTP server, which we will run locally on our machine. (No worries - the local SMTP server can not actually send mails across the Internet.)

The Python aiosmtpd server will do the trick:

$ pip install aiosmtpd

Once installed, we can run the server locally; normally, it runs on port 8025:

$ python -m aiosmtpd -d -n
INFO:mail.log:Server is listening on localhost:8025

We can now connect to the server on the given port and send it commands. The telnet command is handy for this. We give it a hostname (localhost for our local machine) and a port (8025 for our local SMTP server.)

Once connected, anything we type into the telnet input will automatically be relayed to the given port, and hence to the SMTP server. For instance, a QUIT command (followed by Return) will terminate the connection.

$ telnet localhost 8025
Trying ::1...
Connected to localhost.
Escape character is '^]'.
220 localhost.example.com Python SMTP 1.4.6
QUIT
221 Bye
Connection closed by foreign host.

Try this for yourself! What happens if you invoke telnet, introducing yourself with HELO client.example.com?

Solution

When sending HELO client.example.com, the server replies with its own hostname. This is the name of the local computer, in our example localhost.example.com.

$ telnet localhost 8025
Trying ::1...
Connected to localhost.
Escape character is '^]'.
220 localhost.example.com Python SMTP 1.4.6
HELO client.example.com
250 localhost.example.com
QUIT
221 Bye

A simple SMTP grammar

We use telnet only for illustrative purposes here; later in this chapter, you will see how to have Fandango directly connect to servers (and clients!)

With fandango talk, we have seen a Fandango facility that allows us to connect to the standard input and output channels of a given program and interact with it. The idea would now be to use the telnet program for this very purpose. By invoking

$ fandango talk -f smtp-telnet.fan telnet 8025

we could interact with the telnet program as described above. All we now need is a grammar that describes the telnet interaction.

The following grammar has two parts:

1. First, we expect some output from the telnet program.

2. Then, we interact with the SMTP server - just sending a QUIT command and then exiting.

A typical interaction thus would be:

        sequenceDiagram
    Fandango->>telnet: (invoke)
    telnet->>Fandango: Trying ::1...
    telnet->>Fandango: Connected to localhost.
    telnet->>Fandango: Escape character is '^]'.
    SMTP Server (via telnet)->>Fandango: 220 localhost.example.com Python SMTP 1.4.6
    Fandango->>SMTP Server (via telnet): QUIT
    SMTP Server (via telnet)->>Fandango: 221 Bye
    SMTP Server (via telnet)->>telnet: (closes connection)
    telnet->>Fandango: (ends execution)
    

The following I/O grammar smtp-telnet.fan implements this interaction via telnet:

1. First, <telnet-intro> lets Fandango expect the telnet introduction;

2. Then, <smtp> takes care of the actual SMTP interaction.

<start> ::= <Out:telnet_intro> <smtp>
<telnet_intro> ::= \
    r"Trying.*" "\r\n" \
    r"Connected.*" "\r\n" \
    r"Escape.*" "\r\n"

<smtp> ::= <Out:m220> <In:quit> <Out:m221>
<m220> ::= "220 " r".*" "\r\n"
<quit> ::= "QUIT\r\n"
<m221> ::= "221 " r".*" "\r\n"

Note

Again, note that In and Out describe the interaction from the perspective of the program under test; hence, Out is what telnet and the SMTP server produce, whereas In is what the SMTP server (and telnet) get as input.

With this, we can now connect to our (hopefully still running) SMTP server and actually send it a QUIT command:

$ fandango talk -f smtp-telnet.fan telnet 8025

To track the data that is actually exchanged, use the verbose -v flag. The In: and Out: log messages show the data that is being exchanged.

$ fandango -v talk -f smtp-telnet.fan telnet 8025

Interacting as Network Client

Using telnet to communicate with servers generally works, but it has a number of drawbacks. Most importantly, telnet is meant for human interaction. Hence, our I/O grammars have to reflect the telnet output (which actually might change depending on operating system and configuration); also, telnet is not suited for transmitting binary data.

Fortunately, Fandango offers a means to be invoked directly as a network client, not requiring external programs such as telnet. The fandango talk option --client allows Fandango to be used as a network client. The argument to --client is a network address to connect to. In the simplest form, it is just a port number on the local machine.

Hence, to have Fandango act as an SMTP client for the local server, we can enter

$ fandango talk -f SPEC.fan --client 8025

Since Fandango directly talks to the SMTP server now, we can also simplify the grammar by removing the <telnet_intro> part. Also, there is no more In and Out parties, since we do not interact with the standard input and output of an invoked program. Instead,

• Client is the party representing the client, connecting to an external server on the network.

• Server is the party representing a server on the network, accepting connections from clients.

Consequently,

• all outputs produced by the client (and processed by the server) are prefixed with Client: in the respective nonterminals; and

• all outputs produced by the server (and processed by the client) are prefixed with Server:.

With this, we can reframe and simplify our SMTP grammar, using Client and Server to describe the respective interactions. The spec smtp-simple.fan reads as follows:

<start> ::= <smtp>
<smtp> ::= <Server:m220> <Client:quit> <Server:m221>
<m220> ::= "220 " <hostname> " " r".*" "\r\n"
<quit> ::= "QUIT\r\n"
<m221> ::= "221 " r".*" "\r\n"

<hostname> ::= r"[-a-zA-Z0-9.:]*" := "host.example.com"

Note how we added <hostname> as additional specification of the hostname that is typically part of the initial server message.

With this, we have Fandango act as client and connect to the (hopefully still running) server on port 8025:

$ fandango talk -f smtp-simple.fan --client 8025

        sequenceDiagram
    Fandango->>SMTP Server: (connect)
    SMTP Server->>Fandango: 220 host.example.com <more data>
    Fandango->>SMTP Server: QUIT
    SMTP Server->>Fandango: 221 <more data>
    SMTP Server->>Fandango: (closes connection)
    

From here on, we can have Fandango directly “talk” to network components such as servers.

Interacting as Network Server

Obviously, our SMTP specification is still very limited. Before we go and extend it, let us first highlight a particular Fandango feature. From the same specification, Fandango can act as a client and as a server. When invoked with the --server option, Fandango will create a server at the given port and accept client connections. So if we invoke

$ fandango talk -f smtp-simple.fan --server 8125

we can then connect to our running Fandango “SMTP Server” and interact with it according to the smtp-simple.fan spec:

$ telnet localhost 8125
Trying ::1...
Connected to localhost.
Escape character is '^]'.
220 host.example.com 26%
QUIT
221 26yn

As server, Fandango produces its own 220 and 221 messages, effectively fuzzing the client. Note how the interaction diagram reflects how Fandango is now taking the role of the client:

        sequenceDiagram
    SMTP Client (or telnet)->>Fandango: (connect)
    Fandango->>SMTP Client (or telnet): 220 host.example.com <random data>
    SMTP Client (or telnet)->>Fandango: QUIT
    Fandango->>SMTP Client (or telnet): 221 <random data>
    Fandango->>SMTP Client (or telnet): (closes connection)
    

Under Construction

Fandango can actually create and mock an arbitrary number of clients and servers, all interacting with each other. The interface for this is currently under construction.

A Bigger Protocol Spec

So far, our SMTP server is not great at testing SMTP clients – all it can handle is a single QUIT command. Let us extend it a bit with a few more commands, reflecting the interaction in the introduction:

<start> ::= <connect>
<connect> ::= <Server:id> <helo>

<id> ::= '220 ' <hostname> ' ESMTP Postfix\r\n'
<hostname> ::= r"[-a-zA-Z0-9.:]+" := "host.example.com"

<helo> ::= <Client:HELO> \
    (<Server:hello> <mail_from> | <Server:error>)
<HELO> ::= 'HELO ' <hostname> '\r\n'

<hello> ::= '250 Hello ' <hostname> ', glad to meet you\r\n' \
    <mail_from>

<error> ::= '5' <digit> <digit> ' ' <error_message> '\r\n'
<error_message> ::= r'[^\r]*' := "Error"

<mail_from> ::= <Client:MAIL_FROM> \
    (<Server:ok> <mail_to> | <Server:error>)

<MAIL_FROM> ::= 'MAIL FROM:<' <email> '>\r\n'
# Actual email addresses are much more varied
<email> ::= r"[-a-zA-Z0-9.]+" '@' <hostname> := "alice@example.com"

<ok> ::= '250 Ok\r\n'

<mail_to> ::= <Client:RCPT_TO> \
    (<Server:ok> <data> | <Server:ok> <mail_to> | <Server:error>)

<RCPT_TO> ::= 'RCPT TO:<' <email> '>\r\n'

<data> ::= <Client:DATA> <Server:end_data> <Client:message> \
    (<Server:ok> <quit> | <Server:error>)

<DATA> ::= 'DATA\r\n'

<end_data> ::= '354 End data with <CR><LF>.<CR><LF>\r\n'

<message> ::= r'[^.\r\n]*\r\n[.]\r\n'

<quit> ::= <Client:QUIT> <Server:bye>

<QUIT> ::= 'QUIT\r\n'

<bye> ::= '221 Bye\r\n'

This spec can actually handle the initial interaction (check it!). You may note the following points:

First, the commands (and replies) follow a particular order, implying the state the server and client are in. In the “happy” path (assuming no errors), this is the order of possible commands:

        stateDiagram
    [*] --> 1: ‹connect›
    1 --> 2: ‹helo›
    2 --> 3: ‹mail_from›
    3 --> 3: ‹mail_to›
    3 --> 4: ‹mail_to›
    4 --> 5: ‹data›
    5 --> [*]: ‹quit›
    

Note how the I/O grammar (and the above state diagram) accepts multiple <mail_to> interactions, allowing mails to be sent to multiple destinations.

Second, the spec actually accounts for errors, always entering an <error> state if the client command received cannot be parsed properly. Hence, the state diagram induced in the above grammar actually looks like this:

        stateDiagram
    [*] --> 1: ‹connect›
    1 --> 2: ‹helo›
    2 --> [*]: ‹error›
    2 --> 3: ‹mail_from›
    3 --> [*]: ‹error›
    3 --> 3: ‹mail_to›
    3 --> 4: ‹mail_to›
    4 --> 5: ‹data›
    4 --> [*]: ‹error›
    5 --> [*]: ‹quit›
    5 --> [*]: ‹error›
    

As described in the chapter on checking outputs, we can use the fuzz command to actually show generated outputs of individual parties:

$ fandango fuzz --party=Client -f smtp-extended.fan

HELO host.example.com

MAIL FROM:<alice@example.com>

RCPT TO:<alice@example.com>

RCPT TO:<alice@example.com>

RCPT TO:<alice@example.com>

RCPT TO:<alice@example.com>

RCPT TO:<alice@example.com>

RCPT TO:<alice@example.com>

DATA

lgun6

.

QUIT

$ fandango fuzz --party=Server -f smtp-extended.fan

220 host.example.com ESMTP Postfix

250 Hello host.example.com, glad to meet you

250 Ok

502 Error

250 Ok

250 Ok

354 End data with <CR><LF>.<CR><LF>

250 Ok

221 Bye

That’s it for now. GO and thoroughly test your programs!