Case Study: FTP

This Fandango specification allows testing servers and clients for File Transfer Protocol (FTP, RFC 959). It demonstrates

• how a nontrivial protocol with text commands is implemented; and

• how additional channels (ClientData and ServerData) are used on demand, based on the ports returned by the FTP server

The FTP spec consists of three parts, all available for download:

• ftp.fan - the core FTP spec without specific party definitions

• ftp_client.fan - for using Fandango as an FTP client (includes ftp.fan)

• ftp_server.fan - for using Fandango as an FTP server (includes ftp.fan)

To test it, run an FTP server on the local host at port 50100, and invoke Fandango as

Omit -n 1 if you want Fandango to test until protocol coverage is achieved.

$ fandango talk -n 1 -f ftp_client.fan

Added in version 1.1: These features are available in Fandango 1.1 and later.

The FTP Parties

In contrast to other (simpler) protocols, FTP maintains two communication channels: one for control (issuing commands and getting responses), and one for data (for transferring data). In our spec, we name these ClientControl and ClientData as well as ServerControl and ServerData.

A very simple interaction involving all four, first logging in, and then sending a LIST command to get the contents of the current directory, is illustrated below.

        sequenceDiagram
    box Client
    participant ClientData
    participant ClientControl
    end
    box Server
    participant ServerControl
    participant ServerData
    end
    ClientControl ->> ServerControl: (connect)
    ServerControl ->> ClientControl: 220 FTP Server ready.
    ClientControl ->> ServerControl: USER the_user
    ServerControl ->> ClientControl: 331 Password required for the_user.
    ClientControl ->> ServerControl: PASS the_password
    ServerControl ->> ClientControl: 230 User the_user logged in.
    ClientControl ->> ServerControl: LIST
    ServerControl ->> ClientControl: 150 Opening ASCII mode data connection for file list
    ServerData ->> ClientData: (data)
    ServerControl ->> ClientControl: 226 Transfer complete.
    ClientControl ->> ServerControl: (disconnect)
    

Control Parties

In our setting, we assume that Fandango is acting as client to test an FTP server. Fandango connects to a server running on port 25521 on the local host. Whenever it receives a 150 message initiating a data transfer, it starts the ClientData party. Here, we use the instance() method to access and reconfigure the individual parties.

class ClientControl(NetworkParty):
    def __init__(self):
        super().__init__(
            connection_mode=ConnectionMode.CONNECT,
            uri="tcp://[::1]:25521"
        )
        self.start()

    def receive(self, message: str | bytes, sender: Optional[str]) -> None:
        if message.decode("utf-8").startswith("150"):
            ClientData.instance().start()

When the server sends a 226 message, this indicates the end of a data transfer; so we stop the ServerData instance to disconnect.

class ServerControl(NetworkParty):
    def __init__(self):
        super().__init__(
            connection_mode=ConnectionMode.EXTERNAL,
            uri="tcp://[::1]:25522"
        )
        self.start()

    def receive(self, message: str | bytes, sender: Optional[str]) -> None:
        super().receive(message.decode("utf-8"), sender="ClientControl")

    def send(self, message: DerivationTree, recipient: str):
        super().send(message, recipient)
        if message.to_string().startswith("226"):
            ServerData.instance().stop()

Data Parties

In our setting, the FTP data transfer takes place via port 50100 on the local host.

class ClientData(NetworkParty):
    def __init__(self):
        super().__init__(
            connection_mode=ConnectionMode.CONNECT,
            uri="tcp://[::1]:50100"
        )

    def receive(self, message: str | bytes, sender: Optional[str]) -> None:
        super().receive(message.decode("utf-8"), sender="ServerData")

class ServerData(NetworkParty):
    def __init__(self):
        super().__init__(
            connection_mode=ConnectionMode.EXTERNAL,
            uri="tcp://[::1]:50100"
        )

    def receive(self, message: str | bytes, sender: Optional[str]) -> None:
        super().receive(message.decode("utf-8"), sender="ClientData")

Connecting to the FTP Server

The interaction with an FTP server starts with the server ServerControl sending a 220 message <response_server_info> to the client ClientControl, indicating that it is ready for a new user. Afterwards, we are in the state state_logged_out_1.

<start> ::= <state_setup>
<state_setup> ::= <service_ready> <state_logged_out_1>
<service_ready> ::= <ServerControl:ClientControl:response_server_info>
<response_server_info> ::= r'(220-(?:[\x20-\x7E]*\r\n))*220 (?:[\x20-\x7E]*)\r\n'

Logging In

While we’re logged out (in state <state_logged_out_1>), we can

• via username and password (<exchange_login_...>)

Login with username and password can

• fail (<exchange_login_fail>, after which we still stay logged out); or

• succeed (<exchange_login_ok>), then, we are logged in (<state_logged_in>).

We only support logging in via username and password, so authentication via

• TLS (<exchange_auth_tls>) or

• SSL (<exchange_auth_ssl>)

are never successful, we remain logged out.

        stateDiagram
    [*] --> #lt;state_logged_out_1#gt;
    #lt;state_logged_out_1#gt; --> state_logged_out_1#gt;: #lt;exchange_auth_tls#gt; | #lt;exchange_auth_ssl#gt; | #lt;exchange_login_fail#gt;
    #lt;state_logged_out_1#gt; --> #lt;state_logged_in#gt;: #lt;exchange_login_ok
    

In our spec, this is modeled as exchanges followed by the resulting state.

<state_logged_out_1> ::= (
  <exchange_login_ok> <state_logged_in> |
  <exchange_login_fail> <state_logged_out_1>
  <exchange_auth_tls> <state_logged_out_1> |
  <exchange_auth_ssl> <state_logged_out_1> |
  )

Logging in with username and password

Our FTP server assumes one user with username the_user and a password the_password.

        sequenceDiagram
    ClientControl ->> ServerControl: (connect)
    ServerControl ->> ClientControl: 220 FTP Server ready.
    ClientControl ->> ServerControl: USER the_user
    ServerControl ->> ClientControl: 331 Password required for the_user.
    ClientControl ->> ServerControl: PASS the_password
    ServerControl ->> ClientControl: 230 User the_user logged in.
    

<exchange_login_ok> ::= (
    <ClientControl:ServerControl:request_login_user_ok>
    <ServerControl:ClientControl:response_login_user>
    <ClientControl:ServerControl:request_login_pass_ok>
    <ServerControl:ClientControl:response_login_pass_ok>
)

<request_login_user_ok> ::= 'USER the_user\r\n'
<response_login_user> ::= '331 ' <command_tail> '\r\n'
<request_login_pass_ok> ::= 'PASS the_password\r\n'
<response_login_pass_ok> ::= '230 ' <command_tail> '\r\n'

Failure to log in

There are two ways logging in can go wrong - an incorrect password (not the_password):

<wrong_user_password> ::= r'^(?!the_password$)([a-zA-Z0-9_]+)'

and an incorrect username (not the_user):

<wrong_user_name> ::= r'^(?!the_user$)([a-zA-Z0-9_]+)'

Let’s discuss these two options:

<exchange_login_fail> ::= <exchange_wrong_password> | <exchange_wrong_username>

First, we can have the client send a correct username, but a wrong password.

<exchange_wrong_password> ::= (
  <ClientControl:ServerControl:request_login_user_ok>
  <ServerControl:ClientControl:response_login_user>
  <ClientControl:ServerControl:request_login_pass_fail>
  <ServerControl:ClientControl:response_login_pass_fail>)

<request_login_user_ok> ::= 'USER the_user\r\n'
<request_login_pass_fail> ::= 'PASS ' <wrong_user_password> '\r\n'
<response_login_pass_fail> ::= '530 ' <command_tail> '\r\n'
<command_tail> ::= r'[\x20-\x7E]+'

Second, we can have the client send an incorrect username (with a correct or incorrect password).

<exchange_wrong_username> ::= (
   <ClientControl:ServerControl:request_login_user_fail>
   <ServerControl:ClientControl:response_login_user>
     (<ClientControl:ServerControl:request_login_pass_fail> |
      <ClientControl:ServerControl:request_login_pass_ok>)
   <ServerControl:ClientControl:response_login_pass_fail>
  )

<request_login_user_fail> ::= 'USER ' <wrong_user_name> '\r\n'

In both cases, we end up staying logged out (<state_logged_out_1>).

(Not) logging in via TLS and SSL

We do not support logging in via TLS and SSL and accept a 500 (syntax error) or 530 (not logged in) message from the server.

<exchange_auth_tls> ::= <ClientControl:ServerControl:request_auth_tls><ServerControl:ClientControl:response_auth_tls>
<request_auth_tls> ::= 'AUTH TLS\r\n'
<response_auth_tls> ::= r'(530|500)' ' ' <command_tail> '\r\n'

<exchange_auth_ssl> ::= <ClientControl:ServerControl:request_auth_ssl><ServerControl:ClientControl:response_auth_ssl>
<request_auth_ssl> ::= 'AUTH SSL\r\n'
<response_auth_ssl> ::= r'(530|500)' ' ' <command_tail> '\r\n'

First FTP Commands

When the client is logged in, it can send commands to the server. In the <state_logged_in> state, we support the commands listed in <logged_in_cmds>.

(The <exchange_set_type> and <exchange_set_passive> commands change the FTP state; see States below.)

        stateDiagram
    [*] --> #lt;state_logged_out_1#gt;
    #lt;state_logged_out_1#gt; --> #lt;state_logged_in#gt;: #lt;exchange_login_ok#gt;

    #lt;state_logged_in#gt; --> #lt;state_logged_in#gt;: #lt;logged_in_cmds#gt;
    #lt;state_logged_in#gt; --> #lt;state_in_binary#gt;: #lt;exchange_set_type#gt;
    #lt;state_logged_in#gt; --> #lt;state_in_passive#gt;: #lt;exchange_set_epassive#gt;
    

<state_logged_in> ::= <logged_in_cmds> <state_logged_in>
 | <exchange_set_type> <state_in_binary>
 | <exchange_set_epassive> <state_in_passive>

<logged_in_cmds> ::= (
    <exchange_pwd> | 
    <exchange_syst> | 
    <exchange_feat> | 
    <exchange_set_utf8>)

The PWD command

PWD requests the current working directory. The server answers with a (random) path.

<exchange_pwd> ::= (
  <ClientControl:ServerControl:request_pwd>
  <ServerControl:ClientControl:response_pwd>
)
<request_pwd> ::= 'PWD\r\n'
<response_pwd> ::= '257 \"' <directory> '\" is the current directory\r\n'
<directory> ::= '/' | ('/' <filesystem_name>)+
<filesystem_name> ::= r'[a-zA-Z0-9_]+'
<client_name> ::= r'[a-zA-Z0-9]+'

The SYST command

With the SYST command, we can request a 215 reply, followed by the server system name. We use Linux as default.

<exchange_syst> ::= (
  <ClientControl:ServerControl:request_syst>
  <ServerControl:ClientControl:response_syst>
)
<request_syst> ::= 'SYST\r\n'
<response_syst> ::= '215 ' <syst_name> '\r\n'
<syst_name> ::= r'[\x20-\x7E]+' := 'Linux'

The FEAT command

The FEAT command returns a list of features that the server supports. If Fandango generates the feature list, we return a fixed value - in our model, we only support the EPSV command. When receiving a feature list, we parse it according to the provided regex.

<exchange_feat> ::= (
  <ClientControl:ServerControl:request_feat>
  <ServerControl:ClientControl:response_feat>
)
<request_feat> ::= 'FEAT\r\n'
<response_feat> ::= '211-Features:\r\n' <feat_entry>+ '211 End\r\n' := feat_response()
<feat_entry> ::= ' ' r'[\x20-\x7E]+' '\r\n'

def feat_response():
    features = '211-Features:\r\n EPSV\r\n211 End\r\n'
    return features

The OPTS UTF8 command

We can send a command to set the character set to UTF-8, expecting a 200 (okay) response.

<exchange_set_utf8> ::= (
   <ClientControl:ServerControl:request_set_utf8>
   <ServerControl:ClientControl:response_set_utf8>
)
<request_set_utf8> ::= 'OPTS UTF8 ON\r\n'
<response_set_utf8> ::= '200 ' <command_tail> '\r\n'

Changing FTP States

Let us now explore more states. In our model, the FTP server can be in four states:

1. <state_logged_in> - the default state

2. <state_in_binary> - binary mode

3. <state_in_passive> - passive mode

4. <state_in_binary_passive> - binary and passive mode

Binary and passive modes are activated via <exchange_set_type> and <exchange_set_epassive> interactions, as shown below. We want to be in binary and passive mode, so we can actually retrieve data using LIST (<exchange_list>) and finally quit (exchange_quit).

        stateDiagram
    [*] --> #lt;state_logged_out_1#gt;
    #lt;state_logged_out_1#gt; --> #lt;state_logged_in#gt;: #lt;exchange_login_ok#gt;
    #lt;state_logged_in#gt; --> #lt;state_logged_in#gt;: #lt;logged_in_cmds#gt;
    #lt;state_logged_in#gt; --> #lt;state_in_binary#gt;: #lt;exchange_set_type#gt;
    #lt;state_logged_in#gt; --> #lt;state_in_passive#gt;: #lt;exchange_set_epassive#gt;
    #lt;state_in_passive#gt; --> #lt;state_in_passive#gt;: #lt;logged_in_cmds#gt;
    #lt;state_in_passive#gt; --> #lt;state_in_binary_passive#gt;: #lt;exchange_set_type#gt;
    #lt;state_in_binary_passive#gt; --> #lt;state_in_binary_passive#gt;: #lt;logged_in_cmds#gt;
    #lt;state_in_binary_passive#gt; --> #lt;state_in_binary#gt;: #lt;exchange_list#gt;
    #lt;state_in_binary_passive#gt; --> #lt;state_finished#gt;: #lt;exchange_quit#gt;
    #lt;state_in_binary#gt; --> #lt;state_in_binary#gt;: #lt;logged_in_cmds#gt;
    #lt;state_in_binary#gt; --> #lt;state_in_binary_passive#gt;: #lt;exchange_set_epassive#gt;
    #lt;state_finished#gt; --> [*]
    

<state_in_binary> ::= (
  <logged_in_cmds> <state_in_binary> |
  <exchange_set_epassive> <state_in_binary_passive>
)
<state_in_passive> ::= (
  <logged_in_cmds> <state_in_passive> |
  <exchange_set_type> <state_in_binary_passive>
)
<state_in_binary_passive> ::= (
  <logged_in_cmds> <state_in_binary_passive> |
  <exchange_list> <state_in_binary> |
  <exchange_quit> <state_finished>
)

The EPSV command - entering passive mode

By default, FTP uses “active” mode, in which the FTP server accesses a port on the client machine. Since most firewalls block such behavior, “passive” mode is now the typical use.

The EPSV command directs the server to open a port for data transmission. With this, we prepare client and server for actual data transfer. The server returns the port number by which it can be accessed; we have to get and process it.

<exchange_set_epassive> ::= \
  <ClientControl:ServerControl:request_set_epassive> \
  <ServerControl:ClientControl:response_set_epassive>

<request_set_epassive> ::= 'EPSV\r\n'
<response_set_epassive> ::= '229 Entering Extended Passive Mode (|||' <open_port> '|)\r\n'

When producing or parsing <open_port>, we call the open_data_port() generator function to reconfigure the data parties. This method gets called both when parsing and producing:

• When producing, Fandango produces a parameter <open_port_param>. <open_port_param> consists of <passive_port> and another generator that depends on <open_port>. This generator does not get executed when generating parameters for the generator from <open_port>, such that we do not get caught in an infinite loop between those to generators. Instead, we generate <passive_port> directly without invoking the generator.

• When parsing <open_port>, Fandango derives the argument <open_port_param> used in the generator by executing the generator from <open_port_param> which depends on <open_port>

<open_port> ::= <passive_port> := open_data_port(int(<open_port_param>))
<open_port_param> ::= <passive_port> := open_data_port(int(<open_port>))

When generating a passive port, we use a generator to randomly generate a port in [50100, 50100]

<passive_port> ::= r'[1-9][0-9]{0,4}' := randint(50100, 50100)

The function open_data_port(port) is a generator. When executed, it returns the value that was given to it and reconfigures the data party definitions to use that port. Again, we use the instance() method to access and reconfigure the individual parties.

As we may use the spec file to run with fandango fuzz as server or client, we protect against the fact that the ClientData or ServerData instances may not have been created.

def open_data_port(port):
    try:
        client_data = ClientData.instance()
        server_data = ServerData.instance()
    except KeyError:
        # Party instances not created
        return port

    if client_data.port != port:
        client_data.stop()
        client_data.port = port
    if  server_data.port != port:
        server_data.stop()
        server_data.port = port
    client_data.start()
    server_data.start()
    return port

The TYPE command - set binary mode

Using the FTP TYPE command, we can set the server into binary mode.

<exchange_set_type> ::= (
  <ClientControl:ServerControl:request_set_type>
  <ServerControl:ClientControl:response_set_type>
)
<request_set_type> ::= 'TYPE I\r\n'
<response_set_type> ::= '200 ' <command_tail> '\r\n'

The LIST command

Finally, after all these preparations, we can actually retrieve data via FTP. The FTP LIST command makes the FTP server send the contents of the current directory.

<exchange_list> ::= (
  <ClientControl:ServerControl:request_list>
  <ServerControl:ClientControl:open_list>
  <list_transfer>
)
<request_list> ::= 'LIST\r\n'
<open_list> ::= '150 ' <command_tail> '\r\n'

<list_data> gets sent using the data channel. Therefore, we use ServerData and ClientData as sending and receiving parties.

<list_transfer> ::= <ServerData:ClientData:list_data>?<ServerControl:ClientControl:finalize_list>
<finalize_list> ::= '226 ' <command_tail> '\r\n'

The list data itself contains file names, user names, permissions, and dates:

<list_data> ::= (<list_data_file>)+
<list_data_file> ::= <permissions> ' '+ <link_count> ' ' <user> ' '+ <group> ' '+ <file_size> ' ' <date> ' ' <filename> '\r\n'
<filename>    ::= r'[\x20-\x7E]+'
<number>      ::= r'[0-9]+' := str(randint(1, 1000))
<file_size>   ::= <number> := str(randint(0, 9999999))
<link_count>  ::= <number>

<permissions> ::= <file_type> <perm> <perm> <perm>
<file_type>   ::= r'[-dlcb]'
<perm>        ::= r'[r-]' r'[w-]' r'[x-]'
<user>        ::= r'[0-9a-zA-Z_\-]+'
<group>       ::= r'[0-9a-zA-Z_\-]+'
<date>        ::= <month> ' ' <day> ' ' <time>
<month>       ::= r'(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)'
<day>         ::= r'[0-9]{2}' := "{:02d}".format(randint(1, 28))
<time>        ::= <hour> ':' <minute>
<hour>        ::= r'[0-9]{2}' := "{:02d}".format(randint(0, 23))
<minute>      ::= r'[0-9]{2}' := "{:02d}".format(randint(0, 59))

This is what a typical generated entry looks like:

$ fandango fuzz -f ftp_client.fan --start-symbol '<list_data>' --party ServerData -n 1

-r-xrw-r-x      654 yWnjXekEDTDQozk90iNW         hq              2295528 Sep 21 17:49 )grx_D~w2D!VB[4NE5rV

l---r---w-                  863 erc4         Uil           9162501 Oct 13 12:40  J!zA6rTj>rZ[GmB8

The QUIT command

After a number of LIST commands, it is time to quit. We use the FTP QUIT command for this purpose.

<exchange_quit> ::= (
  <ClientControl:ServerControl:request_quit>
  <ServerControl:ClientControl:response_quit>
)
<request_quit> ::= 'QUIT\r\n'
<response_quit> ::= '221 ' <command_tail> '\r\n'

After that, the FTP server enters <state_finished>. There is no other interaction or state following, so we’re done.

<state_finished> ::= ''

Example Interactions

We can use fandango fuzz in conjunction with the --party option to simulate the messages produced by a single party.

A Typical Client Interaction

Here is a valid sequence of commands as issued by a client:

$ fandango fuzz -f ftp_client.fan --party ClientControl -n 1

AUTH SSL
AUTH TLS
AUTH SSL
AUTH SSL
USER the_user
PASS the_password
TYPE I
EPSV
LIST
OPTS UTF8 ON
OPTS UTF8 ON
FEAT
EPSV
QUIT

A Typical Server Interaction

Here is a valid sequence of responses as issued by a server:

$ fandango fuzz -f ftp_client.fan --party ServerControl -n 1

220-FP!;BY%`u,@'i{R
220-Job~?(T1{]%C0
220- o0Y#"?aPsmX
220-=DcAmomGg}0]69!xdZ 
220-)pS
220-w8!.*+
220 '
500 1(o
500 dC;eB_~xxZ5I(D&W"8~G
331 @4W*`-'
230 m@KQH@WLF@|C
229 Entering Extended Passive Mode (|||50100|)
200 "8>nBd
200 7o~Z>^;Hgoq{f
211-Features:
 EPSV
211 End
211-Features:
 EPSV
211 End
150 ;weAiZ.wB@*2=E.~V
226 V.
257 "/TBLG/nrGjtHYQtWawt1/UWktEdeB8Z/fh2hn0RK94j13M" is the current directory
215 Linux
229 Entering Extended Passive Mode (|||50100|)
200 1y6!~6'N9;"q
150 UK'EQXu^mvmW
226 LS
229 Entering Extended Passive Mode (|||50100|)
200 Db*E<o|sEKs{[
221 p>YkyPe*(5j_*k<>

Now go and try things out for yourself!