--- jupytext: formats: md:myst text_representation: extension: .md format_name: myst kernelspec: display_name: Python 3 language: python name: python3 --- (sec:ftp)= # Case Study: FTP This Fandango specification allows testing servers and clients for [File Transfer Protocol](https://en.wikipedia.org/wiki/File_Transfer_Protocol) (FTP, [RFC 959](https://datatracker.ietf.org/doc/html/rfc959)). It demonstrates * how a nontrivial protocol with text commands is implemented; and * how additional channels (`ClientData` and `ServerData`) are used on demand, based on the ports returned by the FTP server The FTP spec consists of three parts, all available for download: * [`ftp.fan`](ftp.fan) - the core FTP spec without specific party definitions * [`ftp_client.fan`](ftp_client.fan) - for using Fandango as an FTP client (includes `ftp.fan`) * [`ftp_server.fan`](ftp_server.fan) - for using Fandango as an FTP server (includes `ftp.fan`) To test it, run an FTP server on the local host at port 50100, and invoke Fandango as ```{margin} Omit `-n 1` if you want Fandango to test until protocol coverage is achieved. ``` ```shell $ fandango talk -n 1 -f ftp_client.fan ``` ```{versionadded} 1.1 These features are available in Fandango 1.1 and later. ``` (sec:ftp-parties)= ## The FTP Parties In contrast to other (simpler) protocols, FTP maintains _two_ communication channels: one for _control_ (issuing commands and getting responses), and one for _data_ (for transferring data). In our spec, we name these `ClientControl` and `ClientData` as well as `ServerControl` and `ServerData`. A very simple interaction involving all four, first logging in, and then sending a `LIST` command to get the contents of the current directory, is illustrated below. ```{mermaid} sequenceDiagram box Client participant ClientData participant ClientControl end box Server participant ServerControl participant ServerData end ClientControl ->> ServerControl: (connect) ServerControl ->> ClientControl: 220 FTP Server ready. ClientControl ->> ServerControl: USER the_user ServerControl ->> ClientControl: 331 Password required for the_user. ClientControl ->> ServerControl: PASS the_password ServerControl ->> ClientControl: 230 User the_user logged in. ClientControl ->> ServerControl: EPSV ServerControl ->> ClientControl: 229 Entering Extended Passive Mode (|||50100|) ClientData ->> ServerData: (connect) ClientControl ->> ServerControl: LIST ServerControl ->> ClientControl: 150 Opening ASCII mode data connection for file list ServerData ->> ClientData: (data) ServerControl ->> ClientControl: 226 Transfer complete. ServerData ->> ClientData: (disconnect) ``` ### Control Parties In our setting, we assume that Fandango is acting as _client_ to test an FTP server. Fandango connects to a server running on port 25521 on the local host. Whenever received a message using the `ClientControl` party, it marks the message as received from the `ServerControl` party. The call to `self.start()` in the constructior starts the party and lets it connect to the socket. ```python class ClientControl(NetworkParty): def __init__(self): super().__init__( connection_mode=ConnectionMode.CONNECT, uri="tcp://[::1]:25521" ) self.start() def receive(self, message: str | bytes | None, sender: Optional[str]) -> None: super().receive(message.decode("utf-8"), sender="ServerControl") ``` The `ServerControl` party is the counterpart to `ClientControl`. As in our scenario we are assuming that Fandango is acting as _client_ we set the `connection_mode` to `ConnectionMode.EXTERNAL`. The party does, therefore, not connect to a socket. The call to `self.start()` in the constructior is ignored. ```python class ServerControl(NetworkParty): def __init__(self): super().__init__( connection_mode=ConnectionMode.EXTERNAL, uri="tcp://[::1]:25522" ) self.start() def receive(self, message: str | bytes | None, sender: Optional[str]) -> None: super().receive(message.decode("utf-8"), sender="ClientControl") ``` ### Data Parties In our setting, the FTP data transfer takes place via port 50100 on the local host. In our setting, the FTP data port is updated at runtime, the port number specified in the `uri` is a placeholder. When starting a new protocol interaction, this party is not connected to a socket. We therefore don't call `self.start()` in the constructor. The updating procedure of the port number is performed in the [`open_data_port(port)-function`](sec:ftp-epsv). Whenever `ClientData`'s data socket is closed by the server, it received a `None` message. This automatically shuts down the `ClientData` party. `ClientData` forwards a `Data socket closed.` message to `SocketControlServer` which is a dummy party, that we use to describe port disconnections in the grammar. ```python class ClientData(NetworkParty): def __init__(self): super().__init__( connection_mode=ConnectionMode.CONNECT, uri="tcp://[::1]:50100" ) # Tell FANDANGO that all received messages come from ServerData. def receive(self, message: str | bytes | None, sender: Optional[str]) -> None: if message is None: super().receive("Data socket closed.\r\n", sender="SocketControlServer") return super().receive(message.decode("utf-8"), sender="ServerData") ``` ```python class ServerData(NetworkParty): def __init__(self): super().__init__( connection_mode=ConnectionMode.EXTERNAL, uri="tcp://[::1]:50100" ) def receive(self, message: str | bytes | None, sender: Optional[str]) -> None: if message is None: super().receive("Data socket closed.\r\n", sender="SocketControlClient") return super().receive(message.decode("utf-8"), sender="ClientData") ``` ### Dummy socket control parties `SocketControlClient` is a dummy party used to describe socket closures in the grammar. Whenever FANDANGO produces a message sent as `SocketControlClient`, the party implementation doesn't send the message. Instead, it treats the message as a command to stop the `ClientData` party. Here, we use the [`instance()`](sec:party-instance) method to access and stop the data party and therefore closing the data socket. ```python class SocketControlClient(FandangoParty): def __init__(self): super().__init__(connection_mode=ConnectionMode.CONNECT) def send(self, message: str | bytes, recipient: Optional[str]) -> None: if str(message).startswith("Data socket closed.\r\n"): ClientData.instance().stop() def start(self): pass def stop(self): pass ``` ```python class SocketControlServer(FandangoParty): def __init__(self): super().__init__(connection_mode=ConnectionMode.EXTERNAL) def send(self, message: str | bytes, recipient: Optional[str]) -> None: if str(message).startswith("Data socket closed.\r\n"): ServerData.instance().stop() def start(self): pass def stop(self): pass ``` (sec:ftp-connect)= ## Connecting to the FTP Server The interaction with an FTP server starts with the server `ServerControl` sending a 220 message `` to the client `ClientControl`, indicating that it is ready for a new user. Afterwards, we are in the state `state_logged_out_1`. ```python ::= ::= ::= ::= r'(220-(?:[\x20-\x7E]*\r\n))*220 (?:[\x20-\x7E]*)\r\n' ``` (sec:ftp-login)= ## Logging In While we're logged out (in state ``), we can * via username and password (``) Login with username and password can * fail (``, after which we still stay logged out); or * succeed (``), then, we are logged in (``). We only support logging in via username and password, so authentication via * TLS (``) or * SSL (``) are never successful, we remain logged out. ```{mermaid} stateDiagram [*] --> #lt;state_logged_out_1#gt; #lt;state_logged_out_1#gt; --> state_logged_out_1#gt;: #lt;exchange_auth_tls#gt; | #lt;exchange_auth_ssl#gt; | #lt;exchange_login_fail#gt; #lt;state_logged_out_1#gt; --> #lt;state_logged_in#gt;: #lt;exchange_login_ok ``` In our spec, this is modeled as exchanges followed by the resulting state. ```python ::= ( | | | ) ``` ### Logging in with username and password Our FTP server assumes one user with username `the_user` and a password `the_password`. ```{mermaid} sequenceDiagram ClientControl ->> ServerControl: (connect) ServerControl ->> ClientControl: 220 FTP Server ready. ClientControl ->> ServerControl: USER the_user ServerControl ->> ClientControl: 331 Password required for the_user. ClientControl ->> ServerControl: PASS the_password ServerControl ->> ClientControl: 230 User the_user logged in. ``` ```python ::= ( ) ``` ```python ::= 'USER the_user\r\n' ::= '331 ' '\r\n' ::= 'PASS the_password\r\n' ::= '230 ' '\r\n' ``` ### Failure to log in There are two ways logging in can go wrong - an incorrect password (not `the_password`): ``` ::= r'^(?!the_password$)([a-zA-Z0-9_]+)' ``` and an incorrect username (not `the_user`): ```python ::= r'^(?!the_user$)([a-zA-Z0-9_]+)' ``` Let's discuss these two options: ```python ::= | ``` First, we can have the client send a correct username, but a wrong password. ```python ::= ( ) ``` ```python ::= 'USER the_user\r\n' ::= 'PASS ' '\r\n' ::= '530 ' '\r\n' ::= r'[\x20-\x7E]+' ``` Second, we can have the client send an incorrect username (with a correct or incorrect password). ```python ::= ( ( | ) ) ``` ```python ::= 'USER ' '\r\n' ``` In both cases, we end up staying logged out (``). ### (Not) logging in via TLS and SSL We do not support logging in via TLS and SSL and accept a 500 (syntax error) or 530 (not logged in) message from the server. ```python ::= ::= 'AUTH TLS\r\n' ::= r'(530|500)' ' ' '\r\n' ``` ```python ::= ::= 'AUTH SSL\r\n' ::= r'(530|500)' ' ' '\r\n' ``` ## First FTP Commands When the client is logged in, it can send commands to the server. In the `` state, we support the commands listed in ``. (The [``](sec:ftp-type) and [``](sec:ftp-epsv) commands change the FTP state; see [States](sec:ftp-states) below.) ```{mermaid} stateDiagram [*] --> #lt;state_logged_out_1#gt; #lt;state_logged_out_1#gt; --> #lt;state_logged_in#gt;: #lt;exchange_login_ok#gt; #lt;state_logged_in#gt; --> #lt;state_logged_in#gt;: #lt;logged_in_cmds#gt; #lt;state_logged_in#gt; --> #lt;state_in_binary#gt;: #lt;exchange_set_type#gt; #lt;state_logged_in#gt; --> #lt;state_in_passive#gt;: #lt;exchange_set_epassive#gt; ``` ``` ::= | | ``` ```python ::= ( | | | ) ``` (sec:ftp-pwd)= ### The PWD command PWD requests the current working directory. The server answers with a (random) path. ```python ::= ( ) ::= 'PWD\r\n' ::= '257 \"' '\" is the current directory\r\n' ::= '/' | ('/' )+ ::= r'[a-zA-Z0-9_]+' ::= r'[a-zA-Z0-9]+' ``` (sec:ftp-syst)= ### The SYST command With the `SYST` command, we can request a `215` reply, followed by the server system name. We use `Linux` as default. ```python ::= ( ) ::= 'SYST\r\n' ::= '215 ' '\r\n' ::= r'[\x20-\x7E]+' := 'Linux' ``` (sec:ftp-feat)= ### The FEAT command The FEAT command returns a list of features that the server supports. If Fandango generates the feature list, we return a fixed value - in our model, we only support the [`EPSV`](sec:ftp-epsv) command. When receiving a feature list, we parse it according to the provided regex. ```python ::= ( ) ::= 'FEAT\r\n' ::= '211-Features:\r\n' + '211 End\r\n' := feat_response() ::= ' ' r'[\x20-\x7E]+' '\r\n' def feat_response(): features = '211-Features:\r\n EPSV\r\n211 End\r\n' return features ``` (sec:ftp-opts)= ### The OPTS UTF8 command We can send a command to set the character set to UTF-8, expecting a `200` (okay) response. ```python ::= ( ) ::= 'OPTS UTF8 ON\r\n' ::= '200 ' '\r\n' ``` (sec:ftp-states)= ## Changing FTP States Let us now explore more states. In our model, the FTP server can be in four states: 1. `` - the default state 2. `` - binary mode 3. `` - passive mode 4. `` - binary _and_ passive mode Binary and passive modes are activated via [``](sec:ftp-type) and [``](sec:ftp-epsv) interactions, as shown below. We want to be in binary _and_ passive mode, so we can actually retrieve data using `LIST` ([``](sec:ftp-list)) and finally quit ([`exchange_quit`](sec:ftp-quit)). ```{mermaid} stateDiagram [*] --> #lt;state_logged_out_1#gt; #lt;state_logged_out_1#gt; --> #lt;state_logged_in#gt;: #lt;exchange_login_ok#gt; #lt;state_logged_in#gt; --> #lt;state_logged_in#gt;: #lt;logged_in_cmds#gt; #lt;state_logged_in#gt; --> #lt;state_in_binary#gt;: #lt;exchange_set_type#gt; #lt;state_logged_in#gt; --> #lt;state_in_passive#gt;: #lt;exchange_set_epassive#gt; #lt;state_in_passive#gt; --> #lt;state_in_passive#gt;: #lt;logged_in_cmds#gt; #lt;state_in_passive#gt; --> #lt;state_in_binary_passive#gt;: #lt;exchange_set_type#gt; #lt;state_in_binary_passive#gt; --> #lt;state_in_binary_passive#gt;: #lt;logged_in_cmds#gt; #lt;state_in_binary_passive#gt; --> #lt;state_in_binary#gt;: #lt;exchange_list#gt; #lt;state_in_binary_passive#gt; --> #lt;state_finished#gt;: #lt;exchange_quit#gt; #lt;state_in_binary#gt; --> #lt;state_in_binary#gt;: #lt;logged_in_cmds#gt; #lt;state_in_binary#gt; --> #lt;state_in_binary_passive#gt;: #lt;exchange_set_epassive#gt; #lt;state_finished#gt; --> [*] ``` ```python ::= ( | ) ::= ( | ) ::= ( | | ) ``` (sec:ftp-epsv)= ### The EPSV command - entering passive mode ```{margin} By default, FTP uses "active" mode, in which the FTP server accesses a port on the client machine. Since most firewalls block such behavior, "passive" mode is now the typical use. ``` The `EPSV` command directs the server to open a port for data transmission. With this, we prepare client and server for actual data transfer. The server returns the port number by which it can be accessed; we have to get and process it. ```python ::= \ \ ::= 'EPSV\r\n' ::= '229 Entering Extended Passive Mode (|||' '|)\r\n' ``` When producing or parsing ``, we call the `open_data_port()` generator function to reconfigure the data parties. This method gets called both when parsing and producing: * When _producing_, Fandango produces a parameter ``. `` consists of `` and another generator that depends on ``. This generator does not get executed when generating parameters for the generator from ``, such that we do not get caught in an infinite loop between those to generators. Instead, we generate `` directly without invoking the generator. * When _parsing_ ``, Fandango derives the argument `` used in the generator by executing the generator from `` which depends on `` ```python ::= := open_data_port(int()) ::= := open_data_port(int()) ``` When generating a passive port, we use a generator to randomly generate a port in [50100, 50100] ``` ::= r'[1-9][0-9]{0,4}' := randint(50100, 50100) ``` The function `open_data_port(port)` is a generator. When executed, it returns the value that was given to it and reconfigures the data party definitions to use that port. Again, we use the [`instance()`](sec:party-instance) method to access and reconfigure the individual parties. ```{margin} As we may use the spec file to run with `fandango fuzz` as server or client, we protect against the fact that the `ClientData` or `ServerData` instances may not have been created. ``` ```python def open_data_port(port): try: client_data = ClientData.instance() server_data = ServerData.instance() except KeyError: # Party instances not created return port if client_data.port != port: client_data.stop() client_data.port = port if server_data.port != port: server_data.stop() server_data.port = port client_data.start() server_data.start() return port ``` (sec:ftp-type)= ### The TYPE command - set binary mode Using the FTP `TYPE` command, we can set the server into binary mode. ```python ::= ( ) ::= 'TYPE I\r\n' ::= '200 ' '\r\n' ``` (sec:ftp-list)= ### The LIST command Finally, after all these preparations, we can actually retrieve data via FTP. The FTP `LIST` command makes the FTP server send the contents of the current directory. ```python ::= ( ) ::= 'LIST\r\n' ::= '150 ' '\r\n' ``` `` gets sent using the data channel. Therefore, we use `ServerData` and `ClientData` as sending and receiving parties. `SocketControlServer` is used to specify situations where the data channel should be closed by the server. ```python ::= ?(|?) ::= '226 ' '\r\n' ::= 'Data socket closed.\r\n' ``` The list data itself contains file names, user names, permissions, and dates: ```python ::= ()+ ::= ' '+ ' ' ' '+ ' '+ ' ' ' ' '\r\n' ::= r'[\x20-\x7E]+' ::= r'[0-9]+' := str(randint(1, 1000)) ::= := str(randint(0, 9999999)) ::= ::= ::= r'[-dlcb]' ::= r'[r-]' r'[w-]' r'[x-]' ::= r'[0-9a-zA-Z_\-]+' ::= r'[0-9a-zA-Z_\-]+' ::= ' ' ' '