--- jupytext: formats: md:myst text_representation: extension: .md format_name: myst kernelspec: display_name: Python 3 language: python name: python3 --- (sec:iso8601)= # Case Study: ISO 8601 Date + Time Let us make use of what we have explored so far in a more elaborate case study. [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) is an international standard for exchange and communication of _date and time-related data_, providing an unambiguous method of representing calendar dates and times. In this chapter, we will define a full Fandango spec for ISO 8601 date and time formats, ensuring both syntactic and semantic validity. To make matters more interesting, we will create the spec _programmatically_ - that is, by having a number of Python functions that generate the spec for us. ## Creating Grammars Programmatically We start with a number of functions that help us create fragments of the grammar. `make_rule()` creates a grammar rule from `symbol` and its possible `expansions`: ```{code-cell} import sys def print_s(s: str) -> str: print(s, end="") return s def make_rule(symbol: str, expansions: list[str], sep: str = '') -> str: return print_s(f"{sep}<{symbol}> ::= " + " | ".join(expansions) + "\n") make_rule("start", [""]); # a final ";" suppresses result ``` We can also use `make_rule()` to quickly create a list of expansions: ```{code-cell} make_rule("digit", [f"'{digit}'" for digit in range(0, 10)]); ``` These additional functions help in adding non-grammar elements such as headers, code, and constraints: ```{code-cell} def make_header(title: str) -> str: return print_s(f"\n# {title}\n") def make_comment(comment: str, sep: str = '') -> str: return print_s(f"{sep}# {comment}\n") def make_constraint(constraint: str) -> str: return print_s(f"where {constraint}\n") def make_code(code: str) -> str: return print_s(f"{code}\n") make_header("ISO 8601 grammar"); ``` ## Spec Header Let us create a Fandango spec. For now, we store the final spec in the `iso8601lib` string variable, to be saved in a file at the end. Every time we add a new fragment, the new fragment will be output here, so we can see the actual content of `iso8601lib` incrementally. ```{code-cell} iso8601lib = make_header("ISO 8601 date and time") iso8601lib += make_comment("Generated by docs/ISO8601.md. Do not edit.") ``` We will need the `datetime` library below, so we import it. ```{code-cell} iso8601lib += make_code("\nimport datetime\n") ``` ## Date We start with a `` symbol. An ISO 8601 date/time spec starts with a date, and an optional time, separated by `T`: ```{code-cell} iso8601lib += make_rule("start", [""]) iso8601lib += make_rule("iso8601datetime", [" ('T' )?"]) ``` A date can either be a _calendar date_, but also a _week date_ or an _ordinal date_. ```{code-cell} iso8601lib += make_rule("iso8601date", ["", "", ""], sep='\n') ``` ### Calendar Dates An ISO 8601 calendar date has the format YYYY-MM-DD, but can also be YYYY-MM or YYYYMMDD. The year can be prefixed with `+` or `-` to indicate [AC/BC](https://en.wikipedia.org/wiki/Anno_Domini) (or [CE/BCE](https://en.wikipedia.org/wiki/Common_Era)) designators. ```{code-cell} iso8601lib += make_rule("iso8601calendardate", [" '-' ('-' )?", " "], sep='\n') iso8601lib += make_rule("iso8601year", ["('+'|'-')? {4}"]) ``` % And yes, we need digits for specifying a year: % ```{code-cell} % iso8601lib += make_rule("digit", [f"'{digit}'" for digit in range(0, 10)]) % ``` ### Months Months are easy: ```{code-cell} iso8601lib += make_rule("iso8601month", [f"'{month:02d}'" for month in range(1, 13)]) ``` ### Days As with months, we define each day individually to avoid biasing the grammar. The fact that some months have only 28, 29, or 30 days will be handled later on in a constraint. ```{code-cell} iso8601lib += make_rule("iso8601day", [f"'{day:02d}'" for day in range(1, 32)]) ``` ```{tip} For testing purposes, it makes sense to test with _extreme_ values - in our case, January 1, February 28 or 29, and December 31. ``` ### Week Dates ISO 8601 also allows specifying dates by week numbers (1 - 53) and optional days of the week (1 - 7). `2025-W12-1` is the Monday of the 12th week in 2025. ```{code-cell} iso8601lib += make_rule("iso8601weekdate", [" '-'? 'W' ('-' )?"], sep='\n') iso8601lib += make_rule("iso8601week", [f"'{week:02d}'" for week in range(1, 54)]) iso8601lib += make_rule("iso8601weekday", [f"'{weekday:1d}'" for weekday in range(1, 8)]) ``` ### Ordinal Dates ISO 8601 also allows specifying dates by day numbers (1 - 366). `2000-366`, for instance, specifies the last day in 2000 (which was a leap year). Again, we specify each day programmatically (and individually). ```{code-cell} iso8601lib += make_rule("iso8601ordinaldate", [" ('-'? )?"], sep='\n') iso8601lib += make_rule("iso8601ordinalday", [f"'{day:03d}'" for day in range(1, 367)]) ``` ```{tip} For testing purposes, it makes sense to test with _extreme_ values - in our case, `1`, `365` and `366`. ``` ## Time Now for _time_ specifications. In ISO 8601, time is specified as `HH:MM:SS`, where `HH` comes in 24-hour format. `MM` and `SS` are optional, as is the colon separator in an "unambiguous" context. ```{code-cell} iso8601lib += make_rule("iso8601time", ["'T'? (':'? (':'? (('.' | ',') )? )? )? ?"], sep='\n') ``` ### Hours Hours normally go from `00` to `23`, but `24:00:00` is allowed to represent midnight at the end of a day. ```{code-cell} iso8601lib += make_comment("24:00:00 is allowed to represent midnight at the end of a day", sep='\n') iso8601lib += make_rule("iso8601hour", [f"'{hour:02d}'" for hour in range(0, 25)]) ``` ### Minutes Minutes go from `00` to `59`. Nothing special here. ```{code-cell} iso8601lib += make_rule("iso8601minute", [f"'{minute:02d}'" for minute in range(0, 60)]) ``` ### Seconds Seconds normally go from `00` to `59`, but `60` can be used to represent leap seconds. ```{code-cell} iso8601lib += make_comment("xx:yy:60 is allowed to represent leap seconds", sep='\n') iso8601lib += make_rule("iso8601second", [f"'{second:02d}'" for second in range(0, 61)]) ``` Seconds can be followed by `,` or a `.` (see ``, above) and a fraction - an arbitrary number of digits. ```{code-cell} iso8601lib += make_rule("iso8601fraction", ["+"]) ``` ```{tip} Testing with _extreme_ values would mandate times such as `20:15:00,99999999999999999999` to test for possible buffer overflows. ``` ### Time Zones Any time can be followed by a time zone designator. This is either `Z`, indicating Coordinated Universal Time (UTC), or an offset designating the time zone. ```{code-cell} iso8601lib += make_rule("iso8601timezone", ["'Z'", "'+' (':'? )?", "'-' (':'? )?"], sep='\n') ``` ## Ensuring Validity So far, our dates and times are _syntactically_ valid, but not necessarily _semantically_. We can easily create invalid dates such as `2025-02-31` (February 31) or invalid times such as `24:10:00` (10 minutes past midnight - only `24:00:00` is allowed). We could now add a number of additional _rules_ to check for all these properties, and/or extend the grammar accordingly. However, we can also simply be lazy and have the existing Python `datetime` module check all this for us: ```{code-cell} import datetime ``` ```{code-cell} def is_valid_iso8601datetime(iso8601datetime: str) -> bool: """Return True iff `iso8601datetime` is valid.""" try: datetime.datetime.fromisoformat(iso8601datetime) return True except ValueError: return False ``` ```{code-cell} is_valid_iso8601datetime("2025-01-01T14:00") ``` ```{code-cell} is_valid_iso8601datetime("2025-02-31") ``` ```{code-cell} is_valid_iso8601datetime("2000-02-29T00:00:00") ``` With this, we can now add a _constraint_ that limits our generator to only valid dates and times: ```{code-cell} iso8601lib += make_constraint("is_valid_iso8601datetime(str())") ``` ```{code-cell} iso8601lib += ''' def is_valid_iso8601datetime(iso8601datetime: str) -> bool: """Return True iff `iso8601datetime` is valid.""" try: datetime.datetime.fromisoformat(iso8601datetime) return True except ValueError: return False ''' ``` ## Even Better Validity The Python `datetime` module has a some limitations, which extend to our spec. For instance, `datetime` does not support `24:00:00` as a valid time: ```{code-cell} is_valid_iso8601datetime("2024-12-31T24:00:00") ``` Hence, our `iso8601.fan` spec may miss out a number of ISO 8601 features. The [`dateutil` alternative](https://dateutil.readthedocs.io/en/stable/index.html) provides an ISO 8601 parser without these deficiencies. ```{code-cell} :tags: ["remove-input", "remove-output"] import dateutil ``` Let us redefine `is_valid_iso8601datetime()` to make use of the `dateutil` parser: ```{code-cell} def is_valid_iso8601datetime(iso8601datetime: str) -> bool: """Return True iff `iso8601datetime` is valid.""" try: dateutil.parser.isoparse(iso8601datetime) return True except ValueError: return False ``` Let us see if the above example now works: ```{code-cell} is_valid_iso8601datetime("2024-12-31T24:00:00") ``` This now works! How about the earlier examples? ```{code-cell} is_valid_iso8601datetime("2025-01-01T14:00") ``` ```{code-cell} is_valid_iso8601datetime("2025-02-31") ``` ```{code-cell} is_valid_iso8601datetime("2000-02-29T00:00:00") ``` These also work. Let us fix the spec to use `dateutil` instead: ```{code-cell} iso8601lib = iso8601lib.replace('datetime.datetime.fromisoformat', 'dateutil.parser.isoparse') iso8601lib = iso8601lib.replace('import datetime', 'import dateutil # See https://dateutil.readthedocs.io'); ``` ## Fuzzing Dates and Times Our ISO 8601 spec is now complete. Let us write it into a `.fan` file, so we can use it for fuzzing: ```{code-cell} open('ISO8601.fan', 'w').write(iso8601lib); ``` Here comes [`iso8601.fan`](iso8601.fan) in all its glory: ```{code-cell} :tags: ["remove-input"] !fold -s ISO8601.fan | ./fan_lexer.py ``` With this, we can now create a bunch of random date/time elements: ```shell $ fandango fuzz -f iso8601.fan -n 10 ``` ```{code-cell} :tags: ["remove-input"] !fandango fuzz -f ISO8601.fan -n 10 --validate --progress-bar=off assert _exit_code == 0 ``` And we can use additional constraints to further narrow down date intervals: ```shell $ fandango fuzz -f ISO8601.fan -n 10 -c 'int() > 1950 and int() < 2000' ``` ```{code-cell} :tags: ["remove-input"] !fandango fuzz -f ISO8601.fan -n 10 -c 'int() > 1950 and int() < 2000' --validate --progress-bar=off assert _exit_code == 0 ``` Or produce individual elements, again with individual constraints: ```shell $ fandango fuzz -f ISO8601.fan -n 10 --start-symbol='' -c ' == "00"' ``` ```{code-cell} :tags: ["remove-input"] !fandango fuzz -f iso8601.fan -n 10 --start-symbol='' -c ' == "00"' --progress-bar=off assert _exit_code == 0 ``` Try out more constraints for yourself! The generated [`ISO8601.fan`](ISO8601.fan) file is available for download.