Accessing Input Elements

When dealing with complex input formats, attaching constraints can be complex, as elements can occur multiple times in a generated input. Fandango offers a few mechanisms to disambiguate these, and to specify specific contexts in which to search for elements.

Derivation Trees

So far, we have always assumed that Fandango generates strings from the grammar. Behind the scenes, though, Fandango creates a far richer data structure - a so-called derivation tree that maintains the structure of the string and allows accessing individual elements. Every time Fandango sees a grammar rule

<symbol> ::= ...

it generates a derivation tree whose root is <symbol> and whose children are the elements of the right-hand side of the rule.

Let’s have a look at our persons.fan spec:

<start> ::= <person_name> "," <age>
<person_name> ::= <first_name> " " <last_name>
<first_name> ::= <name>
<last_name> ::= <name>
<name> ::= <ascii_uppercase_letter><ascii_lowercase_letter>+
<age> ::= <digit>+

The <start> rule says

<start> ::= <person_name> "," <age>

Then, a resulting derivation tree for <start> looks like this:

As Fandango expands more and more symbols, it expands the derivation tree accordingly. Since the grammar definition for <person_name> says

<person_name> ::= <first_name> " " <last_name>

the above derivation tree would be extended to

And if we next extend <age> and then <digit> based on their definitions

<age> ::= <digit>+

If one has a Kleene operator like +, *, or ?, the elements operated on become children of the symbol being defined. Hence, the <digit> elements become children of the <age> symbol.

our tree gets to look like this:

Repeating the process, it thus obtains a tree like this:

Note how the tree records the entire history of how it was created - how it was derived, actually.

To obtain a string from the tree, we traverse its children left-to-right, ignoring all nonterminal symbols (in <...>) and considering only the terminal symbols (in quotes). This is what we get for the above tree:

Ex Pltz,18

And this is the string Fandango produces. However, viewing the Fandango results as derivation trees allows us to access elements of the Fandango-produced strings and to express constraints on them.

Diagnosing Derivation Trees

To examine the derivation trees that Fandango produces, use the --format=grammar output format. This produces the output in a grammar-like format, where children are indented under their respective parents. As an example, here is how to print a derivation tree from persons.fan:

$ fandango fuzz -f persons.fan -n 1 --format=grammar

<start> ::= <person_name> ',' <age>  # Position 0x0000 (0); Kibuvlxaulbcgmkjov Ktp,179870
  <person_name> ::= <first_name> ' ' <last_name>  # Position 0x0001 (1); Kibuvlxaulbcgmkjov Ktp
    <first_name> ::= <name>
      <name> ::= <ascii_uppercase_letter> <ascii_lowercase_letter> <ascii_lowercase_letter> <ascii_lowercase_letter> <ascii_lowercase_letter> <ascii_lowercase_letter> <ascii_lowercase_letter> <ascii_lowercase_letter> <ascii_lowercase_letter> <ascii_lowercase_letter> <ascii_lowercase_letter> <ascii_lowercase_letter> <ascii_lowercase_letter> <ascii_lowercase_letter> <ascii_lowercase_letter> <ascii_lowercase_letter> <ascii_lowercase_letter> <ascii_lowercase_letter>  # Kibuvlxaulbcgmkjov
        <ascii_uppercase_letter> ::= <_ascii_uppercase_letter>
          <_ascii_uppercase_letter> ::= 'K'  # Position 0x0002 (2)
        <ascii_lowercase_letter> ::= <_ascii_lowercase_letter>
          <_ascii_lowercase_letter> ::= 'i'  # Position 0x0003 (3)
        <ascii_lowercase_letter> ::= <_ascii_lowercase_letter>
          <_ascii_lowercase_letter> ::= 'b'  # Position 0x0004 (4)
        <ascii_lowercase_letter> ::= <_ascii_lowercase_letter>
          <_ascii_lowercase_letter> ::= 'u'  # Position 0x0005 (5)
        <ascii_lowercase_letter> ::= <_ascii_lowercase_letter>
          <_ascii_lowercase_letter> ::= 'v'  # Position 0x0006 (6)
        <ascii_lowercase_letter> ::= <_ascii_lowercase_letter>
          <_ascii_lowercase_letter> ::= 'l'  # Position 0x0007 (7)
        <ascii_lowercase_letter> ::= <_ascii_lowercase_letter>
          <_ascii_lowercase_letter> ::= 'x'  # Position 0x0008 (8)
        <ascii_lowercase_letter> ::= <_ascii_lowercase_letter>
          <_ascii_lowercase_letter> ::= 'a'  # Position 0x0009 (9)
        <ascii_lowercase_letter> ::= <_ascii_lowercase_letter>
          <_ascii_lowercase_letter> ::= 'u'  # Position 0x000a (10)
        <ascii_lowercase_letter> ::= <_ascii_lowercase_letter>
          <_ascii_lowercase_letter> ::= 'l'  # Position 0x000b (11)
        <ascii_lowercase_letter> ::= <_ascii_lowercase_letter>
          <_ascii_lowercase_letter> ::= 'b'  # Position 0x000c (12)
        <ascii_lowercase_letter> ::= <_ascii_lowercase_letter>
          <_ascii_lowercase_letter> ::= 'c'  # Position 0x000d (13)
        <ascii_lowercase_letter> ::= <_ascii_lowercase_letter>
          <_ascii_lowercase_letter> ::= 'g'  # Position 0x000e (14)
        <ascii_lowercase_letter> ::= <_ascii_lowercase_letter>
          <_ascii_lowercase_letter> ::= 'm'  # Position 0x000f (15)
        <ascii_lowercase_letter> ::= <_ascii_lowercase_letter>
          <_ascii_lowercase_letter> ::= 'k'  # Position 0x0010 (16)
        <ascii_lowercase_letter> ::= <_ascii_lowercase_letter>
          <_ascii_lowercase_letter> ::= 'j'  # Position 0x0011 (17)
        <ascii_lowercase_letter> ::= <_ascii_lowercase_letter>
          <_ascii_lowercase_letter> ::= 'o'  # Position 0x0012 (18)
        <ascii_lowercase_letter> ::= <_ascii_lowercase_letter>
          <_ascii_lowercase_letter> ::= 'v'  # Position 0x0013 (19)
    <last_name> ::= <name>
      <name> ::= <ascii_uppercase_letter> <ascii_lowercase_letter> <ascii_lowercase_letter>  # Ktp
        <ascii_uppercase_letter> ::= <_ascii_uppercase_letter>
          <_ascii_uppercase_letter> ::= 'K'  # Position 0x0014 (20)
        <ascii_lowercase_letter> ::= <_ascii_lowercase_letter>
          <_ascii_lowercase_letter> ::= 't'  # Position 0x0015 (21)
        <ascii_lowercase_letter> ::= <_ascii_lowercase_letter>
          <_ascii_lowercase_letter> ::= 'p'  # Position 0x0016 (22)
  <age> ::= <digit> <digit> <digit> <digit> <digit> <digit>  # 179870
    <digit> ::= <_digit>
      <_digit> ::= '1'  # Position 0x0017 (23)
    <digit> ::= <_digit>
      <_digit> ::= '7'  # Position 0x0018 (24)
    <digit> ::= <_digit>
      <_digit> ::= '9'  # Position 0x0019 (25)
    <digit> ::= <_digit>
      <_digit> ::= '8'  # Position 0x001a (26)
    <digit> ::= <_digit>
      <_digit> ::= '7'  # Position 0x001b (27)
    <digit> ::= <_digit>
      <_digit> ::= '0'  # Position 0x001c (28)

We see how the produced derivation tree consists of a <start> symbol, whose <first_name> and <last_name> children expand into <name> and letters; the <age> symbol expands into <digit> symbols.

The comments (after #) show the individual positions into the input, as well as the values of compound symbols.

What is the full string represented by the above derivation tree?

Solution

You can find on the right-hand side of the first line.

Tip

The --format=grammar option is great for debugging, especially binary formats.

Specifying Paths

One effect of Fandango producing derivation trees rather than “just” strings is that we can define special operators that allow us to access subtrees (or sub-elements) of the produced strings - and express constraints on them. This is especially useful if we want constraints to apply only in specific contexts - say, as part of some element <a>, but not as part of an element <b>.

Accessing Children

These selectors are similar to XPath, but better aligned with Python. In XPath, the first child has the index 1, in Fandango, it has the index 0.

The expression <foo>[N] accesses the N-th child of <foo>, starting with zero.

If <foo> is defined in the grammar as

<foo> ::= <bar> ":" <baz>

then <foo>[0] returns the <bar> element, <foo>[1] returns ":", and <foo>[2] returns the <baz> element.

In our persons.fan derivation tree for Ex Pltz, for instance, <start>[0] would return the <person_name> element ("Ex Pltz"), and <start>[2] would return the <age> element (18).

We can use this to access elements in specific contexts. For instance, if we want to refer to a <name> element, but only if it is the child of a <first_name> element, we can refer to it as <first_name>[0] - the first child of a <first_name> element:

<first_name> ::= <name>

Here is a constraint that makes Fandango produce first names that end with x:

Since a <first_name> is defined to be a <name>, we could also write <first_name>.endswith("x")

$ fandango fuzz -f persons.fan -n 10 -c '<first_name>[0].endswith("x")'

Mhytix Irmvwazvurakhkdh,0012905

Coelzscejfmrcakx Iuwxmrkhbdsyicgmzma,15212109

Ynpgteijocmvkx Lya,231574747568199

Yarpndhwx Ymucoprgk,642980580030948723

Mhytix Uwdvzskjwunqxk,94817298627

Yarpndhwx Lya,642980580030948723
Ynpgteijocmvkx Ymucoprgk,231574747568199

Mhytix Irmvwazmurakhkdh,0012905
Mhytix Coelzscejfmrcakx,0012905
Mhytix Irmvwazvurakhkdh,0012605

Tip

As in Python, you can use negative indexes to refer to the last elements. <age>[-1], for instance, gives you the last child of an <age> subtree.

Important

While symbols act as strings in many contexts, this is where they differ. To access the first character of a symbol <foo>, you need to explicitly convert it to a string first, as in str(<foo>)[0].

Slices

The Fandango slices maintain the property that <name>[n:m] = <name>[n] + <name>[n + 1] + … + <name>[m - 1], which holds for all sequences in Python.

Fandango also allows you to use Python slice syntax to access multiple children at once. <name>[n:m] returns a new (unnamed) root which has <name>[n], <name>[n + 1], …, <name>[m - 1] as children. This is useful, for instance, if you want to compare several children against a string:

$ fandango fuzz -f persons-faker.fan -n 10 -c '<name>[0:2] == "Ch"'

Chad Chase,90055357670399

Christina Chase,14092432449

Chad Chase,90051357670399
Chad Chase,20754357677399

Chad Chase,95051357670399

Chad Chase,20054357677399
Chad Chase,90025357670399

Charles Chase,20754357677399
Chad Chase,50754357677399
Chad Chase,90052357670399

Would one also be able to use <start>[0:2] == "Ch" to obtain inputs that all start with "Ch"?

Solution

No, this would not work. Remember that in derivation trees, indexes refer to children, not characters. So, according to the rule

<start> ::= <person_name> "," <age>

<start>[0] is a <person_name>, <start>[1] is a ",", and <start>[2] is an <age>. Hence, <start>[0:2] refers to <start> itself, which cannot be "Ch".

Indeed, to have the string start with "Ch", you (again) need to convert <start> into a string first, and then access its individual characters:

$ fandango fuzz -f persons-faker.fan -n 10 -c 'str(<start>)[0:2] == "Ch"'

Christine Hodge,0594059

Charles Ward,94924945103238148899

Christopher Juarez,65769

Christian Vang,12

Christopher Juarez,61769
Christine Hodge,254815951

Christine Hodge,097502
Charles Ward,55205564
Charles Crane,768022
Christopher Garza,65769

The Fandango slices maintain the property that <name>[i:] + <name>[:i] = <name>

Fandango supports the full Python slice semantics:

• An omitted first index defaults to zero, so <foo>[:2] returns the first two children.

• An omitted second index defaults to the size of the string being sliced, so <foo>[2:] returns all children starting with <foo>[2].

• Both the first and the second index can be negative again.

Selecting Children

Referring to children by number, as in <foo>[0], can be a bit cumbersome. This is why in Fandango, you can also refer to elements by name.

In XPath, the corresponding operator is /.

The expression <foo>.<bar> allows accessing elements <bar> when they are a direct child of a symbol <foo>. This requires that <bar> occurs in the grammar rule defining <foo>:

<foo> ::= ...some expansion that has <bar>...

To refer to the <name> element as a direct child of a <first_name> element, you thus write <first_name>.<name>. This allows you to express the earlier constraint in a possibly more readable form:

$ fandango fuzz -f persons.fan -n 10 -c '<first_name>.<name>.endswith("x")'

Mysdx Pveyudqnoobkxerjsoyd,7124319824939

Trx Klcptisjifz,5628084022004518

Ghqtrmsyyx Jrzsvsictinbnixte,4311

Jupvjmxabbioilqjdgex Sgvn,18305715894

Vhrvqvdgemx Acgkdxtjkokyxwbbz,6

Trx Pveyudqnoobkxerjsoyd,5628084022004518
Mysdx Klcptisjifz,7124319824939
Trx Klcptisjifz,5628084022004588
Jupvjmxabbioilqjdgex Sgvn,18305715194
Jupvjmxabbioilqjdgex Rgvn,18305715894

Note

You can only access nonterminal children this way; <person_name>." " (the space in the <person_name>) gives an error.

Selecting Descendants

Often, you want to refer to elements in a particular context set by the enclosing element. This is why in Fandango, you can also refer to descendants.

In XPath, the corresponding operator is //.

The expression <foo>..<bar> allows accessing elements <bar> when they are a descendant of a symbol <foo>. <bar> is a descendant of <foo> if

<foo>..<bar> includes <foo>.<bar>.

• <bar> is a child of <foo>; or

• one of <foo>’s children has <bar> as a descendant.

If that sounds like a recursive definition, that is because it is. A simpler way to think about <foo>..<bar> may be “All <bar>s that occur within <foo>”.

Let us take a look at some rules in our persons.fan example:

<first_name> ::= <name>
<last_name> ::= <name>
<name> ::= <ascii_uppercase_letter><ascii_lowercase_letter>+
<ascii_uppercase_letter> ::= "A" | "B" | "C" | ... | "Z"

To refer to all <ascii_uppercase_letter> element as descendant of a <first_name> element, you thus write <first_name>..<ascii_uppercase_letter>.

Hence, to make all uppercase letters X, but only as part of a first name, you may write

$ fandango fuzz -f persons.fan -n 10 -c '<first_name>..<ascii_uppercase_letter> == "X"'

Xkaqfjivtmcjr Vvfizlitwqpzjoewdl,816079091

Xdjqrobekyhhycpsbfrcw Ydfzisvon,6020606935284652195

Xgkzykzsxclrtpaxbarc Ztqjfpvpv,0755254282
Xshjjnkjnt Dekoflclyroujh,913908660
Xn Kixvy,4436218090

Xkeejlbgmsiiukovqunxf Uhmcswicdgwejjmtjbnl,64041784482483
Xqwfwwfzsfyuqpfraszk Gedq,553131419480424514

Xvaxetvpmfmftmnp Zvveakpvwpudtuab,9
Xdi Wewunrrravl,4774
Xpe Mfxjfl,2

Chains

You can freely combine [], ., and .. into chains that select subtrees. What would the expression <start>[0].<last_name>..<ascii_lowercase_letter> refer to, for instance?

Solution

This is easy:

• <start>[0] is the first element of start, hence a <person_name>.

• .<last_name> refers to the child of type <last_name>

• ..<ascii_lowercase_letter> refers to all descendants of type <ascii_lowercase_letter>

Let’s use this in a constraint:

$ fandango fuzz -f persons.fan -n 10 -c '<start>[0].<last_name>..<ascii_lowercase_letter> == "x"'

Hocgvddcc Xxxxxxxxxxxxxxxxxxxx,55
Hxubcniay Exx,905
Wzzqgmdulj Pxxxxxxxxxxxx,268132399233199641

Bviusd Xxxxxxxxxxxxx,4213540321396195365
Lir Kxxxxx,594937815799

Pdbzehpucxpfqtri Kxxxxxxxx,1860625761
Zvj Exxxx,2470857474732
Tyrxk Gxx,9996996925

Etsdsis Oxxxxxxxxxx,68100
Gcemwo Cxxxxx,71881076222135954

Quantifiers

By default, whenever you use a symbol <foo> in a constraint, this constraint applies to all occurrences of <foo> in the produced output string. For your purposes, though, one such instance already may suffice. This is why Fandango allows expressing quantification in constraints.

Star Expressions

In Fandango, you can prefix an element with * to obtain a collection of all these elements within an individual string. Hence, *<name> is a collection of all <name> elements within the generated string. This syntax can be used in a variety of ways. For instance, we can have a constraint check whether a particular element is in the collection:

Not every constraint that can be expressed also can be solved by Fandango.

"Pablo" in *<name>

This constraint evaluates to True if any of the values in *<name> (= one of the two <name> elements) is equal to "Pablo". *-expressions are mostly useful in quantifiers, which we discuss below.

Existential Quantification

To express that within a particular scope, at least one instance of a symbol must satisfy a constraint, use a *-expression in combination with any():

any(CONSTRAINT for ELEM in *SCOPE)

where

• SCOPE is a nonterminal (e.g. <age>);

• ELEM is a Python variable; and

• CONSTRAINT is a constraint over ELEM

Hence, the expression

any(n.startswith("A") for n in *<name>)

ensures there is at least one element <name> that starts with an “A”:

Let us decompose this expression for a moment:

• The expression for n in *<name> lets Python iterate over *<name> (all <name> objects within a person)…

• … and evaluate n.startswith("A") for each of them, resulting in a collection of Boolean values.

• The Python function any(list) returns True if at least one element in list is True.

So, what we get is existential quantification:

$ fandango fuzz -f persons.fan -n 10 -c 'any(n.startswith("A") for n in *<name>)'

Az Yle,46711692945
Agnebr Zkijhevcvvtcmqg,0202

Abeslltksztrumjbk Unhcxgx,56439408190666457878

Omjuayjawavoj Afxpbtgovnu,5734

Ihewakrpog Arzfryhpfbzurnjxwk,475538343194

Ahjsjrztxkksxhes Mybkiiakfnrwz,5429882393654

Aezwpkcebx Tqn,6700179369394

Aahtpypsrx Pwskwfvmwcbnvbyt,2489483271804

Oslsmdjrstxstbxotydh Avkkgqsyikyyzvznq,0071

Almhwizrjc Qyiwdgdedxiqjzxqixzup,90404236

Universal Quantification

Where there are existential quantifiers, there also are universal quantifiers. Here we use the Python all() function; all(list) evaluates to True only if all elements in list are True.

We use a *-expression in combination with all():

all(CONSTRAINT for ELEM in *SCOPE)

Hence, the expression

all(c == "a" for c in *<first_name>..<ascii_lowercase_letter>)

ensures that all sub-elements <ascii_lowercase_letter> in <first_name> have the value “a”.

Again, let us decompose this expression:

• The expression for c in *<first_name>..<ascii_lowercase_letter> lets Python iterate over all <ascii_lowercase_letter> objects within <first_name>…

• … and evaluate c == "a" for each of them, resulting in a collection of Boolean values.

• The Python function all(list) returns True if all elements in list are True.

So, what we get is universal quantification:

$ fandango fuzz -f persons.fan -n 10 -c 'all(c == "a" for c in *<first_name>..<ascii_lowercase_letter>)'

Iaaaaaaaaaaa Laplhkdlixkbpxf,039458940359083
Saaaaaaaaaaaaaaa Aklhfsc,9823

Yaaaaaaaaaaaaaaaaaa Xvstdkatlohxsmva,740057
Jaaaaaa Kiqabrzcdjossdy,7566985425364

Daaaaaaaaaa Dsx,8124396581169338059
Faaaaaaaaaaaaaaaaaaaa Wr,26

Qaaaaaaaaaaaaaaaaaa Dugsmunsrsxcjrphw,922290544466438004
Maaa Rairiafpymsaoq,149408685475

Oaaaaaaaaaaaaaaa Mpzofsxgmnq,2075858107969396

Oaaaaaaaaaa Nbsfjbjniyzigok,29261722862

By default, all symbols are universally quantified within <start>, so a dedicated universal quantifier is only needed if you want to limit the scope to some sub-element. This is what we do here with <first_name>..<ascii_lowercase_letter>, limiting the scope to <first_name>.

Given the default universal quantification, you can actually achieve the same effect as above without all() and without a *. How?

Solution

You can access all <ascii_lowercase_letter> elements within <first_name> directly:

$ fandango fuzz -f persons.fan -n 10 -c '<first_name>..<ascii_lowercase_letter> == "a"'

Old-Style Quantifiers

Prior to version 1.0, Fandango supported another style of quantifiers:

• The expression forall SYMBOL in EXPRESSION: CONSTRAINT is equivalent to all(CONSTRAINT for ELEM in *EXPRESSION: )

• The expression exists SYMBOL in EXPRESSION: CONSTRAINT is equivalent to any(CONSTRAINT for ELEM in *EXPRESSION: )

with SYMBOL being a symbol (in <...>, so not a Python object) which can be referred to in CONSTRAINT.

To express that at least one letter in <first_name> should be 'a', write

exists <c> in <first_name>: str(<c>) == 'a'

$ fandango fuzz -f persons.fan -n 5 -c 'exists <c> in <ascii_lowercase_letter>: str(<c>) == "a"'

Ykqxdjfh Vxsvafvllizbfwtphva,1182352495286734
Yaaaaaaa Vxaaafvalaaafaaahva,1182352495286734

Arzv Qwmktfvpfptrapv,1834352538
Aaaa Qaaaaavafptrapv,1834352538
Iaaaaaaxaaai Iaaabaasaatakifzddha,63355368499994

$ fandango fuzz -f persons.fan -n 5 -c 'forall <c> in <ascii_lowercase_letter>: str(<c>) == "a"'

Yaaaaa Aaaa,1345
Laaaaaaaaaaaaaaa Saaaaaaaa,204209675

Baaaaa Iaaaaaaaaaa,193504347546845

Vaaaaaaaaaa Haaaaaa,7729
Iaaaaa Gaaaaaaaaaaa,2

Deprecated since version 1.0: Old-style quantifiers will be deprecated in a future Fandango version. Use all() and any() instead.