The Linker class has collapsed!

[Edward K. Ream]

Documentation often primes the mental pump. So it was yesterday. In this post I said:

QQQ

- The results list exists only to hold data to be transferred to the tokens list.

- Syncing the tokens and results lists is the only way make this transfer!

- ...significant (non-optional) tokens appear in exactly the same order in the two lists.

QQQ

Any time I assert that there is only one way of doing something my subconscious starts screaming at me ;-) Eventually, I realized that the notion of significant tokens could collapse the problem.  Just synchronize the two token lists on their significant tokens! Instantly everything becomes dead easy.

Here are the three main methods of the Linker class, shown in simplified form for clarity:


def assign_links(self, results, tokens, tree):
    # Create the lists of significant tokens and results.
    sig_tokens = list(filter(self.is_significant, tokens))
    sig_results = list(filter(self.is_significant, results))
    # Raise an exception if the two lists are not compatible.
    self.check(sig_results, sig_tokens)
    # Make two-way links between tokens and results.
    for r, t in zip(sig_results, sig_tokens):
        self.set_links(r, t, tokens)

tx = 0  # The index of the last patched token.

def set_links(self, r, t, tokens):
    # Patch all previous assignable tokens.
    while self.tx <= t.index:
        token = tokens[self.tx]
        if self.should_be_assigned(token, r.node):
            # Patch the token.
            token.node = r.node
            # Add the token to r.node.token_list.
            token_list = getattr(r.node, 'token_list', [])
            r.node.token_list = token_list + [token]
        self.tx += 1

def check(self, results, tokens):
    n1, n2 = len(results), len(tokens)
    assert n1 == n2, (n1, n2)
    for r, t in zip(results, tokens):
        assert r.kind == t.kind, (repr(r), repr(t))
        assert self.compare_values(r, t), (repr(r), repr(t))


Subclasses may override the remaining three helper methods, shown here w/o docstrings:

def is_significant(self, token):
    return (
        token.kind in ('name', 'number') or
        token.kind == 'op' and token.value != ';')

def compare_values(self, r, t):
    if t.kind == 'string':
        val = True # to do. Must be more lenient.
    else:
        val = t.value == r.value
    return val

def should_be_assigned(self, token, node):
    return token.kind not in ('encoding', 'endmarker', 'ws')

That's all!

Summary

The linker is now the simplest thing that could possibly work.

It's also more flexible.  Subclasses can define the three helper methods as they like.

This is the way it is written in The Book.

Edward