I partially answered my own question, from what should have been obvious. The variable stack is passed to the lookup plugin.
self.set_options(var_options=variables, direct=kwargs)
x = variables['vars']['<variable_name>']
But I am still stuck with a problem. That returns the literal string value that was assigned to the variable, it does not evaluate that value.
For example, if I have the following in a vars file
x: 'a value'
y: "{{ x }}"
variables['vars']['y'] will return "{{ x }}", not the evaluated result "a value".
Is there some other utility function I need to call to force that evaluation?