Elixir and match_specs

[christ...@gmail.com]

Hey all!

Sitting apparently forever in my personal backlog of stuff to do is implement defguard.

Waaay back in the day I actually conceptualized this as living in a Guard module. Since then I've occasionally advocated adding trivial new modules to the core Elixir namespace, even if just for the sake of representing and furnishing core types with conversion functions (like Atom).

Anyhow, I've been implementing a toy library that demands *really really fast* ETS selection. So I've been poking around the ETS functions match and select.

Currently Elixir guards are macros that expand Elixir own features and abstractions to generate guard clauses compatible with Erlang. However, Erlang match_specs kind of introduce their own AST for building guards dynamically. You can use this advanced functionality from Elixir, but you have to be pretty comfortable with Erlang, and if you want to use an Elixir guard you're familiar with, you have to at least check its origins, if not know its direct implementation.

So I've been thinking: how might we harmonize with this Erlang concept of match specs?

We could introduce an intermediary representation of a guard as a Guard module with associated struct, and offer a to_match function. Possibly we could hook in another to_definition macro for defguard. Or skip the intermediate struct representation of a guard but offer more robust capabilities at the macro layer to slice and dice our version of guards in different ways, by adding clauses for all known guards and demanding new ones furnish that with an entry.

Alternatively/additionally a pretty cool notion would be to introduce a whole MatchSpec module with macros capable of converting something more Elixir-ish into a proper Erlang MatchSpec, guard input and charlist output and all. I'm thinking of Ecto's query DSL, but for matching.

This ties in nicely with Yurko's thought on supporting MatchSpecs within native Elixir features, and begs the question: where else might we adopt this or leverage our macros to provide even better tooling for this? Any thoughts on this concept?

[José Valim]

Hi Chris,

We have discussed match specs a couple times and the tricky part about them is that different parts of Erlang support a different subsets of match specs. For example, the tracing specs adds a bunch of tracing related functions, so the mechanism would need to be extensible.

Implementation wise, we thought about adding a function to the Macro module, which is where we keep AST related functions: Macro.to_matchspec(quoted, extra_functions :: [{name, arity}])

A MatchSpec module could be useful but, because every part of OTP uses a different subset, it may be limited in usage. I would rather see someone solving the tracing problem and then we could explore smaller functions, such as Macro.to_matchspec, that we could extract from tracing.

For some ideas on the topic, please also see: https://github.com/ericmj/ex2ms

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/831757cd-70ae-4386-a4fb-8c2fd9610129%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[christ...@gmail.com]

Hey José!

https://github.com/ericmj/ex2ms is pretty cool--very much what I had in mind!

Re: the tracing problem, as far as I'm aware there are only two match_spec contexts--ETS, and tracing (a superset of ETS). In my mind this could be obviated by an additional optional `context` parameter, supporting differentiation of the tracing context but defaulting to whatever the most common accepted representation of a match_spec is. That is, rather than

see someone solving the tracing problem and then we could explore smaller functions

I was thinking we might explore the smaller functions and extend support to additional options with a context argument, empty by default. This would allow creation of a MatchSpec module sooner that would probably delegate heavy lifting to Macro much as Macro.pipe does.

Learn You Some Erlang decides to introduce parse transforms just to address the pseudo-AST of MatchSpecs in ETS, so I feel as if this is an opportunity for our macros to shine. But it's a hard call to decide how such a convenience would work--quite beyond my experience with MatchSpecs to date, and that's before addressing the discrepancy between Elixir and Erlang guards in such a format. 

Thanks for filling me in on previous discussion!

Cheers,

Chris

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

[Christopher Keele]

Played with this a little during ElixirConf US 2019. Posting some findings here.

https://github.com/ericmj/ex2ms works well today, but I wanted to see if I could take some of my defguard experience and build an implementation that deferred all expansion work to the Elixir compiler, to avoid hard-coded translations like these elixir-to-erl mappings. I stole the actual AST to MS rewrite logic from ex2ms, but tried to leverage :elixir_expand.expand beforehand. I ran into a few challenges that have convinced me that first-class support for MS generation would require changes to the Elixir compiler and/or erlang to get the inclusion-into-core standards I want, mostly evidenced in my test project https://github.com/christhekeele/matcha via code comments or commented out tests (also stolen from ex2ms).

The test project's macros make three passes on the AST:

- expansion

- conversion

- validation

Expansion

The hard part of expanding Elixir code into matchspecs with the compiler is the "tracing problem", which can be best summarized as: trace matchspecs support function-like calls to drive the tracing harness, that don't actually have real-world compiled function analogs anywhere. This makes it difficult for a compiler to verify their correctness.

I started playing around with getting around this by defining :trace and :table modules with stub functions. The idea was to alias them into the AST before expansion, then strip the module from the call out afterwards. Not fully implemented in the test project yet. More on this in a bit.

The other stumbling block I ran into was expanding multiple clauses verbatim. I had to wrap the clauses provided to the macro in a fn block for the compiler to expand them, which prevents interpolating values anywhere into the matchspec via the ^ operator (the way ex2ms allows for).

Conversion

I kept the conversion logic from ex2ms, but assumed all (valid) remaining remote calls would have been expanded into :erlang or :trace/:table module references, after which I'd strip out the module call. This works more or less as expected.

Validation

I ran the results through :erlang.match_spec_test to detect issues at compile time rather than runtime, which seems to work fine.

Conclusion

Since full support for the ^ operator would require a new context or construct in the compiler, and since erlang does not provide authoritative information on what's valid in the matchspec grammar, I think the bespoke approach of ex2ms is the most satisfactory for now. I may find some time to contribute to it with better arity checking for trace functions, more helpful error messages, and an extra validation pass sometime--perhaps even before next ElixirConf.

Ideally I'd like to have concrete no-ops for the tracing functions within erlang, and some sort of predicate function in http://erlang.org/doc/man/erl_internal.html to handle validation of local/remote calls in matchspecs, before pursuing the handling of the pin operator. But that's more work than I'm willing to put in to this for now.