Hi Dmitrii!
About 7 years ago Michael Adams, Michael Ballantyne, and I tried to explore tree automata for miniKanren, inspired by goals similar to yours! Michael Adams implemented a tree automata solver:
We had trouble, though, figuring out how to integrate the tree automata solver with the rest of miniKanren. Also, I couldn't figure out how =/= and absento work in the presence of tree automata.
I suspect there is a way to integrate tree automata and disequality constraints with miniKanren. We didn't spend much time on the integration.
I've wanted to revisit this area of research for many years. I do think tree automata are promising, and I hope you will explore this area! :)
Cheers,
--Will