CARFAC in Tensorflow

[Viktor Tóth]

I aim to include CARFAC in a autoencoder (img->sound->img) machine learning model in order to compress audio in the decoder, quasi modeling the capabilities of human hearing and how it could reconstruct the original image once received the corresponding sound.

I've converted the Python version (https://github.com/vschaik/CARFAC) to a Tensorflow graph: https://pastebin.com/zdV1CfA2

The building of the graph is ridiculously slow. I know this is not a Tensorflow mailing list, my major question is, how should one go about to turn CARFAC into a machine learning model, through which backprop can flow?

This is part of my neuroscience Master's thesis, thank you in advance for any help!

[Dick Lyon]

I'm impressed that you could make a TF graph for CARFAC.  TF is not great for such highly recurrent structures, and I bet it's very hard to backpropagate through the long time histories of the filters.  But I'm also no TF expert, so can't offer much help.  Thanks for taking this on in any case.

Dick

--
You received this message because you are subscribed to the Google Groups "carfac-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to carfac-dev+...@googlegroups.com.
Visit this group at https://groups.google.com/group/carfac-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/carfac-dev/c4d6cc5b-edcb-44ce-abc6-3bb0fe3a7f3a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Viktor Tóth]

Yes it seems to be pretty hard to backprop through it, also the slicing, the updates on subset of tensors is difficult to implement.

This question might be a little high level, but do you think the input-output function of CARFAC, including the inner hair cell, can be approximated by a RNN, maybe structurally engineered RNN fitting CARFAC? Approximate the functionality only for a subspace of soundscapes that are well defined and I could generate a myriad of examples of? I'd like to replicate the suppression and masking mechanisms mainly.

P.S. I just read your book, it's a huge help in my visual to auditory sensory substitution research, thank you!

[Viktor Tóth]

I have implemented a tensorflow version that seemingly gives the same response as the python implementation. You can find it here: https://pastebin.com/kt2zc1mH. It takes an awful lot of time and memory to build the network, but it works.