next meeting this friday

[Leif Johnson]

Our next meeting will be this Friday, March 13th, at 2:30pm in GDC 3.516.

This week we'll cover a couple of papers looking to unseat the LSTM as
the go-to recurrent model of the day. The primary paper is from
Schmidhuber's lab:

http://arxiv.org/abs/1402.3511

A Clockwork RNN
Jan Koutník, Klaus Greff, Faustino Gomez, Jürgen Schmidhuber

Sequence prediction and classification are ubiquitous and challenging
problems in machine learning that can require identifying complex
dependencies between temporally distant inputs. Recurrent Neural
Networks (RNNs) have the ability, in theory, to cope with these
temporal dependencies by virtue of the short-term memory implemented
by their recurrent (feedback) connections. However, in practice they
are difficult to train successfully when the long-term memory is
required. This paper introduces a simple, yet powerful modification to
the standard RNN architecture, the Clockwork RNN (CW-RNN), in which
the hidden layer is partitioned into separate modules, each processing
inputs at its own temporal granularity, making computations only at
its prescribed clock rate. Rather than making the standard RNN models
more complex, CW-RNN reduces the number of RNN parameters, improves
the performance significantly in the tasks tested, and speeds up the
network evaluation. The network is demonstrated in preliminary
experiments involving two tasks: audio signal generation and TIMIT
spoken word classification, where it outperforms both RNN and LSTM
networks.

I'd also like to cover a paper on the Gated RNN, which takes an
alternate approach to simplify the LSTM. For this model, have a look
at section 3.2 of:

http://arxiv.org/abs/1412.3555

I think these are fairly complementary views of RNNs so it seems like
we can cover both of them in a meeting.

lmj

--
http://www.cs.utexas.edu/~leif