The paper introduces a new RNN variant that uses multiplicative (or “gated”) connections which allow the current input character to determine the transition matrix from one hidden state vector to the next. The goal of the paper is to demonstrate the power of large RNNs trained with the new Hessian-Free optimizer by applying them to the task of predicting the next character in a stream of text. “MRNN” architecture uses multiplicative connections to allow the current input character to determine the hidden-to-hidden weight matrix.

The Tensor RNN

The dynamics of the RNN’s hidden states depend on the hidden-to-hidden matrix and on the inputs. In a standard RNN, the current input x t is first transformed via the visible-to-hidden weight matrix W hx and then contributes additively to the input for the current hidden state. A more powerful way for the current input character to affect the hidden state dynamics would be to determine the entire hidden-to-hidden matrix (which de- fines the non-linear dynamics) in addition to providing an additive bias.

For example, the character string “ing” is quite probable after “fix” and also quite probable after “break”. If the hidden state vectors that represent the two histories “fix” and “break” share a common representation of the fact that this could be the stem of a verb, then this common representation can be acted upon by the character “i” to produce a hidden state that predicts an “n”. For this to be a good prediction we require the conjunctionof the verb-stem representation in the previous hidden state and the character “i”. One or other of these alone does not provide half as much evidence for predicting an “n”: It is their conjunction that is important. This strongly suggests that we need a multiplicative interaction. To achieve this goal we modify the RNN so that its hidden- to-hidden weight matrix is a (learned) function of the current input x t :

This allows each character to specify a different hidden-to-hidden weight matrix.

It is natural to define W(xt) hh using a tensor. If we store M matrices, W1 hh ,…,W(M) hh , where M is the number of dimensions of x t , we could define W(xt) hh by the equation

where x(m) t is the m-th coordinate of x t . When the input x t is a 1-of-M encoding of a character, it is easily seen that every character has an associated weight matrix and W(xt) hh is the matrix assigned to the character represented by x t .

The Multiplicative RNN (MRNN)

The above scheme, while appealing, has a major drawback: Fully general 3-way tensors are not practical because of their size. In particular, if we want to use RNNs with a large number of hidden units (say, 1000) and if the dimensionality of x t is even moderately large, then the storage required for the tensor W(xt) hh becomes prohibitive.

It turns out we can remedy the above problem by factoring the tensor W (xt) hh . This is done by introducing the three matrices W fx , W hf , and W fh , and reparameterizing the matrix W(xt) hh by the equation

The Multiplicative RNN (MRNN) is the result of factorizing the Tensor RNN. The MRNN computes the hidden state sequence (h 1 , . . . , h T ), an additional “factor state sequence” (f 1 , . . . , f T ), and the output sequence (o 1 , . . . , o T ) by iterating the equations

The tensor factorization has the interpretation of an additional layer of multiplicative units between each pair of consecutive layers, so the MRNN actually has two steps of nonlinear processing in its hidden states for every input timestep. Each of the multiplicative units outputs the value f t which is the product of the outputs of the two linear filters connecting the multiplicative unit to the previous hidden states and to the inputs.

The Result

The MRNN predicts the test set more accurately than the sequence memoizer but less accurately than the dictionary-free PAQ on the three datasets.

Disclosure

Most of the things are directly from the paper. This post is meant to be a one place for all the papers that I read and take notes. You can read the paper in its entirety here.