Hopfield Networks
Hopfield networks are sometimes called associative networks since they associate a class pattern to each input pattern.
The Neural Networks package supports two types of Hopfield networks, a continuous-time version and a discrete-time version. Both network types have a matrix of weights W defined as
where D is the number of class patterns {
, …, 
}, vectors consisting of +/-1 elements, to be stored in the network, and n is the number of components, the dimension, of the class pattern vectors.
Discrete-time Hopfield networks have the following dynamics:
Eq. (2.26) is applied to one state, x(t), at a time. At each iteration the state to be updated is chosen randomly. This asynchronous update process is necessary for the network to converge, which means that x(t)=Sign[W x(t)].
A distorted pattern, x(0), is used as initial state for the Eq. (2.0), and the associated pattern is the state toward which the difference equation converges. That is, starting with x(0) and then iterating Eq. (2.0) gives the associated pattern when the equation converged.
For a discrete-time Hopfield network, the energy of a certain vector x is given by
It can be shown that, given an initial state vector x(0), x(t) in Eq. (2.26) will converge to a value having minimum energy. Therefore, the minima of Eq. (2.27) constitute possible convergence points of the Hopfield network and, ideally, these minima are identical to the class patterns {
, …, 
}. Hence, one can guarantee that the Hopfield network will converge to some pattern, but one cannot guarantee that it will converge to the right pattern.
Note that the energy function can take negative values; this is, however, just a matter of scaling. Adding a sufficiently large constant to the energy expression it can be made positive.
The continuous Hopfield network is described by the following differential equation
where x(t) is the state vector of the network, W represents the parametric weights, and 
is a nonlinearity acting on the states x(t). The weights W are defined in Eq. (2.25). The differential equation, Eq. (2.28), is solved using an Euler simulation.
To define a continuous-time Hopfield network, you have to choose the nonlinear function . There are two choices supported by the package, SaturatedLinear and the default nonlinearity of Tanh.
For a continuous-time Hopfield network, defined by the parameters given in Eq. (2.25), one can define the energy of a particular state vector x as
As for the discrete-time network, it can be shown that given an initial state vector x(0)
the state vector x(t) in Eq. (2.28) converges to a local energy minimum. Hence, the minima of Eq. (2.29) constitute the possible convergence points of the Hopfield network and ideally these minima are identical to the class patterns {
, …, 
}. However, there is no guarantee that the minima will coincide with this set of class patterns.
source: http://documents.wolfram.com/applications/neuralnetworks/NeuralNetworkTheory/2.7.0.html
