A question about neural networks and backpropogation

[blackllotus]

I know the backpropogation algorithm has been replaced by newer algorithms, however I have never formally studied neural networks and have only recently started looking into them as a hobby. Anyways, my question is whether a network trained using backpropogation is guarenteed to converge on learnable functions. For example, I wrote some code for a network (3 layer) and tried to train it (with 3 hidden nodes) to do the XOR function. It succeeds around 75% of the time however 25% of the time it fails and doesn't converge within 100,000 training trials. Is this a bug with my code or a property of the network?

 

[CTho9305]

Just some thoughts:

* Could you provide your code?

* What function are you applying to the sum of (inputs * weights) for the nonlinearity?

* What are you initializing the weights to? If you set them to be very far from zero, it can take an extremely long time to converge - especially if there's a gradient-descent type of thing going on and your function looks like this one (since the gradient is extremely shallow as you get far from the center).

* Are you using 0/+1 or -1/+1? I remember having issues when using 0/+1. I actually ended up using -.9 / +.9

 

[blackllotus]

I'm using tanh as the nonlinear function

Weights are initialized to random values between zero and one

I can post the code but its written in D

 

[CTho9305]

Any reason you don't initialize between -1 and 1? Are you adding a negative bias? It's not clear to me how you would compute out = !in with one neuron.

edit: Or maybe out = in1 & !in2...

 

[blackllotus]

Originally posted by: CTho9305
Any reason you don't initialize between -1 and 1? Are you adding a negative bias? It's not clear to me how you would compute out = !in with one neuron.

edit: Or maybe out = in1 & !in2...

I changed it so it initializes between -1 and 1. The bias is positive 1. I'm trying to get the network to compute exclusive or with 2 input neurons, 3 hidden neurons, and 1 output neuron.

Btw, I don't know if this helps but when the network fails to converge the deltas are off the wall (like -2) for the output neuron.

EDIT: I also assume that this means that my code is bugged in which case this thread no longer belongs in HT. Thanks for the help.

 

[imported_shader]

i'll have to doublecheck my textbook, but i'm pretty sure a nn is not guaranteed to converge. that being said, i have used nn's on several problems and at first they didn't converge, but after changing some of the settings (acceptance criteria, learning rate) i was able to get them to converge.
also i don't know if you already have this implemented, but adding a momentum parameter to the weight calculation speeds up learning drastically. if you do not know what momentum is, basically you take a percentage of the last adjustment and add that to the newest adjustment. example:
(probably not realistic numbers here)
last adjustment delta_old = 1
new adjustment delta_new = 1
new weight = 1 (delta_new) + (1 * MOMENTUM)
where MOMENTUM is from 0 - 1.
usually momentum is set very high, .7 -.9

also my teacher mentioned to my class that some reasearchers say using a separate learning rate for the hidden layers helps out. for instance i implemented this with a learning rate of .3 on the outer layers, and .5 on the hidden layers, and seemed to work very well.

 

[blackllotus]

Alright I think I found my problem but I'm not entirely sure as to how to fix it

Activation function: tanh
Delta calculation (for output layer): (expected - actual) * actual * (1 - actual)

The delta will always be negative no more what combination of expected output value and actual output value is used. What is the correct equation?

 

[CTho9305]

What is the correct equation?

Dunno. Hope this helps:

Activation function:

float Perceptron::getValue() {
float x = LinearUnit::getValue();
float s = 1/(1+pow(E, (double)-x));
float rv = s * 2.0f - 1.0f;
return rv;
}

Training:

// Update our weights, and train our inputs
void LinearUnit::train(int aIteration) {
if (mTrainIteration == aIteration) {
cout << "already trained!" << endl;
return; // already trained
}

// Update our weights
// Compute error term d = o(1-o)(blame), use prevErr for d
float prevOutput = getValue();
float err = getError(aIteration);
float prevErr = prevOutput*(1-prevOutput)*err;

int inpSize = inputNodes.size();
for (int i=0; i<inpSize; i++) {
WeightedNode *wn = inputNodes BRACKET i BRACKET ;
float v = wn->n->getValue();
float eta = 0.3f;
float delta = -eta*prevErr*v;
wn->weight += delta;
}

mTrainIteration = aIteration;

// train our inputs
for (i=0; i<inpSize; i++) {
inputNodes BRACKET i BRACKET ->n->train(aIteration);
}
}

 

[blackllotus]

Thanks for the code CTho! Fixing my problem was as simple as fixing the activation function.