zlacker

The mathematics of training a neural network. I understand how they work once trained, but that you can train them almost seems too good to be true.

>>elgfar+(OP)
Also, for me, how all these advances in ML/computing are alleged to be on the horizon, when I hear that A/D/C(NN)s are actually so slow in learning. How can something that has to be trained with 1M of <xyz> be smart? What's the next thing?

>>elgfar+(OP)
You likely understand minimizing a continuously differentiable function. Now, you are minimizing a continuously differentiable error function (which measures the difference between the output of your hypothesis function and actual data), with respect to adjustable weights and biases that determine the value for the neurons going from one layer to the next. The complexity is in that the hypothesis function is a composition of many functions due to the layering, and there usually are a large number of neurons. However, you are basically doing the same thing many times.

>>elgfar+(OP)
A partial answer is that big fully connected neural networks _are_ pretty much untrainable. Neural networks only became successful once programmers started constraining the space they were optimizing over pretty radically (like requiring convolutional layers if you know you are trying to detect something local).

>>elgfar+(OP)
On the contrary, it is pretty simple, it is just the same process of "a little bit to the left, a little bit to the right" when you are trying to hang a poster.

>>cambal+Zs
On the contrary, this is a problem that goes "complicated-easy-complicated" the more you think about it. You're at stage two, so here's the mindfuck: Why aren't we constantly stuck at local minima? Surely these problems we're throwing at ANNs aren't all convex.

>>supern+k3
You and I probably took 4 or 5 years to recognize the alphabet. Cut the machines some slack.

>>OkayPh+XZ
How do you KNOW you are not a local minima? You are putting the cart before the horse.