zlacker

the difference is the basis that is chosen. Fourier use sin and cos as a basis (or equivalently complex exponentials). You can choose other bases and get wavelets, or hermite functions, or any other particular independent functions.

Weights on neural networks don't have to be independent functions.

Independence gives you a set of mathematical guarantees that insure you fully cover the space you're representing. For example that given a 2 dimensional space, X and Y are pointing in different directions. If they pointed in the same direction you could not fully decompose all vectors on the plane into two coefficients of X and Y.