Should I normalize data?

[abc-z]

I want to pass chunk of time-series with fixed size to NN, so my input will be a same [12, 45, 33, 44, 55, 22, 3]. Docs requires normalizate each element to [0, 1], but in recommended video author doesn't do it.

I suppose that I must. Perhaps add a parameter "normalize the data" of the NN is a not silly idea?

[robertleeplummerjr]

I believe you may not need to with Relu and Leaky Relu. Try this:

new brain.NeuralNetwork({ activation: 'relu' });
or
new brain.NeuralNetwork({ activation: 'leaky-relu' });

The default activation is sigmoid, which does need normalization.

[robertleeplummerjr]

Also: I know this is super lame, to use a string to define this type of thing, this will change in v2, fyi.

[abc-z]

Thanks.
So, it's a not necessary. I think this should be reflected in the docs.

to use a string to define this type of thing

Maybe better use something like below? I'm not sure but it can decrease performance.

\\ brain.js
const activation = {
    sigmoid: (sum) => 1 / (1 + Math.exp(-sum)),
    relu: (sum) => sum < 0 ? 0 : sum,
    leaky_relu: (sum) => sum < 0 ? 0 : 0.01 * sum,
    tanh: (sum) => Math.tanh(sum)
}
...
\\ remove all runInput... to one with code
       ....
        }
        // sigmoid, relu, etc
        this.outputs[layer][node] = this.activation(sum);
}

\\ Usage
var net = new brain.NeuralNetwork({ 
    activation: brain.activation.sigmoid
});

[robertleeplummerjr]

I like the suggestion! Part of the reason it is implemented as it is is that I'm trying to focus on the new GPU layer composition stuff, and lack of time. Too, the activation actually consists of two parts, an activation, and the activation's derivative. The layer composition stuff is somewhat structured the way you mention:

https://github.com/BrainJS/brain.js/tree/nn-gpu-layers/src/activation

Want to link them up? Pull Requests are welcome!

More on v2, layer composition: #96 & #99

[robertleeplummerjr]

So with the api as: https://gist.github.com/robertleeplummerjr/4aaf8afb177c9c80f8452d5025117e26

The activation is a matrix transformation, and therefore by its very nature is a "layer", albeit a simple one. But with cells (groups of layers of neurons), and gates (synonym of cell?), we are talking about layers composing layers. In short, the api looks a bit like:

export default function feedForward(settings, input) {
  const { width, height } = settings;
  const weights = random({ width, height });
  const bias = random({ width });
  return sigmoid(
    add(
      multiply(
        input,
        weights
      ),
      bias
    )
  );
}

This is a composed sigmoid FeedForward Layer, which the initial NeuralNetwork class won't ever support, but would be nice for reuse to use the activations.
The plan is to continue to full on support the original neural network concept, which is hard coded FeedForward neural network.

[robertleeplummerjr]

Let me backtrack, because the answer I gave was overly wordy.

Maybe better use something like below?

I think this is exactly what we want for v2, some of the groundwork for it has been laid with https://github.com/BrainJS/brain.js/tree/nn-gpu-layers/src/activation, would you be open to possibly implementing it?

[abc-z]

I'm trying to focus on the new GPU layer composition stuff

If main platform for brain.js is a Node.js then it's a bad idea because gpu.js bursts only browser code.
I searched what the Node.js can offer as number cracker. At the moment it's nothing. Both OpenCL and CUDA don't have a native implementation. Npm has some packages but they look abandoned. So we can only wait :(

would you be open to possibly implementing it?

I'm a newbie in NN. So I don't understand exactly what the code does. Before dive into NN I must check that NN can solve my problem (anomaly detection such as spike, shift or distribution change). If answer will be YES then I can join the brain.js development.

[robertleeplummerjr]

bursts only browser code

That is incorrect. There is a browser.js file, but brain.js is written in node javascript. It executes on browser (javascript and WebGL) and node (javascript, and eventually OpenCL). We transpile it for convenience.

Node ES6 source: https://github.com/BrainJS/brain.js/tree/master/src
Node ES5 (redistributable npm package) source: https://github.com/BrainJS/brain.js/tree/master/dist
Browser source: https://github.com/BrainJS/brain.js/blob/master/browser.js

I'm a newbie in NN.

The item you recommended it outside the realm of neural nets, and is more of a design principle. It is a great suggestion, and wouldn't require foreknowledge other than simple javascript.

[abc-z]

and eventually OpenCL

How? I don't find any information how make js code/Node faster without compile (or use node-gyp) C/C++ extensions. I think that it's impossible at the moment.

[robertleeplummerjr]

It isn't built yet (proof of concept only), but via http://gpu.rocks/, we'll use https://github.com/mikeseven/node-opencl or https://github.com/unbornchikken/NOOOCL

Likely it'll be node-opencl since it supports opencl v2

I'm on the GPU.js team :P, so I can make these types of predictions.

[jkob]

Just a question @robertleeplummerjr.
Once OpenCL is implemented, will training a NN in Node.js be as fast as a browser powered by WebGL?

Did I understand that right?

[robertleeplummerjr]

This is my hope, but it will likely come down to drivers.

[mubaidr]

Closing this as the main question was answered.