Neural network in JS does not train

Neural network in JS does not train - javascript

I am trying to create a simple neural network in javascript with 2 inputs, 3 hidden and 1 output neurons that use matrixes of neurons and weights to pass forward, and backpropagation training to solve XOR problem for example. The problem is I get weights always fading to 0 fast and as a result I thing outputs equal to sigmoid(0) 0.5 value or really close. I am using a sigmoid function for activation while finding neuron values like this:
var multyplayMatrix = (a,b) => {
if(a.length !== b.length) return "length of matrix1 !== height of matrix2 :(";
let c = [];
let temp = 0;
for(var n = 0; n < b[0].length; n++){
for(var nn = 0; nn < a.length; nn++){
temp = temp + (a[nn] * b[nn][n]);
}
//sigmoid
c.push( 1 / (1 + Math.exp(-temp)) );
temp = 0;
}
return c;
}
After finding output from training data input I compare it with training data desired output and calculate error and weights delta with this:
var getError = () => {
let predicted;
if(desiredOutput !== undefined){
predicted = net.neurons[net.neurons.length-1][0]
}else{
return console.log("desired output not defined")
}
let error = predicted - desiredOutput[0];
let weights_delta = error * (predicted * (1 - predicted));
console.log("output error weight delta = " + weights_delta)
propogateBackward(weights_delta);
};
and start backpropagating with some learning rate - My backdrop function looks like this:
var learningRate = 0.5;
var propogateBackward = (output_weights_delta) => {
for(var n = 0; n < 3; n ++ ){
net.weights[net.weights.length-(1 + n)][0] = (net.weights[net.weights.length-(1 + n)][0] - net.neurons[net.neurons.length-2][2-n]) * output_weights_delta * learningRate;
}
let neuronErrors = [];
for(var n = 0; n < 3; n ++ ){
let hiddenError = output_weights_delta * net.weights[net.weights.length-(1 + n)][0];
let hidden_weights_delta = hiddenError * (net.neurons[net.neurons.length-2][2-n] * (1 - net.neurons[net.neurons.length-2][2-n]));
neuronErrors.push(hidden_weights_delta);
}
for(var n = 0; n < 3; n++){
for(var nn = 0; nn < 2; nn++){
net.weights[net.weights.length-(4 + nn)][n] = (net.weights[net.weights.length-(4 + nn)][0] - net.neurons[net.neurons.length-3][1-nn]) * neuronErrors[n] * learningRate;
}
}
draw();
}
Im using this function and XOR dataset to make a iteration/training loop like this and just after some iterations it already coverges close to 0.5
var train = (iterations) => {
for(var it = 0; it < iterations;it++){
for(var n = 0; n < dataset.length; n++){
addInput(dataset[n].inputs);
addOutput(dataset[n].outputs);
activate();
getError();
}
}
}
After googling most similar problems are connected with weight initiation in longer/deeper neural networks but that does not help here, and I did a couple of different variants of this net all get similar result so I'm probably loose on my theory/math..but where?:(
I'm just learning stats and this can be confusing, for me so excuse me if I didn't make 2 much sense - You can check the js live implementation here: https://codepen.io/sanchopanza/pen/MWaZJJe

Related

How to find all possible combinations of triplets equaling a given sum in JavaScript? ('Stars and Bars' or 'Balls and Bins' problem)

I found a similar post on StackOverflow that answers the question but in Python, and I'm looking for a solution for javascript (google script editor for a google sheet).
Balls and Bins problem
How can I solve this using JavaScript?

Figured it out using nested for loops. How to output arrays of all triples which equal to sum (numSum) by interval.
The reason I used an internal is because I am using much bigger numbers to calculate these for monetary loans, so the interval lets me calculate by $100 chunk instead of each dollar of a huge sum.
function calcSum(SUM, INTERVAL) {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var dataSheet = ss.getSheetByName('Data');
var sumNum = SUM/INTERVAL
var a = 0
var b = 0
var c = sumNum
var aTicker = 0
var bTicker = sumNum
var cTicker = sumNum
//Loop through all triple combinations of sumNum
for (i = 0; i < sumNum + 1; i++) {
for (j = aTicker; j < sumNum + 1; j++) {
var array = [];
array.push(a * INTERVAL, b * INTERVAL, c * INTERVAL);
dataSheet.getRange("A" + rowCounter + ":D" + rowCounter).setValues([array]);
rowCounter++;
}
if (b < bTicker) {
b++;
} else {
b = 0;
bTicker--;
}
if (c > 0){
c--;
} else {
cTicker--;
c = cTicker;
}
}
a++;
aTicker++;
}
}

Javascript collision explanation for b-tree (Birthday Paradox)?

I want to create a b-tree for storing some data in local storage. I was thinking of using this to find the index of an ID in a sorted list.
If I index an array normally (i.e. to append like array[20032] = 123, what's the big-O of that in Javascript arrays?).
function sortedIndex(array, value) {
var low = 0,
high = array.length;
while (low < high) {
var mid = (low + high) >>> 1;
if (array[mid] < value) low = mid + 1;
else high = mid;
}
return low;
}
When I test this with random numbers, I get some collision and it exits before 10k.
for (i = 0; i < 10000; i++) {
var r = Math.random();
array[sortedIndex(array,r)] = r;
}
This exits after a certain time (I'm assuming because of a collision).
I'm thinking it's a birthday paradox kind of thing because the collisions seem to be more likely when the list is already populated (see graph link) (but no exception is thrown...).
I wanted to see the final length of the array after many iterations, I get distribution of final lengths that look like this:
sortedList = []
listLengths = []
for (j = 0; j < 100; j++) {
for (i = 0; i < 10000; i++) {
var r = Math.random();
sortedList[sortedIndex(sortedList,r)] = r;
}
listLengths.push(sortedList.length);
}
graph of final lengths of sorted array after 1-100 iterations of appending attempts
I honestly don't want to deal with this and would also appreciate some pointers on efficient localStorage libraries.

The problem is that you're not shifting all the old elements up when you insert a new element in the array. So you'll extend the array by 1 when the new item is higher than anything else, but just overwrite an existing element when it's less than or equal to the maximum element.
array.splice will insert and move everything over to make room.
array = [];
listLengths = [];
for (j = 0; j < 100; j++) {
for (i = 0; i < 100; i++) {
var r = Math.random();
array.splice(sortedIndex(array, r), 0, r);
}
listLengths.push(array.length);
}
console.log(listLengths);
function sortedIndex(array, value) {
var low = 0,
high = array.length;
while (low < high) {
var mid = (low + high) >>> 1;
if (array[mid] < value) low = mid + 1;
else high = mid;
}
return low;
}

compute determinant of a matrix

I'm trying to compute determinant of a matrix in JS. I used algorithm from http://www.sanfoundry.com/java-program-compute-determinant-matrix/ but I lost my mind on the last condition. I just do not understand. Can you help me?
This is how looks like my code right now. In another function I create an empty 2d array and then copy it to det function. Next I retrieve values from html and then trying to compute determinant of a matrix. The first 2 cases are simple but I have a problem with the last one. I couldn't find working example in JS.
function det() {
var det = 0;
var array1 = array.slice();
for (i = 0; i < array1.length; i++) {
for (j = 0; j < array1[i].length; j++) {
array1[i][j] = parseInt(document.getElementById("element" + (i + 1) + (j + 1)).value, 10);
}
}
if (array1.length == 1) {
det = array1[0][0];
} else if (array1.length == 2) {
det = (array1[0][0] * array1[1][1]) - (array1[1][0] * array1[0][1]);
} else {
}
}

I may suggest my solution, based on recursive algorithm, which takes only few lines of code and, I guess, will suit most of practical applications:
const determinant = m =>
m.length == 1 ?
m[0][0] :
m.length == 2 ?
m[0][0]*m[1][1]-m[0][1]*m[1][0] :
m[0].reduce((r,e,i) =>
r+(-1)**(i+2)*e*determinant(m.slice(1).map(c =>
c.filter((_,j) => i != j))),0)
const test1 = [[3]] // 3
const test2 = [[3,-2],[7,4]] // 26
const test3 = [[1,3,7],[2,-1,4],[5,0,2]] // 81
console.log(determinant(test1))
console.log(determinant(test2))
console.log(determinant(test3))
.as-console-wrapper {min-height: 100%}

I created a Matrix class with some function for the basic operations, one of them the determinant calculation
Here we have the constructor
constructor(rows,cols) {
this.rows = rows;
this.cols = cols;
this.vals = new Array(rows*cols);
for(let i = 0; i < this.vals.length; i++) this.vals[i] = 0;
}
and here the determinant function
determinant() {
if (this.rows != this.cols ) {
console.log("Matrix must be square");
return;
}
let size = this.rows;
if (size == 1) return this.vals[0];
if (size == 2) return this.vals[0]*this.vals[3]-this.vals[1]*this.vals[2];
let sign = 1;
let result = 0;
for (let k = 0 ; k < size ; k++){
let cofactors = new Matrix(size-1,size-1);
cofactors.vals = this.vals.slice(size,size*size).filter((_,index)=>index%size!=k);
result += sign * this.vals[k] * cofactors.determinant();
sign*=(-1);
}
return result;
}

You can see definition of the determinant for square matrix here https://en.wikipedia.org/wiki/Determinant#n_.C3.97_n_matrices.
Algorithm used in http://www.sanfoundry.com/java-program-compute-determinant-matrix/ use some properties of determinat to calculate it in recursive way as sum over all permutations. In this way you get N * N! operations! It is very big even for small N.
For solving this problem you can first transform matrix to triangular with the same determinant and after that calculate determinant as product of all diagonal elements.

is this the right approach to calculate cosine similarity?

If you guys can please review if the following approach (pseudo-code) is good to go to calcualte cosine similarity between 2 vectors:
var vectorA = [2,5,7,8];
var referenceVector= [1,1,1,1];
//Apply weights to vectors (apply positive or negative weights to elements)
var weightageVector = [1,0.5,2,1.5];
var weighted vectA = GetWeightedVector(vectorA);
//normalize each element to a value beteen 0 and 1
//#see http://stn.spotfire.com/spotfire_client_help/norm/norm_scale_between_0_and_1.htm
as calcuated here:http://jsfiddle.net/snehilw/86jqo1sm/4/
var normalizedVectorA = GetNormalizedVector(vectorA); //using the formula above
var cosineSimilarityScore = GetCosineSimilarityScore(referenceVector, normalizedVectorA );
can someone please advise if this is correct approach as this is not giving me correct results.
As requested, here is the code snippet:
var defaultVectorWeights = [1,0.5,2,1.5];
var referenceVector = [1, 1, 1, 1] //Default values for the reference vector (Do not change these);
var supportedVectorLength = referenceVector.length;
function getNormalizedVector(multiDimArray, vector){
var normalizedVector = [];
if(vector.length == supportedVectorLength){
var normalizedValue = 0;
for(var j = 0; j < supportedVectorLength ; j++){
var min = getMinMaxForMultidimensionalArrayColumn(multiDimArray,j)[0];
var max = getMinMaxForMultidimensionalArrayColumn(multiDimArray,j)[1];
normalizedValue = (max == min) ? 0.5 : (vector[j] - min) / (max - min);
normalizedVector.push(normalizedValue);
}
}
//console.log('normalizedVector='+normalizedVector);
return normalizedVector;
}
function getCosineSimilarityScore(vectorA, vectorB) {
var similarityScore;
if((vectorA.length == supportedVectorLength) && (vectorB.length == supportedVectorLength)){
var lenVectA = vectorA.length,
product = 0,
normVectorA = 0,
normVectorB = 0;
for (var i = 0; i < lenVectA ; i++) {
product += vectorA[i] * vectorB[i];
normVectorA += vectorA[i] * vectorA[i];
normVectorB += vectorB[i] * vectorB[i];
}
similarityScore = product / (Math.sqrt(normVectorA) * Math.sqrt(normVectorB));
}
else {
//TODO: Handle exception/ Fire an event to notify the server about this exception
console.log("Cosine similarity workload vectors are of unequal lengths");
}
return similarityScore;
}
function getWeightedVector(vector) {
var vectorArray = []; //Initialize
if(vector.length == supportedVectorLength){
for(var j = 0; j < supportedVectorLength ; j++){
vectorArray.push(defaultVectorWeights[j]*vector[j]);
}
}
else{
//TODO: Handle exception/ Fire an event to notify the server about this exception
console.log("Cosine similarity workload vector is of unsupported length");
}
return vectorArray;
}
function getMinMaxForMultidimensionalArrayColumn(multiDimArray, column){
var _MIN_MAX = []; //[min,max]
var columnarArray = [];
if(column < supportedVectorLength){
//Extract columnar array from the multi-dimensional array
$.map(multiDimArray, function( arrayVect) {
columnarArray.push(arrayVect[column]);
});
//Find the MIN and MAX
_MIN_MAX.push(Math.min.apply(Math,columnarArray));
_MIN_MAX.push(Math.max.apply(Math,columnarArray));
}
else{
//TODO: Handle exception/ Fire an event to notify the server about this exception
console.log("Cosine similarity workload vectors are of unequal lengths");
}
return _MIN_MAX;
}
function getAssociateWorkloadScore(multiDimArray,queryVector){
var workloadScore;
var weightedQueryVector = [];
var weightedMultiDimArr = [];
var normalizedMultiDimArr = [];
var normalizedQueryVector = [];
//Apply feature scaling
weightedQueryVector = getWeightedVector(queryVector);
weightedMultiDimArr = getWeightedMultiDimArr(multiDimArray);
normalizedQueryVector = getNormalizedVector(weightedMultiDimArr, weightedQueryVector);
workloadScore = getCosineSimilarityScore(referenceVector, normalizedQueryVector);
console.log('weightedQueryVector='+weightedQueryVector);
console.log('weightedMultiDimArr='+JSON.stringify(weightedMultiDimArr));
console.log('normalizedMultiDimArr='+JSON.stringify(normalizedMultiDimArr));
console.log('normalizedQueryVector='+normalizedQueryVector);
console.log('workloadScore='+JSON.stringify(workloadScore));
return workloadScore;
}
function getTeamWorkloadScore(multiDimArray){
var workloadScores = [];
for(var j = 0; j < multiDimArray.length ; j++){
workloadScores.push(getAssociateWorkloadScore(multiDimArray,multiDimArray[j]));
}
return workloadScores;
}

A cosine similarity is just a dot product divided by the product of norms. So why not make a dot product function and a norm function and divide the results? (dotproduct from http://c2.com/cgi/wiki?DotProductInManyProgrammingLanguages)
function dotproduct(a,b) {
var n = 0, lim = Math.min(a.length,b.length);
for (var i = 0; i < lim; i++) n += a[i] * b[i];
return n;
}
function norm2(a) {var sumsqr = 0; for (var i = 0; i < a.length; i++) sumsqr += a[i]*a[i]; return Math.sqrt(sumsqr);}
function similarity(a, b) {return dotproduct(a,b)/norm2(a)/norm2(b);}
Now similarity([1,0,0], [0,1,1]) == 0

If you necessarily need scale-invariance (i.e., the original cosine similarity), then use Gavin's code augmented with checks for zero-vectors
function cosine_sim(x, y) {
xnorm = norm2(x);
if(!xnorm) return 0;
ynorm = norm2(y);
if(!ynorm) return 0;
return dotproduct(x, y) / (xnorm * ynorm);
}
If you do not need scale-invariance, just use the dot product (i.e., cosine_sim(x, y) is dotproduct(x, y)).

Filling up a 2D array with random numbers in javascript

I'm really sorry if anything like this has been posted here before but I couldn't find anything, I'm kinda new to the site still!
So for a while now I've been learning a bit about game development through html5 and javascript and I stumbled upon making tileset maps, I now have a tileset and an 2D array that I want to put certain tiles in (the number varies between 6 and 10 in this case).
I figured it could be a cool function to make the map choose between a small set of similar tiles so I don't have to specifically number every tile in the array(just define the type)
The method I have currently is probably the best for being able to define types but I want something that looks a bit cleaner and/or information to why my "cleaner" version dosen't work.
var ground = [
[tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile()],
[tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile()],
[tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile()],
[tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile()],
[tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile()],
[tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile()],
[tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile()],
[tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile()],
[tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile(),tile()]];
function tile() {
var y = (Math.random() * 5 | 0) + 6;
return y;
}
This is the code I've been using so far, I have to edit every element of the code with the tile() function to get a random number in each one, what I wanted to have was something like this:
for (var i = 0 ; i < 15; i++) {
for (var j = 0; j < 9; j++) {
ground[[i],[j]] = (Math.random() * 5 | 0) + 6;
}
}
to fill the array without having to add the function to each spot.
I have a feeling that I'm missing a return function or something along those lines but honestly I have no idea.

You were thinking in the right direction but there are some errors in your code ;)
You have to initialize the array first before you can push elements into it.
And you were counting i++ twice
Javascript
var ground = []; // Initialize array
for (var i = 0 ; i < 15; i++) {
ground[i] = []; // Initialize inner array
for (var j = 0; j < 9; j++) { // i++ needs to be j++
ground[i][j] = (Math.random() * 5 | 0) + 6;
}
}
Maybe even better (reusable)
function createGround(width, height){
var result = [];
for (var i = 0 ; i < width; i++) {
result[i] = [];
for (var j = 0; j < height; j++) {
result[i][j] = (Math.random() * 5 | 0) + 6;
}
}
return result;
}
// Create a new ground with width = 15 & height = 9
var ground = createGround(15, 9);

Here's a quick example. I've created a function that will take in a width and height parameter and generate the size requested. Also I placed your tile function inside generate ground to keep it private, preventing other script from invoking it.
var ground = generateGround(10, 10); //Simple usage
function generateGround(height, width)
{
var ground = [];
for (var y = 0 ; y < height; y++)
{
ground[y] = [];
for (var x = 0; x < width; x++)
{
ground[y][x] = tile();
}
}
return ground;
function tile()
{
return (Math.random() * 5 | 0) + 6;
}
}
http://jsbin.com/sukoyute/1/edit

Try removing the comma from...
ground[[i],[j]] = (Math.random() * 5 | 0) + 6;
...in your 'clean' version. Also, your incrementing 'i' in both for loops:
for (var i = 0 ; i < 15; i++) {
for (var j = 0; j < 9; i++) {
Hopefully these changes make it work for you :)

Develop Reference

JavaScript is the programming language of the Web.

Neural network in JS does not train - javascript

Related

How to find all possible combinations of triplets equaling a given sum in JavaScript? ('Stars and Bars' or 'Balls and Bins' problem)

Javascript collision explanation for b-tree (Birthday Paradox)?

compute determinant of a matrix

is this the right approach to calculate cosine similarity?

Filling up a 2D array with random numbers in javascript

Categories

Resources