I really need help implementing a continuous tanh-sigmoid activation function in a very basic neural network. If you could give a basic example that would be great, but if you could change it in my source code I would be extremely grateful! Also, what range should the random weights be initiated with (i.e. what range)?
The weight range depends on what input data range you have. In some implementations the weights can also be negative.
For possible Sigmoid functions, check here (tanh is not the only possibility):
http://en.wikipedia.org/wiki/Sigmoid_function
Tip: You can typically compute the NN with matrix multiplications.
http://www.dtreg.com/mlfn.htm
http://en.wikipedia.org/wiki/Neural_network
P.S.: probably not a good idea to do this in JavaScript.
you can either implement it via exp(x) , See: http://www.javascripter.net/faq/mathfunc.htm
sinh(x) exp(x) - exp(-x) exp(2x) - 1
tanh(x) = ------- = ------------------ = -------------
cosh(x) exp(x) + exp(-x) exp(2x) + 1
that gives you:
function tanh(x) {
e = Math.exp(2*x);
return (e - 1) / (e + 1) ;
};
another solution is to store a table with the tanh function values in an array, and define a JavaScript function which interpolates the tanh values for x based on the tanh values stored in the array
typically people don't want [-inf...+inf] as the range of the input values, and don't want [-1...+1] as the range of output values -- therefore you might need a different sigmoid function!
you need to take the expected range of input values, and the expected range of output values, and use those to shift the actual sigmoid function, the weight-ranges and the value of the threshhold.
a threshhold of 0.7 or larger is typically used. You need to experiment with that.
this.output = 2 / (1 + Math.exp(-2 * input)) - 1;
Related
I have an array of N HTMLCanvasElements that come from N frames of a video, and I want to compute the "median canvas" in the sense that every component (r, g, b, opacity) of every pixel is the median of the corresponding component in all the canvases.
The video frames are 1280x720, so that the pixels data for every canvas (obtained with canvas.getContext('2d').getImageData(0, 0, canvas.width, canvas.height).data) is a Uint8ClampedArray of length 3.686.400.
The naive way to compute the median is to:
prepare a result Uint8ClampedArray of length 3.686.400
prepare a temporary Uint8ClampedArray of length N
loop from 0 to 3.686.399
a) loop over the N canvases to fill the array
b) compute the median of the array
c) store the median to the result array
But it's very slow, even for 4 canvases.
Is there an efficient way (or existing code) to do that? My question is very similar to Find median of list of images, but I need to to this in JavaScript, not Python.
Note: for b), I use d3.median() which doesn't work on typed arrays, as far as I understand, so that it implies converting to numbers, then converting back to Uint8Clamped.
Note 2: I don't know much of GLSL shaders, but maybe using the GPU would be a way to get faster results. It would require to pass data from the CPU to the GPU though, which takes time if done repeatedly.
Note 3: the naive solution is there: https://observablehq.com/#severo/compute-the-approximate-median-image-of-a-video
You wrote
I use d3.median() which doesn't work on typed arrays…
Although that is not exactly true it leads into the right direction. Internally d3.median() uses the d3.quantile() method which starts off like this:
export default function quantile(values, p, valueof) {
values = Float64Array.from(numbers(values, valueof));
As you can see, this in fact does make use of typed arrays, it is just not your Uint8ClampedArray but a Float64Array instead. Because floating-point arithmetic is much more computation-intensive than its integer counterpart (including the conversion itself) this has a dramatic effect on the performance of your code. Doing this some 3 million times in a tight loop kills the efficiency of your solution.
Since you are retrieving all your pixel values from a Uint8ClampedArray you can be sure that you are always dealing with integers, though. That said, it is fairly easy to build a custom function median(values) derived from d3.median() and d3.quantile():
function median(values) {
// No conversion to floating point values needed.
if (!(n = values.length)) return;
if (n < 2) return d3.min(values);
var n,
i = (n - 1) * 0.5,
i0 = Math.floor(i),
value0 = d3.max(d3.quickselect(values, i0).subarray(0, i0 + 1)),
value1 = d3.min(values.subarray(i0 + 1));
return value0 + (value1 - value0) * (i - i0);
}
On top of getting rid of the problematic conversion on the first line this implementation additionally applies some more micro-optimizations because in your case you are always looking for the 2-quantile (i.e. the median). That might not seem much at first, but doing this multiple million times in a loop it does make a difference.
With minimal changes to your own code you can call it like this:
// medianImageData.data[i] = d3.median(arr); Instead of this use line below.
medianImageData.data[i] = median(arr);
Have a look at my working fork of your Observable notebook.
From what I understand you use a sigmoid function to reduce a number to the range of 0-1.
Using the function found in this library
function sigmoid(z) {
return 1 / (1 + Math.exp(-z));
}
This works for a numbers 1-36. Any number higher than this will just return 1.
sigmoid(36) -> 0.9999999999999998
sigmoid(37) -> 1
sigmoid(38) -> 1
sigmoid(9000) -> 1
How do you increase the range so this function can handle a number larger than 36.
A sigmoid function is any function which has certain properties which give it the characteristic s-shape. Your question has many answers. For example, any function whose definition looks like
const k = 2;
function sigmoid(z) {
return 1 / (1 + Math.exp(-z/k));
}
will fit the bill. The larger the k, the larger the useful domain.
A Sigmoid Function doesn't have bounds, that means it accept from infinitely small to infinitely large values.
Javascript, on the other hand, will round numbers (IEEE).
Anyway, what you can do is reescale your input before passing it to the formula.
Another option is tinker with the formula values, most notably the z value.
I'm experiencing an issue with a formula in JavaScript.
var animMoveX = $(this).attr('data-start') + (animPercentage / 100) * ($(this).attr('data-finish') - $(this).attr('data-start'));
To my eyes it's a fairly simple piece of math, but the console outputs 120-[variable no relative to animPercentage, eg. 126.49681528662421].
I've double-checked all variables, and they are correct, and if I replace one of the $(this).attr('data-start') variable in one of the positions with a fixed number, then the calculation is run just fine. Can someone shed some light on why this is, and how I could potentially work around it?
From my comment: Precedence means it will calculate a number on the right and add it to the string from data-start (i.e. using string concatenation). That needs to be converted to a number too. #Pointy's suggestion will do that as data converts strings to appropriate data types (when it can).
So basically change all the attr() calls to data() calls and "numbers" (stored in attributes) will become numbers:
var animMoveX = $(this).data('start') + (animPercentage / 100) * ($(this).data('finish') - $(this).data('start'));
As an added bonus, using data instead of attr is shorter code too :)
I'm currently making a Conway's Game of Life reproduction in JavaScript and I've noticed that the function Math.random() is always returning a certain pattern. Here's a sample of a randomized result in a 100x100 grid:
Does anyone knows how to get better randomized numbers?
ApplyRandom: function() {
var $this = Evolution;
var total = $this.Settings.grid_x * $this.Settings.grid_y;
var range = parseInt(total * ($this.Settings.randomPercentage / 100));
for(var i = 0; i < total; i++) {
$this.Infos.grid[i] = false;
}
for(var i = 0; i < range; i++) {
var random = Math.floor((Math.random() * total) + 1);
$this.Infos.grid[random] = true;
}
$this.PrintGrid();
},
[UPDATE]
I've created a jsFiddle here: http://jsfiddle.net/5Xrs7/1/
[UPDATE]
It seems that Math.random() was OK after all (thanks raina77ow). Sorry folks! :(. If you are interested by the result, here's an updated version of the game: http://jsfiddle.net/sAKFQ/
(But I think there's some bugs left...)
This line in your code...
var position = (y * 10) + x;
... is what's causing this 'non-randomness'. It really should be...
var position = (y * $this.Settings.grid_x) + x;
I suppose 10 was the original size of this grid, that's why it's here. But that's clearly wrong: you should choose your position based on the current size of the grid.
As a sidenote, no offence, but I still consider the algorithm given in #JayC answer to be superior to yours. And it's quite easy to implement, just change two loops in ApplyRandom function to a single one:
var bias = $this.Settings.randomPercentage / 100;
for (var i = 0; i < total; i++) {
$this.Infos.grid[i] = Math.random() < bias;
}
With this change, you will no longer suffer from the side effect of reusing the same numbers in var random = Math.floor((Math.random() * total) + 1); line, which lowered the actual cell fillrate in your original code.
Math.random is a pseudo random method, that's why you're getting those results. A by pass i often use is to catch the mouse cursor position in order to add some salt to the Math.random results :
Math.random=(function(rand) {
var salt=0;
document.addEventListener('mousemove',function(event) {
salt=event.pageX*event.pageY;
});
return function() { return (rand()+(1/(1+salt)))%1; };
})(Math.random);
It's not completly random, but a bit more ;)
A better solution is probably not to randomly pick points and paint them black, but to go through each and every point, decide what the odds are that it should be filled, and then fill accordingly. (That is, if you want it on average %20 percent chance of it being filled, generate your random number r and fill when r < 0.2 I've seen a Life simulator in WebGL and that's kinda what it does to initialize...IIRC.
Edit: Here's another reason to consider alternate methods of painting. While randomly selecting pixels might end up in less work and less invocation of your random number generator, which might be a good thing, depending upon what you want. As it is, you seem to have selected a way that, at most some percentage of your pixels will be filled. IF you had kept track of the pixels being filled, and chose to fill another pixel if one was already filled, essentially all your doing is shuffling an exact percentage of black pixels among your white pixels. Do it my way, and the percentage of pixels selected will follow a binomial distribution. Sometimes the percentage filled will be a little more, sometimes a little less. The set of all shufflings is a strict subset of the possibilities generated this kind of picking (which, also strictly speaking, contains all possibilities for painting the board, just with astronomically low odds of getting most of them). Simply put, randomly choosing for every pixel would allow more variance.
Then again, I could modify the shuffle algorithm to pick a percentage of pixels based upon numbers generated from a binomial probability distribution function with a defined expected/mean value instead of the expected/mean value itself, and I honestly don't know that it'd be any different--at least theoretically--than running the odds for every pixel with the expected/mean value itself. There's a lot that could be done.
console.log(window.crypto.getRandomValues(new Uint8Array(32))); //return 32 random bytes
This return a random bytes with crypto-strength: https://developer.mozilla.org/en/docs/Web/API/Crypto/getRandomValues
You can try
JavaScript Crypto Library (BSD license). It is supposed to have a good random number generator. See here an example of usage.
Stanford JavaScript Crypto Library (BSD or GPL license). See documentation for random numbers.
For a discussion of strength of Math.random(), see this question.
The implementation of Math.random probably is based on a linear congruential generator, one weakness of which is that a random number depends on the earlier value, producing predictable patterns like this, depending on the choice of the constants in the algorithm. A famous example of the effect of poor choice of constants can be seen in RANDU.
The Mersenne Twister random number generator does not have this weakness. You can find an implementation of MT in JavaScript for example here: https://gist.github.com/banksean/300494
Update: Seeing your code, you have a problem in the code that renders the grid. This line:
var position = (y * 10) + x;
Should be:
var position = (y * grid_x) + x;
With this fix there is no discernible pattern.
You can using the part of sha256 hash from timestamp including nanoseconds:
console.log(window.performance.now()); //return nanoseconds inside
This can be encoded as string,
then you can get hash, using this: http://geraintluff.github.io/sha256/
salt = parseInt(sha256(previous_salt_string).substring(0, 12), 16);
//48 bits number < 2^53-1
then, using function from #nfroidure,
write gen_salt function before, use sha256 hash there,
and write gen_salt call to eventListener.
You can use sha256(previous_salt) + mouse coordinate, as string to get randomized hash.
I've made a scaling function that takes numbers in an interval [oldMin,oldMax] and scales them linearly to the range [newMin,newMax] . It does not seem to work when using negative values.
function linearScaling(oldMin, oldMax, newMin, newMax, oldValue){
var newValue;
if(oldMin !== oldMax && newMin !== newMax){
newValue = parseFloat((((oldValue - oldMin) * (newMax - newMin)) / (oldMax - oldMin)) + newMin);
newValue = newValue.toFixed(2);
}
else{
newValue = error;
}
return newValue;
}
This function seems to work when scaling a value from 0 -> 32761 to the range range 0 -> 10. However it does not seem to give the correct output when given a new negative range i.e. -10 -> 10
I have done my best to find an answer on this site. However the person who asked the question didn't mention what he ended up doing to fix it. That question says it could have something to do with mixed up data types, but i converted everything to a float did I miss anything?
Now that you showed how you call your function, I can reproduce your problem - namely that quoted numbers that should map to the negative domain don't.
It seems to be due to the fact that Javascript is very loose about the difference between a number and a string - and if it's not sure what to do about two numbers (because one of them appears to be a string), it assumes you want concatenation rather than addition. In other words - by passing the newMin value as '-10' rather than -10 you confused JS.
As a simple example,
document.write('1' + '-2');
produces
1-2
However,
document.write(1*'1' + 1*'-2');
results in
-1
The expression you had included a "possible concatenation" where it added oldMin:
newValue = (((oldValue - oldMin) * (newMax - newMin)) / (oldMax - oldMin)) + newMin;
With newMin set to '-10', you might get newValue to look like 6-10 instead of -4, to give an example. When you then did a parseFloat, Javascript would quietly work its way through the string up to the minus sign, and return 6 instead of evaluating the expression and coming up with -4.
To clear up the confusion, multiply each parameter by 1 to make it "a genuine number":
oldMin = 1*oldMin;
oldMax = 1*oldMax;
newMin = 1*newMin;
newMax = 1*newMax;
oldValue = 1*oldValue;
When you add these lines at the start of your function declaration, everything works smoothly - regardless of how you call the function. Or just call it with the newMin value not in quotes - it is the one causing the trouble in this particular instance.
document.writeln('the new code called with parameter = 100:\n');
document.writeln(linearScaling('0', '32761', '-10', '10', 100)+'<br>');
document.writeln('the old code called with parameter = 100:\n');
document.writeln(linearScalingOld('0.0', '32761.0', '-10.0', '10.0', '100.0')+'<br>');
document.writeln('the old code called with unquoted parameters:\n');
document.writeln(linearScalingOld(0.0, 32761.0, -10.0, 10.0, 100.0)+'<br>');
results in the following:
the new code called with parameter = 100: -9.94
the old code called with parameter = 100: 0.06
the old code called with unquoted parameters: -9.94
I hope this illustrates the cause of the problem, and the solution.