Develop Reference

How to improve accuracy of a FeedForward Neural Network?

I want to draw StackOverflow's logo with this Neural Network:
The NN should ideally become [r, g, b] = f([x, y]). In other words, it should return RGB colors for a given pair of coordinates. The FFNN works pretty well for simple shapes like a circle or a box. For example after several thousands epochs a circle looks like this:
Try it yourself: https://codepen.io/adelriosantiago/pen/PoNGeLw
However since StackOverflow's logo is far more complex even after several thousands of iterations the FFNN's results are somewhat poor:
From left to right:
StackOverflow's logo at 256 colors.
With 15 hidden neurons: The left handle never appears.
50 hidden neurons: Pretty poor result in general.
0.03 as learning rate: Shows blue in the results (blue is not in the orignal image)
A time-decreasing learning rate: The left handle appears but other details are now lost.
Try it yourself: https://codepen.io/adelriosantiago/pen/xxVEjeJ
Some parameters of interest are synaptic.Architect.Perceptron definition and learningRate value.
How can I improve the accuracy of this NN?
Could you improve the snippet? If so, please explain what you did. If there is a better NN architecture to tackle this type of job could you please provide an example?
Additional info:
Artificial Neural Network library used: Synaptic.js
To run this example in your localhost: See repository

By adding another layer, you get better results :
let perceptron = new synaptic.Architect.Perceptron(2, 15, 10, 3)
There are small improvements that you can do to improve efficiency (marginally):
Here is my optimized code:
const width = 125
const height = 125
const outputCtx = document.getElementById("output").getContext("2d")
const iterationLabel = document.getElementById("iteration")
const stopAtIteration = 3000
let perceptron = new synaptic.Architect.Perceptron(2, 15, 10, 3)
let iteration = 0
let inputData = (() => {
const tempCtx = document.createElement("canvas").getContext("2d")
tempCtx.drawImage(document.getElementById("input"), 0, 0)
return tempCtx.getImageData(0, 0, width, height)
})()
const getRGB = (img, x, y) => {
var k = (height * y + x) * 4;
return [
img.data[k] / 255, // R
img.data[k + 1] / 255, // G
img.data[k + 2] / 255, // B
//img.data[(height * y + x) * 4 + 3], // Alpha not used
]
}
const paint = () => {
var imageData = outputCtx.getImageData(0, 0, width, height)
for (let x = 0; x < width; x++) {
for (let y = 0; y < height; y++) {
var rgb = perceptron.activate([x / width, y / height])
var k = (height * y + x) * 4;
imageData.data[k] = rgb[0] * 255
imageData.data[k + 1] = rgb[1] * 255
imageData.data[k + 2] = rgb[2] * 255
imageData.data[k + 3] = 255 // Alpha not used
}
}
outputCtx.putImageData(imageData, 0, 0)
setTimeout(train, 0)
}
const train = () => {
iterationLabel.innerHTML = ++iteration
if (iteration > stopAtIteration) return
let learningRate = 0.01 / (1 + 0.0005 * iteration) // Attempt with dynamic learning rate
//let learningRate = 0.01 // Attempt with non-dynamic learning rate
for (let x = 0; x < width; x += 1) {
for (let y = 0; y < height; y += 1) {
perceptron.activate([x / width, y / height])
perceptron.propagate(learningRate, getRGB(inputData, x, y))
}
}
paint()
}
const startTraining = (btn) => {
btn.disabled = true
train()
}
EDIT : I made another CodePen with even better results:
https://codepen.io/xurei/pen/KKzWLxg
It is likely to be over-fitted BTW.
The perceptron definition:
let perceptron = new synaptic.Architect.Perceptron(2, 8, 15, 7, 3)

Taking some insights from the lecture/slides of Bhiksha Raj (from slides 62 onwards), and summarizing as below:
Each node can be assumed like a linear classifier, and combination of several nodes in a single layer of neural networks can approximate any basic shapes. For example, a rectangle can be formed by 4 nodes for each lines, assuming each nodes contributes to one line, and the shape can be approximated by the final output layer.
Falling back to the summary of complex shapes such as circle, it may require infinite nodes in a layer. Or this would likely hold true for a single layer with two disjoint shapes (A non-overlapping triangle and rectangle). However, this can still be learnt using more than 1 hidden layers. Where, the 1st layer learns the basic shapes, followed by 2nd layer approximating their disjoint combinations.
Thus, you can assume that this logo is combination of disjoint rectangles (5 rectangles for orange and 3 rectangles for grey). We can use atleast 32 nodes in 1st hidden layer and few nodes in the 2nd hidden layer. However, we don't have control over what each node learns. Hence, a few more number of neurons than required neurons should be helpful.

What am I misunderstanding about this fluid dynamics calculation?

The last question was deleted for being too short and vague, so this time I will try to be as verbose as possible.
I am using Javascript to implement a basic physically accurate SPH following this paper. All I want in the beginning is to get the basics working, just to get through the first algorithm.
I start with a test case of just two particles at (0.05, 0.05) and (0.06999, 0.05). Their sample radii just touch, and I am already getting weird numbers.
This is how I calculate the densities and pressures:
for (var i = 0; i < particles.length; i++)
{
var pi = particles[i];
pi.density = 0;
for (var j = 0; j < pi.neighbors.length; j++)
{
var n = pi.neighbors[j];
var pj = n.p;
var h = (pi.sampleDist + pj.sampleDist) / 2;
n.r = subv(pi.pos, pj.pos); // subv() subtracts first vector from second
n.dist = length(n.r);
n.q = n.dist/h;
pi.density += pj.mass * W(n.q, h, 2);
}
pi.pressure = pi.stiffness * (Math.pow(pi.density / pi.restDensity, 7) - 1);
}
using the smoothing kernel
function W(q, h, d)
{
var f = 0;
if (0 <= q && q < 1) f = 2/3 - sqr(q) + 1/2*cube(q); // 2/3 - q² + 1/2q³
else if (1 <= q && q < 2) f = 1/6 * cube(2 - q); // 1/6(2-q)³
return 3/Tau / Math.pow(h, d) * f;
}
These are the settings I'm using:
mass: 0.001 kg
h: 0.01 m
rest density: 1000 kg/m³
stiffness: 0.2
In the case of just two particles with these settings, each particle's density becomes
pi.density = pj.mass * W(n.q, h, 2); = 0.001 * W(1.99899, 0.01, 2) = 0.001 * 0.00000079577 = 0.00000000079...
This number makes sense to me — there is almost nothing around, so the density is almost zero.
But now I go to calculate the pressure force, which I do like this:
Here is my implementation:
var Fp = {x:0, y:0};
if (pi.density !== 0)
{
var Vp = valueGradient(pi, 'pressure');
Fp.x = -pi.mass/pi.density * Vp.x;
Fp.y = -pi.mass/pi.density * Vp.y;
}
function valueGradient(pi, A) // A is a scalar
{
var vA = {x: 0, y: 0};
for (var j = 0; j < pi.neighbors.length; j++)
{
var n = pi.neighbors[j];
var pj = n.p;
var h = (pi.sampleDist + pj.sampleDist) / 2;
var f = pj.mass / pj.density * pj[A];
var vw = VW(n, h);
vA.x += f * vw.x;
vA.y += f * vw.y;
}
return vA;
}
function VW(n, h)
{
var dw =
dW(n.q, h, 2)
/ (n.dist * h);
return {
x: dw * n.r.x,
y: dw * n.r.y };
}
function dW(q, h, d) // derivative of kernel
{
var f = 0;
if (0 <= q && q < 1) f = -2*q + 3/2*sqr(q); // -2q + 3/2q²
else if (1 <= q && q < 2) f = -1/2 * sqr(2 - q); // -1/2(2-q)²
return 3/Tau / Math.pow(h, d) * f;
}
But something is going wrong, because when the particles sample radii just touch, the repulsive force is enormous, and when they're close it goes to zero. I get why this would be happening, because given my settings there are lots of 0.000... everywhere, and with the density so tiny, dividing by all these 0.000... will give huge numbers in the end. So it makes sense why those numbers are there, and so I suspect that I'm misunderstanding something fundamental.
This feels... backwards. What mistake am I making? I keep looking at the kernel derivative...
Another thing is that the forces are not strictly symmetrical as they should be, they tend to stay diagonal until the difference in one of the coordinates is around zero, where they suddenly flip.
I feel like I'm messing up the differential equations somewhere...
A live version of this code that can be stepped through is at www.bourt.com/particles. The code in question is in simulation.js, physics.js and kernel.js.

Animating a "Wobbly Canvas" like in Discord's Login page?

For reference, I'm talking about the dark-gray space in the upper left of Discord's Login Page. For anyone who can't access that link, here's a screenshot:
It has a number of effects that are really cool, the dots and (darker shadows) move with the mouse, but I'm more interested in the "wobbly edge" effect, and to a lesser extent the "fast wobble/scale in" on page load (scaling in the canvas on load would give a similar, if not "cheaper" effect).
Unfortunately, I can't produce much in the way of a MCVE, because I'm not really sure where to start. I tried digging through Discord's assets, but I'm not familiar enough to Webpack to be able to determine what's going on.
Everything I've been able to dig up on "animated wave/wobble" is CSS powered SVG or clip-path borders, I'd like to produce something a bit more organic.

Very interesting problem. I've scaled the blob down so it is visible in the preview below.
Here is a codepen as well at a larger size.
const SCALE = 0.25;
const TWO_PI = Math.PI * 2;
const HALF_PI = Math.PI / 2;
const canvas = document.createElement("canvas");
const c = canvas.getContext("2d");
canvas.width = window.innerWidth;
canvas.height = window.innerHeight;
document.body.appendChild(canvas);
class Blob {
constructor() {
this.wobbleIncrement = 0;
// use this to change the size of the blob
this.radius = 500;
// think of this as detail level
// number of conections in the `bezierSkin`
this.segments = 12;
this.step = HALF_PI / this.segments;
this.anchors = [];
this.radii = [];
this.thetaOff = [];
const bumpRadius = 100;
const halfBumpRadius = bumpRadius / 2;
for (let i = 0; i < this.segments + 2; i++) {
this.anchors.push(0, 0);
this.radii.push(Math.random() * bumpRadius - halfBumpRadius);
this.thetaOff.push(Math.random() * TWO_PI);
}
this.theta = 0;
this.thetaRamp = 0;
this.thetaRampDest = 12;
this.rampDamp = 25;
}
update() {
this.thetaRamp += (this.thetaRampDest - this.thetaRamp) / this.rampDamp;
this.theta += 0.03;
this.anchors = [0, this.radius];
for (let i = 0; i <= this.segments + 2; i++) {
const sine = Math.sin(this.thetaOff[i] + this.theta + this.thetaRamp);
const rad = this.radius + this.radii[i] * sine;
const theta = this.step * i;
const x = rad * Math.sin(theta);
const y = rad * Math.cos(theta);
this.anchors.push(x, y);
}
c.save();
c.translate(-10, -10);
c.scale(SCALE, SCALE);
c.fillStyle = "blue";
c.beginPath();
c.moveTo(0, 0);
bezierSkin(this.anchors, false);
c.lineTo(0, 0);
c.fill();
c.restore();
}
}
const blob = new Blob();
function loop() {
c.clearRect(0, 0, canvas.width, canvas.height);
blob.update();
window.requestAnimationFrame(loop);
}
loop();
// array of xy coords, closed boolean
function bezierSkin(bez, closed = true) {
const avg = calcAvgs(bez);
const leng = bez.length;
if (closed) {
c.moveTo(avg[0], avg[1]);
for (let i = 2; i < leng; i += 2) {
let n = i + 1;
c.quadraticCurveTo(bez[i], bez[n], avg[i], avg[n]);
}
c.quadraticCurveTo(bez[0], bez[1], avg[0], avg[1]);
} else {
c.moveTo(bez[0], bez[1]);
c.lineTo(avg[0], avg[1]);
for (let i = 2; i < leng - 2; i += 2) {
let n = i + 1;
c.quadraticCurveTo(bez[i], bez[n], avg[i], avg[n]);
}
c.lineTo(bez[leng - 2], bez[leng - 1]);
}
}
// create anchor points by averaging the control points
function calcAvgs(p) {
const avg = [];
const leng = p.length;
let prev;
for (let i = 2; i < leng; i++) {
prev = i - 2;
avg.push((p[prev] + p[i]) / 2);
}
// close
avg.push((p[0] + p[leng - 2]) / 2, (p[1] + p[leng - 1]) / 2);
return avg;
}
There are lots of things going on here. In order to create this effect you need a good working knowledge of how quadratic bezier curves are defined. Once you have that, there is an old trick that I've used many many times over the years. To generate smooth linked quadratic bezier curves, define a list of points and calculate their averages. Then use the points as control points and the new averaged points as anchor points. See the bezierSkin and calcAvgs functions.
With the ability to draw smooth bezier curves, the rest is about positioning the points in an arc and then animating them. For this we use a little math:
x = radius * sin(theta)
y = radius * cos(theta)
That converts polar to cartesian coordinates. Where theta is the angle on the circumference of a circle [0 - 2pi].
As for the animation, there is a good deal more going on here - I'll see if I have some more time this weekend to update the answer with more details and info, but hopefully this will be helpful.

The animation runs on a canvas and it is a simple bezier curve animation.
For organic feel, you should look at perlin noise, that was introduced when developing original Tron video FX.
You can find a good guide to understand perlin noise here.
In the example I've used https://github.com/josephg/noisejs
var c = $('canvas').get(0).getContext('2d');
var simplex = new SimplexNoise();
var t = 0;
function init() {
window.requestAnimationFrame(draw);
}
function draw() {
c.clearRect(0, 0, 600, 300);
c.strokeStyle="blue";
c.moveTo(100,100);
c.lineTo(300,100);
c.stroke();
// Draw a Bézier curve by using the same line cooridinates.
c.beginPath();
c.lineWidth="3";
c.strokeStyle="black";
c.moveTo(100,100);
c.bezierCurveTo((simplex.noise2D(t,t)+1)*200,(simplex.noise2D(t,t)+1)*200,(simplex.noise2D(t,t)+1)*200,0,300,100);
c.stroke();
// draw reference points
c.fillRect(100-5,100-5,10,10);
c.fillRect(200-5,200-5,10,10);
c.fillRect(200-5,0-5,10,10);
c.fillRect(300-5,100-5,10,10);
t+=0.001;
window.requestAnimationFrame(draw);
}
init();
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/simplex-noise/2.4.0/simplex-noise.js"></script>
<canvas width="600" height="300"></canvas>
Note: further investigation on Discord source code, I've pointed out that's is using https://www.npm.red/~epistemex libraries. Epistemex NPM packages are still online, while GitHub repos and profile does not exists anymore.
Note 2: Another approach could be relying on physics libraries like this demo, but it can be an overkill, if you just need a single effect.

p5.js move objects along the trajectory of polygons

I just started to learn p5 and canvas. So sorry if it's stupid question.
I've found gif online and decided to repeat this in p5.js. So I've wrote code to generate image below.
var shapes = [];
function setup() {
createCanvas(windowWidth, windowHeight);
for(var i = 1; i < 12; i++){
shapes.push(new Shape(i));
}
console.log(shapes);
}
function draw(){
background(255);
stroke('red')
for(var i = 0; i < shapes.length; i++){
shapes[i].show();
shapes[i].moveDot();
}
}
function Shape(n) {
colors = ['','red','#cd8410','#cdcb10','#8dcd10','#56cea8','#47c4cc','#479ccc','#476acc','#5d47cc','#9847cc','#b547cc','#cc47a2','#cc4760'];
this.x = width/2;
this.y = height/2;
this.vertices = n+2;
this.spaceBetween = 20;
this.edge = this.spaceBetween/(cos(PI/5)/(2*sin(TWO_PI/10))-cos(PI/4)/(2*sin(TWO_PI/8)));
this.oR = this.edge / ( 2 * sin(TWO_PI/ (2 * this.vertices) ));
this.iR = this.oR * cos(PI/this.vertices);
this.degrees = asin(this.iR / this.oR);
this.dotX = this.x;
this.dotY = this.y + this.iR;
this.dotSpeed = 3;
this.dotPCT = 0;
this.vcord = [];
for(var i = 0; i < TWO_PI; i+= TWO_PI / this.vertices){
this.vcord.push([this.x + cos(this.degrees + i) * this.oR, this.y + sin(this.degrees + i) * this.oR]);
}
this.show = ()=>{
stroke(colors[n%14]);
noFill();
beginShape();
for(var i = 0; i < this.vcord.length; i++){
vertex(this.vcord[i][0], this.vcord[i][1]);
}
endShape(CLOSE);
noStroke();
fill(0)
ellipse(this.dotX, this.dotY, 10);
}
this.moveDot = ()=>{
}
}
Now my goal is to make each dot move along trajectory of its polygon. I have access to each coordinate of polygon in this.vcord array, but I can't figure out how to make this right way.

You can use the lerp() function to get a point that's a certain percentage between two other points. More info can be found in the reference.
var xOne = 10;
var yOne = 10;
var xTwo = 100;
var yTwo = 100;
var midX = lerp(xOne, xTwo, 0.5);
var midY = lerp(yOne, yTwo, 0.5);
ellipse(midX, midY, 20, 20);
Then just modify the third value you're passing into the lerp() function to move the point between the two other points. Hint: sin() and cos() are your friends here.
If you can't get it working, I recommend breaking your problem down into smaller pieces and taking those pieces on one at a time. In other words: don't try to get it working in your full program. Instead, create a small example sketch that just does one thing. Try using the lerp() function to show a point moving between two hard-coded points. Then add a third hard-coded point. Work your way forward in small steps like that. Then if you get stuck, you can post a MCVE along with a more specific question. Good luck!
(Also, please credit the original artist if you're planning on posting your work somewhere.)

Julia Set In Javascript

I'm messing around with some Mandlebrot set stuff because I think the pictures it produces are pretty. I thought I might try to tackle the problem of drawing one in javascript to see what I could do. I looked at a couple algorithms, namely:
http://library.thinkquest.org/26242/full/progs/a2.html
Which I translated into this:
drawGraph: function(canvas,resolution,iterations,colors,coefficent){
var context = canvas.getContext('2d');
for(var m = 0; m < resolution.x; m++){
for(var n = 0; n < resolution.y; n++){
var x = m,
x2 = x*x,
y = n,
y2 = y*y;
var i;
for(i = 1; i < iterations; i++){
if(x2 + y2 > 4) break;
var new_x = x2 - y2 + coefficent.a;
var new_y = 2*x*y + coefficent.b;
x = new_x;
y = new_y;
}
var color = i % colors;
DrawUtils.drawPoint(context,m,n,color);
}
}
}
Which essentially draws a box of one color.
Then I tried this one:
http://en.wikipedia.org/wiki/Mandelbrot_set#Escape_time_algorithm
Which I translated into this:
drawGraph: function(canvas,resolution,iterations,colors,coefficent){
var context = canvas.getContext('2d');
for(var m = 0; m < resolution.x; m++){
for(var n = 0; n < resolution.y; n++){
var x = 0,
y = 0,
x0 = ((m/resolution.x) * 3.5) - 2.5,
y0 = ((n/resolution.y) * 2) - 1;
var i = 0;
while(x*x + y*y < 4 && i < iterations){
var x_temp = x*x - y*y + x0;
y = 2*x*y + y0;
x = x_temp;
i++;
}
var color = 0;
if(x*x + y*y >= 4){
color = i % colors;
}
DrawUtils.drawPoint(context,m,n,color);
}
}
}
Which produces a black box. The wording in the algorithm kind of confused me though since it said x0 and y0 scaled are factors of the pixel, but then after the algorithm, it says the coefficient c = x0 + iy0; so, does that mean I don't pass a predetermined coefficient into the function?
For most of these tests I was using the coefficient 0.25 + 0i, but I tried others that produced the exact same results.
What am I doing wrong here?

First point: you need to be clear about the difference between Julia sets and the Mandelbrot set. Both are insights into the behaviour of f(z) = z^2 + c under iteration, but from different perspectives.
For a Julia set, we fix c and make a plot of how different initial zs behave
For the Mandelbrot set, we make a plot of how the same initial z = 0 behaves for different cs.
With that addressed...
For your first code (which tries to draw the a Julia set for the c in coefficient), your translation from the BASIC in the first page you link to is not quite right. Where that has
‘ run through every point on the screen, setting
‘ m and n to the coordinates
FOR m = x_minimum TO x_maximum STEP x_resolution
FOR n = y_minimum TO y_maximum STEP y_resolution
‘ the initial z value is the current pixel,
‘ so x and y have to be set to m and n
x = m: y = n
you have
for(var m = 0; m < resolution.x; m++){
for(var n = 0; n < resolution.y; n++){
which is close, except for the crucial point that you are not taking any steps to implement STEP x_resolution. Your m is an integer that runs from 0 to resolution.x - 1 in steps of 1; and your x is set to m.
So instead of looking at the complex plane from say -2-2i to 2+2i (a decent viewport for seeing a Julia set), you are instead looking at the complex plane from 0 to resolution.x + resolution.y i, which will have at most a few pixels set in its lower-left corner.
The second code (which attempts to draw a Mandelbrot set) does have code to scale to a correct range, and I can't immediately see what's going wrong - I would debug and see if m/resolution.x is always 0, as #user973572 suggests may be the problem.

In your first example, I think you forgot to update x2 and y2 so they are always the same value. You need to update x2 and y2 before checking if the sum is greater than 4. Something like
for(i = 1; i < iterations; i++){
x2 = x*x,
y2 = y*y
if(x2 + y2 > 4) break;
which is probably wrong because I know nothing about javascript.