I'm trying to do some real-time pixel manipulation (mode7-like transformations if you are interested) with canvas.
It works so: it takes a source image data that works like a frame buffer, it does some transformations and writes the result into a another 320x240 destination image data.
function transform() {
var src_pointer, dest_pointer;
var destData = dest_context.getImageData(0,0, 320, 240);
var imgdata = destData.data;
var sourceData= source_context.getImageData(0,0,320,240).data;
for (var y = 0; y< 240; y++) for (var x = 0; x< 320; x++) {
//*** DOING SOME TRANSFORMATIONS HERE, AND WRITE THE RESULT AT SX AND SY
//... Doesn't have impact in perfomance. Suspicious, huh?
dest_pointer = getPointer(x,y);
src_pointer = getPointer(sx, sy);
imgdata[dest_pointer] = sourceData[src_pointer];
imgdata[dest_pointer +1] = sourceData[src_pointer +1];
imgdata[dest_pointer +2] = sourceData[src_pointer +2];
// Alpha? Sad thing that canvas just handle 32bit image data. I don't really need it.
imgdata[dest_pointer +3] = sourceData[src_pointer +3];
}
dest_context.putImageData(destData,0,0);
}
//Function to help map a coordinate into image data array
function getPointer(x,y) {
return ( ( y * 320 ) + x ) * 4;
}
That have a poor perfomance if you execute it continuously (about 12 frames per second). Doing some profiling I discard an specific method bottleneck (getImageData and putImageData have a really little load time).
I used to think that issue was in the transformation section but the profiling throws me that the bottle neck was specifically in the pixel assignment. Maybe optimizing math operations can be possible (think that's hard, because that's in the border line between pure javascript and browser engine), but in array assignment is possible to optimize?
Related
A client required help with a program that extracts the dominant color of a product image.
I was able to quickly implement this in Javascript; the algorithm below only samples the central square of a 3x3 grid on the image for a quick estimate of the t-shirt color in the image.
var image = new Image();
image.onload = function() {
try {
// get dominant color by sampling the central square of a 3x3 grid on image
var dominantColor = getDominantColor();
// output color
$("#output").html(dominantColor);
}
catch(e) {
$("#output").html(e);
}
};
image.src = "sample_image.jpg";
function getDominantColor() {
// Copy image to canvas
var canvas = $("<canvas/>")[0];
canvas.width = image.width;
canvas.height = image.height;
canvas.getContext("2d").drawImage(image, 0, 0);
// get pixels from the central square of a 3x3 grid
var imageData = canvas.getContext("2d").getImageData(canvas.width/3, canvas.height/3, canvas.width/3, canvas.height/3).data;
var colorOccurrences = {};
var dominantColor = "";
var dominantColorOccurrence = 0;
for(var i = 0; i < imageData.length; i += 4) {
var red = imageData[i];
var green = imageData[i+1];
var blue = imageData[i+2];
//var alpha = imageData[i+3]; // not required for this task
var color = RGBtoHEX({"red": red, "green": green, "blue": blue});
if(colorOccurrences[color] == undefined) {
colorOccurrences[color] = 1;
}
else {
colorOccurrences[color] ++;
if(colorOccurrences[color] > dominantColorOccurrence) {
dominantColorOccurrence = colorOccurrences[color];
dominantColor = color;
}
}
}
return dominantColor;
}
function RGBtoHEX(rgb) {
var hexChars = "0123456789ABCDEF";
return "#"
+ (hexChars[~~(rgb.red/16)] + hexChars[rgb.red%16])
+ (hexChars[~~(rgb.green/16)] + hexChars[rgb.green%16])
+ (hexChars[~~(rgb.blue/16)] + hexChars[rgb.blue%16]);
}
The image in question is this (preview below).
However, the results when this image is processed in the code above are varied across machines/browsers: #FF635E is what I see on my machine, running Windows7 and using Firefox 32. My client running Mac gets a result of #FF474B on Safari and #FF474C on Firefox 33.
Though the results are close, why are they ideally not the exact same? Does getImageData indeed vary depending on the local setup, or is the JPG data being interpreted differently on different machines?
Edit: This image isn't a one-off case. Such color variations were noticed across a range of the image that the client requested to process. My client and I obtained different results for the same set of images.
Yes. This fact is exploited by canvas fingerprinting:
The same HTML5 Canvas element can
produce exceptional pixels on a different web browsers, depending on
the system on which it was executed.
This happens for several reasons: at the image format level — web
browsers uses different image processing engines, export options,
compression level, final images may got different hashes even if they
are pixel-perfect; at the pixmap level — operating systems use
different algorithms and settings for anti-aliasing and sub-pixel
rendering. We don't know all the reasons, but we have already
collected more than a thousand unique signatures.
As part of a project, I have to render a video on a JS-player from a text file which has the points - all the changed coordinates along-with the color in each frame. Below is the code I'm using to draw these point on the screen.
But the issue is that the number of changed pixels in each frame are as high as ~20,000 and I need to display these in less than 30ms (inter-frame time difference). So, when I run this code the browser hangs for almost each frame. Could someone suggest an improvement for this?
Any help is really appreciated.
c.drawImage(img,0,0,800,800);
setInterval(
function(){
while(tArr[index]==time) {
var my_imageData = c.getImageData(0,0,width, height);
color(my_imageData,Math.round(yArr[index]),Math.round(xArr[index]),Math.round(iArr[index]),255);
c.putImageData(my_imageData,0,0);
index=index+1;
}
time = tArr[index];
}
,30);
xArr, yArr, iArr, tArr are arrays of x-coordinate, y-coordinate, intensity value and time of appearance respectively for the corresponding point to be rendered
function color(imageData,x,y,i,a){ //wrapper function to color the point
var index = (x + y * imageData.width) * 4;
imageData.data[index+0] = i;
imageData.data[index+1] = i;
imageData.data[index+2] = i;
imageData.data[index+3] = a;
}
A client required help with a program that extracts the dominant color of a product image.
I was able to quickly implement this in Javascript; the algorithm below only samples the central square of a 3x3 grid on the image for a quick estimate of the t-shirt color in the image.
var image = new Image();
image.onload = function() {
try {
// get dominant color by sampling the central square of a 3x3 grid on image
var dominantColor = getDominantColor();
// output color
$("#output").html(dominantColor);
}
catch(e) {
$("#output").html(e);
}
};
image.src = "sample_image.jpg";
function getDominantColor() {
// Copy image to canvas
var canvas = $("<canvas/>")[0];
canvas.width = image.width;
canvas.height = image.height;
canvas.getContext("2d").drawImage(image, 0, 0);
// get pixels from the central square of a 3x3 grid
var imageData = canvas.getContext("2d").getImageData(canvas.width/3, canvas.height/3, canvas.width/3, canvas.height/3).data;
var colorOccurrences = {};
var dominantColor = "";
var dominantColorOccurrence = 0;
for(var i = 0; i < imageData.length; i += 4) {
var red = imageData[i];
var green = imageData[i+1];
var blue = imageData[i+2];
//var alpha = imageData[i+3]; // not required for this task
var color = RGBtoHEX({"red": red, "green": green, "blue": blue});
if(colorOccurrences[color] == undefined) {
colorOccurrences[color] = 1;
}
else {
colorOccurrences[color] ++;
if(colorOccurrences[color] > dominantColorOccurrence) {
dominantColorOccurrence = colorOccurrences[color];
dominantColor = color;
}
}
}
return dominantColor;
}
function RGBtoHEX(rgb) {
var hexChars = "0123456789ABCDEF";
return "#"
+ (hexChars[~~(rgb.red/16)] + hexChars[rgb.red%16])
+ (hexChars[~~(rgb.green/16)] + hexChars[rgb.green%16])
+ (hexChars[~~(rgb.blue/16)] + hexChars[rgb.blue%16]);
}
The image in question is this (preview below).
However, the results when this image is processed in the code above are varied across machines/browsers: #FF635E is what I see on my machine, running Windows7 and using Firefox 32. My client running Mac gets a result of #FF474B on Safari and #FF474C on Firefox 33.
Though the results are close, why are they ideally not the exact same? Does getImageData indeed vary depending on the local setup, or is the JPG data being interpreted differently on different machines?
Edit: This image isn't a one-off case. Such color variations were noticed across a range of the image that the client requested to process. My client and I obtained different results for the same set of images.
Yes. This fact is exploited by canvas fingerprinting:
The same HTML5 Canvas element can
produce exceptional pixels on a different web browsers, depending on
the system on which it was executed.
This happens for several reasons: at the image format level — web
browsers uses different image processing engines, export options,
compression level, final images may got different hashes even if they
are pixel-perfect; at the pixmap level — operating systems use
different algorithms and settings for anti-aliasing and sub-pixel
rendering. We don't know all the reasons, but we have already
collected more than a thousand unique signatures.
I am working on a 2D maze game with a torch effect in canvas without the use of any raytracing. Everything is working great, however the torch effect's algorithm is causing immense lags in several browsers and computers. (It is weird as well, that the game runs smoother on older computers. The funniest is, IExplorer runs the game without any lags, while mozzila dies on every move..)
My general idea for solving this problem was, to get how far the character is from the walls (4 functions) and make the rest of the maze grey.
Here is an example with the Northern wall detection:
http://webprogramozas.inf.elte.hu/~ce0ta3/beadando/maze_example.png
And an example how it is working at the moment and what I would like to achieve without lag issues.
http://webprogramozas.inf.elte.hu/~ce0ta3/beadando/ce0ta3_html5_maze.html
As I mentioned above, the algorithm that tracks the character's distance from the walls is causing incredible lags.
//Get the character's X,Y position as parameter
function distanceFromNorth (posX,posY)
{
distNorth = 0;
var l = false;
//Start getting charSize x 1 px lines from the character position towards the up, until we reach the max viewDistance or we find a black pixel in the charSize x 1 line.
for (var i = posY; i > posY - viewDistance && !l; i--)
{
var mazeWallData = context.getImageData(posX, i, charSize, 1);
var data = mazeWallData.data;
//Check if there are any black pixels in the line
for (var j = 0; j < 4 * charSize && !l; j += 4)
{
l = (data[j] === 0 && data[j + 1] === 0 && data[j + 2] === 0);
}
distNorth++;
}
return distNorth;
}
I am fairly sure, that the ctx.getImageData() is the most costly method in this linear search and if I only requested this method once for a charSize x viewDistance rectangle, and then check for black pixels in that huge array, then the lag could be reduced greatly. However, I still want to keep searching in lines, because finding only one black pixel will return false distNorth value.
I would be grateful if anyone could convert my code into the form I mentioned in the previous paragraph.
Assuming the image data isnt changing then you can precompute all the pixel values that have black pixel. Then use simple binary search on it to get the any black pixels in the given range.
Algorithm : -
cols[];
rows[];
for(int i=0;i<height;i++) {
for(int j=0;j<width;j++) {
if(pixel(j,i)==black) {
row[i].add(j);
col[j].add(i);
}
}
}
for query on (x,y) :
distance = binarysearch(col[x],y,y-distance) - y
I'm trying to implement a simple bilateral filter in javascript. This is what I've come up with so far:
// For each pixel
for (var y = kernelSize; y < height-kernelSize; y++) {
for (var x = kernelSize; x < width-kernelSize; x++) {
var pixel = (y*width + x)*4;
var sumWeight = 0;
outputData[pixel] = 0;
outputData[pixel+1] = 0;
outputData[pixel+2] = 0;
outputData[pixel+3] = inputData[pixel+3];
// For each neighbouring pixel
for(var i=-kernelSize; i<=kernelSize; i++) {
for(var j=-kernelSize; j<=kernelSize; j++) {
var kernel = ((y+i)*width+x+j)*4;
var dist = Math.sqrt(i*i+j*j);
var colourDist = Math.sqrt((inputData[kernel]-inputData[pixel])*(inputData[kernel]-inputData[pixel])+
(inputData[kernel+1]-inputData[pixel+1])*(inputData[kernel+1]-inputData[pixel+1])+
(inputData[kernel+2]-inputData[pixel+2])*(inputData[kernel+2]-inputData[pixel+2]));
var curWeight = 1/(Math.exp(dist*dist/72)*Math.exp(colourDist*colourDist*8));
sumWeight += curWeight;
outputData[pixel] += curWeight*inputData[pixel];
outputData[pixel+1] += curWeight*inputData[pixel+1];
outputData[pixel+2] += curWeight*inputData[pixel+2];
}
}
outputData[pixel] /= sumWeight;
outputData[pixel+1] /= sumWeight;
outputData[pixel+2] /= sumWeight;
}
}
inputData is from a html5 canvas object and is in the form of rgba.
My images are either coming up with no changes or with patches of black around edges depending on how i change this formula:
var curWeight = 1/(Math.exp(dist*dist/72)*Math.exp(colourDist*colourDist*8));
Unfortunately I'm still new to html/javascript and image vision algorithms and my search have come up with no answers. My guess is there is something wrong with the way curWeight is calculated. What am I doing wrong here? Should I have converted the input image to CIElab/hsv first?
I'm no Javasript expert: Are the RGB values 0..255? If so, Math.exp(colourDist*colourDist*8) will yield extremely large values - you'll probably want to scale colourDist to the range [0..1].
BTW: Why do you calculate the sqrt of dist and colourDist if you only need the squared distance afterwards?
First of all, your images turn out black/weird in the edges because you don't filter the edges. A short look at your code would show that you begin at (kernelSize,kernelSize) and finish at (width-kernelSize,height-kernelSize) - this means that you only filter a smaller rectangle inside the image where your have a margin of kernelSize on each side which is unfilterred. Without knowing your javscript/html5, I would assume that your outputData array is initialized with zero's (which means black) and then not touching them would leave them black. See my link the comment to your post for code that does handle the edges.
Other than that, follow #nikie's answer - your probably want to make sure the color distance is clamped to the range of [0,1] - youo can do this by adding the line colourDist = colourDist / (MAX_COMP * Math,sqrt(3)) (directly after the first line to calculate it). where MAX_COMP is the maximal value a color component in the image can have (usually 255)
I've found the error in the code. The problem was I was adding each pixel to itself instead of its surrounding neighbours. I'll leave the corrected code here in case anyone needs a bilateral filter algorithm.
outputData[pixel] += curWeight*inputData[kernel];
outputData[pixel+1] += curWeight*inputData[kernel+1];
outputData[pixel+2] += curWeight*inputData[kernel+2];