I have an ArrayBuffer (representing a PNG/JPG) and I want to create a dictionary so I know for each group of bytes what pixels and at what positions were generated on a canvas. The function that I need would look something like this:
getPixelsCoordinatesForBytes(arrayBuffer, bufferStart, bufferEnd){
// crate Image on Canvas from buffer,
// get location in canvas image for
// the pixels that were generated using the bytes beetween bufferStart and bufferEnd
return x,y, img.width, img.height
}
Basically, I need to reverse engineer the getImageData from Canvas context, tried looking into how V8 implements it but the code is a mess. From what I understand each file type is converted to ImageData based on some standard algorithm (also known as bitmaps/raster graphs). I was not able to find how this algorithm is working in javascript.
A solution in any programming language provided to this problem would also be useful as I may be able to rewrite it in Javascript.
Related
I am writing a data viz app that requires me to process very large 2D arrays of data and convert that data into a scaled down image for display in a canvas in the DOM.
I am bumping up against DOM canvas size limitations. My arrays can be as large as 5000 x 5000. I want to get around the canvas size limitation by using createImageBitmap() to simultaneously scale down and convert the large array to an ImageBitMap of smaller size - 256 x 256 - for insertion into an onscreen canvas.
How can I convert the raw array data into the proper format? Will this approach work?
You can create and manipulate your image before rendering it to canvas. 5000x5000 shouldn't be too large for canvas though. What limitations are you running into? Answer here covers resizing as on canvas then grabbing data.
var raw = new Uint8ClampedArray(5000*5000*4); // 4 for RBGA
var imageData = new ImageData(raw, 5000,5000);
var bitmap = createImageBitmap(imageData);
I have a PNG file generated on the server that is 1536 by 47616 Pixels. The PNG is paletted, and has 20 entries. The size of the PNG is 2.5MB - i dont see why memory can be an issue for the GPU.
This big image file basically contains a combination of a lot of small textures. Each of the texture inside this big file, is 128px width by 512px height each. Meaning that basically this big image contains 3*372 = 1116 images (1536/512 = 3 image every row, 47616/128 = 372 every column)
I put them in the same image because otherwise the end user will need to request 1116 images seperately, and with requesting and processing headers - this is a lot of images to process. Much faster to request a single massive image.
gl.getParameter(gl.MAX_TEXTURE_SIZE); // this gives me 16384 for my browser. My texture's height of 47616 is a lot higher than this.
Currently i try to bind the texture to the buffer as:
this.texture_image = new Image();
var parent = this;
this.texture_image.addEventListener("load", function(e) {
parent.tinfo.width = parent.texture_image.width;
parent.tinfo.height = parent.texture_image.height;
parent.gl.bindTexture(parent.gl.TEXTURE_2D, parent.tinfo.texture);
parent.gl.texImage2D(parent.gl.TEXTURE_2D, 0, parent.gl.RGB, parent.gl.RGB, parent.gl.UNSIGNED_BYTE, parent.texture_image);
});
this.texture_image.src= texture_image_url;
However this results in Google Chrome complaining:
main.js:5118 WebGL: INVALID_VALUE: texImage2D: width or height out of range
Which is obviously the case, since my height is way outside the MAX_TEXTURE_SIZE range.
So now, is there someway on the client that i can refer to smaller parts of the image? Maybe make my own smaller Image objects out of the large this.texture_image Image object?
I guess i can use Canvas and drawImage, but would prefer a WebGL based solution. Since i will be doing some other effects on it with WebGL later on.
Thanks.
The size of the PNG is 2.5MB - i dont see why memory can be an issue
for the GPU
PNG is a compressed file format not optimized for random realtime access, GPUs don't support encodings like that, without the use of any extensions GPUs only support raw pixel data thus your image will be expanded to WIDTHxHEIGHTx4 => 1536x47616x4 = 292552704 Bytes which are a hefty 292.553 Megabyte, even with the use of extensions you would be bound to fixed width block encoding schemes. That being said MAX_TEXTURE_SIZE is not so much about memory but the addressability of it, there is no way around that(in WebGL), your only option is to use a 2D canvas to split your texture atlas into suitable chunks.
I'm still relatively new to working with the canvas tag. What I've done so far is draw an image to the canvas. My goal is to have a fake night/day animation that cycles repeatedly.
I've exhausted quite a few different avenues (SVG, CSS3 filters, etc) and think that canvas pixel manipulation is the best route in my case. I'm trying to:
Loop through all pixels in the image
Select a certain color range
Adjust to new color
Update the canvas
Here's the code I have so far:
function gameLoop(){
requestAnimationFrame(gameLoop);
////////////////////////////////////////////////////////////////
// LOOP PIXEL DATA - PIXEL'S RGBA IS STORED IN SEQUENTIAL ARRAYS
////////////////////////////////////////////////////////////////
for(var i=0; i<data.length; i+=4){
red=data[i+0];
green=data[i+1];
blue=data[i+2];
alpha=data[i+3];
// GET HUE BY CONVERTING TO HSL
var hsl=rgbToHsl(red, green, blue);
var hue=hsl.h*360;
// CHANGE SET COLORRANGE TO NEW COLORSHIFT
if(hue>colorRangeStart && hue<colorRangeEnd){
var newRgb=hslToRgb(hsl.h+colorShift, hsl.s, hsl.l);
data[i+0]=newRgb.r;
data[i+1]=newRgb.g;
data[i+2]=newRgb.b;
data[i+3]=255;
};
};
// UPDATE CANVAS
ctx.putImageData(imgData, 0, 0);
};
The code works and selects a hue ranges and shifts it once, but is incredibly laggy. The canvas dimensions are roughly 500x1024.
My questions:
Is it possible to improve performance?
Is there a better way to perform a defined hue shift animation?
Thanks!
It's hard to do this real-time using high quality HSL conversion. Been there done that, so I came up with a quantized approach which allow you to do this in real-time.
You can find the solution here (GPL3.0 licensed):
https://github.com/epistemex/FastHSL2RGB
Example of usage can be found here (MIT license) incl. demo:
https://github.com/epistemex/HueWheel
Apologies for referencing my own solutions here, but the inner workings (the how to's) is too extensive to present in a simple form here and both of these are free to use for anything..
The key points are in any case:
Quantize the range you want to use (don't use full 360 degrees and not floating points for lightness etc.)
Cache the values in a 3D array (initial setup using web workers or use rough values)
Quantize the input values so they fit in the range of the inner 3D array
Process the bitmap using these values
It is not accurate but good enough for animations (or previews which is what I wrote it for).
There are other techniques such as pre-caching the complete processed bitmap for key positions, then interpolate the colors between those instead. This, of course, requires much more memory but is a fast way.
Hope this helps!
I am retrieving pixels in a canvas imagedata and I'm doing that a lot.
I think the inserting and retrieving from and to the canvas imagedata is expensive in cpu time, so I want to make as few of those as possible.
One way of cutting that would be to make a single insert that would insert multiple pixels in a single sequence, but so far I have not been able to see how that would be done. All the examples I have seen so far retrieve and insert only a single pixel.
So the question is,
in order to speed up canvas imagedata pixel manipulation, how do I insert/retrieve multiple pixels simultaneously?
Just select a larger region when retrieving a pixel buffer:
var imageData = ctx.getImageData(x, y, width, height);
^^^^^^^^^^^^ not limited to one
Now your data buffer will contain all pixels for the given region. To get the whole canvas:
var imageData = ctx.getImageData(0, 0, ctx.canvas.width, ctx.canvas.height);
Adjust them and put back into the same position:
ctx.putImageData(imageData, x, y);
and you're done.
Remember that each pixel consists of four bytes (RGBA). To address a larger buffer you can do:
function getPixelIndex(x, y) {
return (y * width + x) * 4; // width used when getting buffer
}
Tips:
if you plan to update the same buffer often simply retrieve the buffer once and store a pointer to it, update it when you need and put back, then reuse the same buffer. This way you save the time getting the buffer. This won´t work if you in the mean time apply graphics to the canvas with standard methods.
You can also start with an empty buffer using createImageData() instead of getImageData().
If your pixel color data is more or less static you can update the buffer using a Uint32Array instead of the Uint8ClampedArray. You get a 32-bit version like this after getting the imageData:
var buffer32 = new Uint32Array(imageData.data.buffer);
Your new buffer32 will point to the same underlying byte buffer so no significant memory overhead, but it allows you to read and write 32-bit values instead of just 8-bit. Just be aware of that the byte order is (typically) little-endian so order the bytes as ABGR. Then do as before, call ctx.putImageData(imageData, x, y); when you need to update.
I have bytearray returned from ActionScript to Javascript through ExternalInterface call. Now, i have to convert this byteaarray to image in Javascript code. pls help...with any sample code...
Thanks in advance...
I see two possible solutions to this problem, neither of which I have tested so try them out:
HTML5 Canvas
First, using ActionScript convert your byte array to integer array. You will need four values for:
Red
Green
Blue
Alpha
Transfer this to Javascript, either using string representation or plain numbers and then load these numbers into the canvas:
var canvasData = ; // data from actionscript
var c=document.getElementById("myCanvas");
var ctx=c.getContext("2d");
var imgData=ctx.createImageData(100,100);
for (var i=0;i<imgData.width*imgData.height*4;i+=4)
{
imgData.data[i+0]=canvasData[i][0]; // red
imgData.data[i+1]=canvasData[i][1]; // green
imgData.data[i+2]=canvasData[i][2]; // blue
imgData.data[i+3]=canvasData[i][3]; // alpha
}
ctx.putImageData(imgData,10,10);
Base64-encoded through CSS
If you don't want to rely on HTML5, use ActionScript to convert the byte array to a base64 string and then insert the image using the following css rule:
background-image: url(data:image/png;base64,__base64_data__);
and replace __base64_data__ with the generated string. This could be done dynamically using JQuery:
$('#img').css("background-image", "url(data:image/png;base64,__base64_data__)");
This also seems to be a much more efficient method than HTML5 Canvas, although actual performance will depend on the image size.
You can store your data in an ArrayBuffer and inject this into a canvas to display as explained here :
https://hacks.mozilla.org/2011/12/faster-canvas-pixel-manipulation-with-typed-arrays/