I am retrieving pixels in a canvas imagedata and I'm doing that a lot.
I think the inserting and retrieving from and to the canvas imagedata is expensive in cpu time, so I want to make as few of those as possible.
One way of cutting that would be to make a single insert that would insert multiple pixels in a single sequence, but so far I have not been able to see how that would be done. All the examples I have seen so far retrieve and insert only a single pixel.
So the question is,
in order to speed up canvas imagedata pixel manipulation, how do I insert/retrieve multiple pixels simultaneously?
Just select a larger region when retrieving a pixel buffer:
var imageData = ctx.getImageData(x, y, width, height);
^^^^^^^^^^^^ not limited to one
Now your data buffer will contain all pixels for the given region. To get the whole canvas:
var imageData = ctx.getImageData(0, 0, ctx.canvas.width, ctx.canvas.height);
Adjust them and put back into the same position:
ctx.putImageData(imageData, x, y);
and you're done.
Remember that each pixel consists of four bytes (RGBA). To address a larger buffer you can do:
function getPixelIndex(x, y) {
return (y * width + x) * 4; // width used when getting buffer
}
Tips:
if you plan to update the same buffer often simply retrieve the buffer once and store a pointer to it, update it when you need and put back, then reuse the same buffer. This way you save the time getting the buffer. This won´t work if you in the mean time apply graphics to the canvas with standard methods.
You can also start with an empty buffer using createImageData() instead of getImageData().
If your pixel color data is more or less static you can update the buffer using a Uint32Array instead of the Uint8ClampedArray. You get a 32-bit version like this after getting the imageData:
var buffer32 = new Uint32Array(imageData.data.buffer);
Your new buffer32 will point to the same underlying byte buffer so no significant memory overhead, but it allows you to read and write 32-bit values instead of just 8-bit. Just be aware of that the byte order is (typically) little-endian so order the bytes as ABGR. Then do as before, call ctx.putImageData(imageData, x, y); when you need to update.
Related
I have an ArrayBuffer (representing a PNG/JPG) and I want to create a dictionary so I know for each group of bytes what pixels and at what positions were generated on a canvas. The function that I need would look something like this:
getPixelsCoordinatesForBytes(arrayBuffer, bufferStart, bufferEnd){
// crate Image on Canvas from buffer,
// get location in canvas image for
// the pixels that were generated using the bytes beetween bufferStart and bufferEnd
return x,y, img.width, img.height
}
Basically, I need to reverse engineer the getImageData from Canvas context, tried looking into how V8 implements it but the code is a mess. From what I understand each file type is converted to ImageData based on some standard algorithm (also known as bitmaps/raster graphs). I was not able to find how this algorithm is working in javascript.
A solution in any programming language provided to this problem would also be useful as I may be able to rewrite it in Javascript.
I am writing a data viz app that requires me to process very large 2D arrays of data and convert that data into a scaled down image for display in a canvas in the DOM.
I am bumping up against DOM canvas size limitations. My arrays can be as large as 5000 x 5000. I want to get around the canvas size limitation by using createImageBitmap() to simultaneously scale down and convert the large array to an ImageBitMap of smaller size - 256 x 256 - for insertion into an onscreen canvas.
How can I convert the raw array data into the proper format? Will this approach work?
You can create and manipulate your image before rendering it to canvas. 5000x5000 shouldn't be too large for canvas though. What limitations are you running into? Answer here covers resizing as on canvas then grabbing data.
var raw = new Uint8ClampedArray(5000*5000*4); // 4 for RBGA
var imageData = new ImageData(raw, 5000,5000);
var bitmap = createImageBitmap(imageData);
I am streaming video over a WebSocket by sending each frame in the raw ImageData format (4 bytes per pixel in RGBA order). When I receive each frame on the client (as an ArrayBuffer), I want to paint this image directly onto the canvas as efficiently as possible, using putImageData.
This is my current solution:
// buffer is an ArrayBuffer representing a properly-formatted image
var array = new Uint8ClampedArray(buffer);
var image = new ImageData(array, width, height);
canvas.putImageData(image, 0, 0);
But it is rather slow. My theories as to why:
the array (which is ~1MB in size) is being copied thrice, once into the Uint8ClampedArray, once into the ImageData, and lastly into the canvas, each frame (30 times per second).
I am using new twice for each frame, which may be a problem for the garbage collector.
Are these theories correct and if so, what tricks can I employ to make this as fast as possible? I am willing to accept an answer that is browser-specific.
No, both your ImageData image and your TypedArray array share the exact same buffer buffer.
These are just pointers, your original buffer is never "copied".
var ctx = document.createElement('canvas').getContext('2d');
var buffer = ctx.getImageData(0,0,ctx.canvas.width, ctx.canvas.height).data.buffer;
var array = new Uint8ClampedArray(buffer);
var image = new ImageData(array, ctx.canvas.width, ctx.canvas.height);
console.log(array.buffer === buffer && image.data.buffer === buffer);
For your processing time issue, the best way would be to simply send directly the video stream to a videoElement and use drawImage.
I have a series of textures which stack together (using a mip-map-style resolution pyramid) to arrive at a final image.
Because of the way they stack, it's necessary to initialize them to an unsigned int value of 128 when they have not been populated by meaningful image data. IE, 128 is zero because the render shader will subtract .5 from each texture value, which allows subsequent pyramidal layers to offset the final image by positive or negative values.
I cannot seem to figure out how to initialize a (single-channel GL_LUMINANCE) texture to a value!
I've tried setting it as a renderbuffer target and rendering polys to it, but the FBO seems to be marked as incomplete. I've also tried targeting it as a renderbuffer target and using gl.clear(gl.COLOR_BUFFER_BIT) but again it's considered incomplete.
The obvious thing would be to copy values in using gl.texSubImage2D() but that seems really slow... maybe that's the only way? I was hoping for something more elegant that doesn't require allocating so much memory (at least a frame's worth of a single value) and so slow (because all that data must be written to the buffer when it's all a single value).
The only way to set a texture to a (default) value in WebGL seems to be to resize it, or allocate it, which sets it to all zeroes.
Also, there doesn't seem to be a copy mode like gl.SIGNED_BYTE which would allow zero to be zero (IE, signed values coming in)... but this also doesn't solve the problem of initializing the texture to a single value (in this case, zero).
Any ideas? How does one initialize a WebGL texture to a value aside from just plain old copying the value into it?
Being able to render to a particular type of texture is unfortunately not guaranteed by the OpenGL ES 2.0 spec on which WebGL is based on. The only way to tell if it works is to create the texture, attach it to a framebuffer and then call checkFramebufferStatus and see if it returns FRAMEBUFFER_COMPLETE. Unfortunately it won't in a lot of cases.
Only 3 combinations off attachments are guaranteed to work
COLOR_ATTACHMENT0 = RGBA/UNSIGNED_BYTE texture
COLOR_ATTACHMENT0 = RGBA/UNSIGNED_BYTE texture + DEPTH_ATTACHMENT = DEPTH_COMPONENT16 renderbuffer
COLOR_ATTACHMENT0 = RGBA/UNSIGNED_BYTE texture + DEPTH_STENCIL_ATTACHMENT = DEPTH_STENCIL renderbuffer
So your options are
use an RGBA texture and render 128,128,128,??? to it in an fbo (or gl.clear it)
use a LUMINANCE texture and call texImage2D
use a LUMINANCE texture and call copyTexImage2D using the backbuffer or an fbo of the correct size cleared to the color you want (though there is no guarantee this is fast AFAIK)
In my experience (mainly with Chrome in OSX) using Canvas to initialise a texture is fast. I guess it is because the browser allocates the Canvas and draws to it all on the GPU, and its WebGL implementation uses the same GL context as Canvas, so there is no huge CPU-to-GPU memory transfer.
// Quickly init texture to (128,128,128)
var canvas = this._canvas = document.createElement("canvas");
canvas.width = wTex;
canvas.height = hTex;
var ctx = this._ctx = canvas.getContext("2d");
ctx.fillStyle = "#808080";
ctx.fillRect(0, 0, wTex, hTex);
gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGB, gl.RGB, gl.UNSIGNED_BYTE, canvas);
Is it possible to read pixels of an image in canvas A and create pixels on canvas B? And I want to create the new pixels on Canvas B only where the image's pixels are green.
eg. If images' pixel (120,45) is green I need to create a green colored pixel in Canvas B at (120,45)
You can use canvas ImageData to get color values for each pixels. The function:
context.getImageData(left, top, width, height);
returns an ImageData object, which consists of the following properties:
width
height
data (CanvasPixelArray)
The CanvasPixelArray contains the RGBA values for all the pixels, starting from top-left working its way to bottom-right. So in other words, it is an array containing 4*width*height number of entries.
Using that, you can start looping through each pixel (+=4 entries per pixel), and check the RGBA values. If they match a specific value you want, i.e. green, then you would copy that value to a canvas B, which you would create by modifying a newly created ImageData object.
You can create a new empty ImageData object with:
context.createImageData(cssWidth, cssHeight)
Note that you will need to know specific RGBA values that identify "green" or define specific conditions, i.e. if the G value of RGBA is >= 150 then it is "green".
Also note that you cannot get the ImageData that has been tainted resources outside of your origin. i.e., if you draw an image onto the canvas that isn't CORS safe, you won't be able to retrieve the ImageData from that canvas anymore, much like how you can't use toDataURL either.