I'm trying to download 16-bit image data from a server and push it into a WebGL texture without browser plug-ins. texImage2d will work with: ImageData, HTMLImageElement, HTMLCanvasElement, or HTMLVideoElement. I'm looking for some javascript (a library or code sample) which can decode 16-bit TIFF or similar (hdf5, etc.) image data into one of these object types.
I have no problem doing this is 8-bit per channel RGB by using an to load a PNG but this doesn't work with 16-bit per channel data since there aren't any "standard" browser supported image formats which are 16-bit.
In case of combining two PNG images, one with the top 8 bits and the second with the low 8 bits, I think it should be:
highp vec4 texCol = texture2D(tex_low, vec2(vTexCoord.s, vTexCoord.t)) * (1.0 / 257.0);
texCol += texture2D(tex_up, vec2(vTexCoord.s, vTexCoord.t)) * (256.0 / 257.0);
In 8 bits per channel RGB colors will range from 0 to 255 = 2^8 - 1.
In 16 bits per channel RGB colors will range from 0 to 65535 = 2^16 - 1 = 255*257.
Explanation
WebGL works using colour values from 0 to 1 and makes it by dividing 8 bit color value by 255. So the divided value belongs to the range <0,1>.
In case of 16 bit per channel we would like to divide it by 65535 to get the proper number from range <0,1>.
What we want is 16 bit color value reduced to range <0,1>.
Let low and up be color value from range 0..255. up is top 8 bits and low is low 8 bits.
To get 16 bit value we can compute: low + up*256. Now we have number in range 0..65535. To get value from range <0,1> we divide it by 65535. Note that WebGL works using color values from range <0,1> , it is Lw=low/255 and Uw=up/255. So, we don't have to multiply it by 255 and divide it by 65535 because 65535 = 255*257. Instead we just divide by 257.
Also I could not find any software to split 16 bit / channel image into two 8 bit/channel image, so here is my code, feel free to use it, it splits 16 bit / channel Tiff into two 8 bit/channel PNGs:
https://github.com/czero69/ImageSplitter
PNGToy is a pretty featured library for extracting PNG chunks of almost all depths and channel modes with javascript (really client-side / without node.js, just Promise.js dependencies). The decode method will return the desired buffer. Here is an example for 16 bits grayscale PNG (16 bits RGB should work as well) :
var dataObj;
var img = new PngImage();
var buffer;
img.onload = function() {
var pngtoy = this.pngtoy;
dataObj = pngtoy.decode().then(function(results) {
buffer = new Uint16Array(results.bitmap);
for(var i = 0, j; i < buffer.length; i++) {
j = buffer[i];
buffer[i] = ((j & 0xff) << 8) | ((j & 0xff00) >>> 8); // needed to swap bytes for correct unsigned integer values
}
console.log(buffer);
});
};
img.onerror = function(e) {
console.log(e.message);
};
img.src = "image.png";
I don't think the main browsers natively support any 16-bit/channel image format at the moment.
One way to achieve the same effect would be to create two PNG images, one with the top 8 bits of each colour channel in the image and one with the bottom 8 bits.
Then bind the images as two textures and combine the values in your shader, e.g.
highp float val = texture2d(samplerTop8bits, tex_coord) * (256.0 / 257.0);
val += texture2d(samplerBottom8bits, tex_coord) * (1.0 / 257.0);
(Note: you need highp precision to represent your data correctly in a 16-bit range)
Another method is only possible if floating point textures are supported in your target browser(s). You would, in the browser, combine the two PNG images into a floating point texture then access that normally. This may not be any faster and will probably use twice the amount of texture memory.
Related
I have some data in a Uint8array, ( a list of 3d x,y,z coordinates). I need to perform some floating point operations on. (matrix rotation transform multiplication) on this 3d vectors. I am using the http://glmatrix.net/ library for the calculations.
The data is coming from a 3rd party API that packages it as Uint8Array The vertex data is formatted in sets of 8 [x,y,z,w,r,g,b,a]
Up until now, I have been performing this operation on the GPU, passing this Uint8array and the rotation matrix to GLSL. I am moving the operation to a CPU based script that is executed offline.
Update Another piece of the equation. The offline script is a node.js CLI script that writes the final buffer to stdout and eventually to a file using the unix > operator. I then load this in the final app and wrap the arrayBuffer with a typedArray constructor.
When executed in GLSL, the operation looks like:
matrix * vec4(vert.x, vert.y, vert.z, 1.0)
and my results look like:
Now on the CPU My current code attempt is:
const vertData = getVertDataUint8();
const rotation = mat4.fromRotation(mat4.identity([]), -Math.PI/2, [1,0,0]);
var i = 0;
while(i<vertData.length){
const rotatedVerts = vec4.transformMat4([],
vec4.clone([vertData[i],vertData[i+1],vertData[i+2], 1]),
rotation);
vertData[i] = rotatedVerts[0];
vertData[i+1] = rotatedVerts[1];
vertData[i+2] = rotatedVerts[2];
i+=8;
}
and my results look like:
I noticed that when re assigning the values to the vertData, specifically negative values they were being cast as ints. example -10 ~= 246
you can reproduce in a browser console with:
var u = new Uint8Array(1)
undefined
u[0] = -10
-10
u[0]
246
I have tried converting the Uint8Array into a Float32Array by wrapping it in the Float32Array constructor and using Float32Array.from but it seems to produce the same results.
First of all: Consider using the Float32Array for your vertex information. Otherwise there can be massive problems. Or at least use non unsigned data formats.
Now your Problem:
Since Uint8Array is a unsigned data format it will only save positive 8-bit values. All negative values will be interpreted as positives (10000000 as signed integer is -128 but as unsigned it is 128)
But i guess the real problem is another one.
From the code i assume that you are saving the vertex information like this:
[X1,Y1,Z1,...] or [X1,Y1,Z1,W1,...]
Your loop will iterate over the array in steps of 8 since i+=8;.
Therefore at least every second vertex will be skipped in your script.
Assume I can bare the loss of digits/precision to some degree. I find that to send a 64bit (8 bytes) number over the network sometimes is overkilled. I want the data to use less bandwidth but maintain certain accuracy. But I don't know the correct way to store a number in 32 bit or 16 bit data in javascript.
Here's how you can convert a JavaScript number to an array buffer holding a 32 bit float or a 16 bit unsigned integer:
let float64 = 3.141592653589793238462643383279502884197;
let float32View = new DataView(new ArrayBuffer(4));
float32View.setFloat32(0, float64);
let uint16View = new DataView(new ArrayBuffer(2));
uint16View.setUint16(0, float64);
console.log(float64);
console.log(float32View.getFloat32(0));
console.log(uint16View.getUint16(0));
in javascript numbers are always allocated as double precision floats. This is fine if you aren't sending huge amounts of these as binary without compression, or don't need to conserve memory. If you need to make these numbers smaller how do you do so?
The obvious goal would be to store numbers into the smallest possible byte size, for example 208 : 1 byte, 504 : 2 bytes. Even better would be smallest number of bit size, for example 208 : 8 bits, 504 : 9 bits.
example:
//myNetwork is a supposed network API that sends as binary
var x = 208;
myNetwork.send(x); // sends 01000011010100000000000000000000
myNetwork.send(x.toString()); //sends 001100100011000000111000
There is also typed arrays, but turning into a typed array is tricky if it isn't already a blob or file. On certain network APIs in Javascript the raw data is often represented as a string before you can touch it.
encoding
//myNetwork is a supposed network API that sends as binary
var x = 208;
myNetwork.send(String.fromCharCode(x)); //sends 11010000 , also known as Ð
decoding
var receivedString = "Ð";
var decodedNum = receivedString.charCodeAt(0); //208
The string method mentioned is 24 bits, whereas this is only 8 bits.
The drawback of this method is that there is obviously some waste if you want less than byte sized values. For example, you should be able to store 512 values in 9 bits, however you'd be forced to go up to 16 bits (2 bytes) which is 65,535 values because in unicode characters are all byte-sized. However, it is fine if you'll be utilizing the full range of values.
I am using the JSFeat Computer Vision Library and am trying to convert an image to greyscale. The function jsfeat.imgproc.grayscale outputs to a matrix (img_u8 below), where each element is an integer between 0 and 255. I was unsure how to apply this matrix to the original image so I went looking through their example at https://inspirit.github.io/jsfeat/sample_grayscale.htm.
Below is my code to convert an image to grey scale. I adopted their method to update the pixels in the original image but I do not understand how it works.
/**
* I understand this stuff
*/
let canvas = document.getElementById('canvas');
let ctx = canvas.getContext('2d');
let img = document.getElementById('img-in');
ctx.drawImage(img, 0, 0, img.width, img.height);
let imageData = ctx.getImageData(0, 0, img.width, img.height);
let img_u8 = new jsfeat.matrix_t(img.width, img.height, jsfeat.U8C1_t);
jsfeat.imgproc.grayscale(imageData.data, img.width, img.height, img_u8);
let data_u32 = new Uint32Array(imageData.data.buffer);
let i = img_u8.cols*img_u8.rows, pix = 0;
/**
* Their logic to update the pixel values of the original image
* I need help understanding how the following works
*/
let alpha = (0xff << 24);
while(--i >= 0) {
pix = img_u8.data[i];
data_u32[i] = alpha | (pix << 16) | (pix << 8) | pix;
}
/**
* I understand this stuff
*/
context.putImageData(imageData, 0, 0);
Thanks in advance!
It's a wide topic, but I'll try to roughly cover the basics in order to understand what goes on here.
As we know, it's using 32-bit integer values which means you can operate on four bytes simultaneously using fewer CPU instructions and therefor in many cases can increase overall performance.
Crash course
A 32-bit value is often notated as hex like this:
0x00000000
and represents the equivalent of bits starting with the least significant bit 0 on the right to the most significant bit 31 on the left. A bit can of course only be either on/set/1 or off/unset/0. 4 bits is a nibble, 2 nibbles are one byte. The hex value has each nibble as one digit, so here you have 8 nibbles = 4 bytes or 32 bits. As in decimal notation, leading 0s have no effect on the value, i.e. 0xff is the same as 0x000000ff (The 0x prefix also has no effect on the value; it is just the traditional C notation for hexadecimal numbers which was then taken over by most other common languages).
Operands
You can bit-shift and perform logic operations such as AND, OR, NOT, XOR on these values directly (in assembler language you would fetch the value from a pointer/address and load it into a registry, then perform these operations on that registry).
So what happens is this:
The << means bit-shift to the left. In this case the value is:
0xff
or in binary (bits) representation (a nibble 0xf = 1111):
0b11111111
This is the same as:
0x000000ff
or in binary (unfortunately we cannot denote bit representation natively in JavaScript actually, there is the 0b-prefix in ES6):
0b00000000 00000000 00000000 11111111
and is then bit-shifted to the left 24 bit positions, making the new value:
0b00000000 00000000 00000000 11111111
<< 24 bit positions =
0b11111111 00000000 00000000 00000000
or
0xff000000
So why is this necessary here? Well, that's an excellent question!
The 32-bit value in relation to canvas represents RGBA and each of the components can have a value between 0 and 255, or in hex a value between 0x00 and 0xff. However, since most consumer CPUs today uses little-endian byte order each components for the colors is at memory level stored as ABGR instead of RGBA for 32-bit values.
We are normally abstracted away from this in a high-level language such as JavaScript of course, but since we now work directly with memory bytes through typed arrays we have to consider this aspect as well, and in relation to registry width (here 32-bits).
So here we try to set alpha channel to 255 (fully opaque) and then shift it 24 bits so it becomes in the correct position:
0xff000000
0xAABBGGRR
(Though, this is an unnecessary step here as they could just as well have set it directly as 0xff000000 which would be faster, but anyhoo).
Next we use the OR (|) operator combined with bit-shift. We shift first to get the value in the correct bit position, then OR it onto the existing value.
OR will set a bit if either the existing or the new bit is set, otherwise it will remain 0. F.ex starting with an existing value, now holding the alpha channel value:
0xff000000
We then want the blue component of say value 0xcc (204 in decimal) combined which currently is represented in 32-bit as:
0x000000cc
so we need to first shift it 16 bits to the left in this case:
0x000000cc
<< 16 bits
0x00cc0000
When we now OR that value with the existing alpha value we get:
0xff000000
OR 0x00cc0000
= 0xffcc0000
Since the destination is all 0 bits only the value from source (0xcc) is set, which is what we want (we can use AND to remove unwanted bits but, that's for another day).
And so on for the green and red component (the order which in they are OR'ed doesn't matter so much).
So this line then does, lets say pix = 0xcc:
data_u32[i] = alpha | (pix << 16) | (pix << 8) | pix;
which translates into:
alpha = 0xff000000 Alpha
pix = 0x000000cc Red
pix << 8 = 0x0000cc00 Green
pix << 16 = 0x00cc0000 Blue
and OR'ed together would become:
value = 0xffcccccc
and we have a grey value since all components has the same value. We have the correct byte-order and can write it back to the Uint32 buffer using a single operation (in JS anyways).
You can optimize this line though by using a hard-coded value for alpha instead of a reference now that we know what it does (if alpha channel vary then of course you would need to read the alpha component value the same way as the other values):
data_u32[i] = 0xff000000 | (pix << 16) | (pix << 8) | pix;
Working with integers, bits and bit operators is as said a wide topic and this just scratches the surface, but hopefully enough to make it more clear what goes on in this particular case.
I am trying to get pixel data from PNG images for processing. The current way is by using canvas.drawImage followed canvas.getImageData (example here). I am looking for alternatives.
The problem with the current approach is that browsers modify pixel values influenced by alpha, as discussed here and here.
This question has been asked before, but no satisfactory answers are available.
The only way to do this without using canvas and getImageData() is to load the PNG file as a binary typed array and parse the file in code "manually".
Prerequisites:
For this you need the PNG specification which you can find here.
You need to know how to use typed arrays (for this a DataView is the most suitable view).
PNG files are chunk based and you will need to know how to parse chunks
A typical chunk based file has a four byte header called a FourCC identifier, followed by the size and misc. data depending on the file format definition.
Then chunks are placed right after this containing often a FOURCC (or four character code) and then the size of the chunk without the chunk header. In principle:
MAGIC FOURCC
SIZE/MISC - depending on definition
...
CHK1 - Chunk FourCC
SIZE - unsigned long
.... data
CHK2
SIZE
.... data
This format principle came originally from the Commodore Amiga platform and EA/IFF (Interleaved File Format) back in mid 80's.
But in modern days some vendors has extended or vary the chunk format, so for PNG chunks it will actually look like this:
Header (always 8 bytes and the same byte values):
‰PNG (first byte is 0x89, see specs for reason)
CR + LF 0x0C0A
EOC + LF 0x1A0A
Chunks:
SIZE (4 bytes, may be 0 (f.ex. IEND). Excl. chunk header and crc)
FOURCC (4 bytes, ie. "IHDR", "IDAT")
[...data] (length: SIZE x bytes)
CRC32 (4 bytes representing the CRC-32 checksum of the data)
(see the referenced specification link above for details).
And the byte-order (endianess) for PNG is always big-endian ("network" order).
This makes it easy to parse through the file supporting only some (or all) chunks. For PNG you would need to support at least (source):
IHDR must be the first chunk; it contains (in this order) the image's width, height, bit depth and color type.
IDAT contains the image, which may be split between multiple IDAT chunks. Such splitting increases the file size slightly, but makes it easier to stream the PNG. The IDAT chunk contains the actual image data, which is the output stream of the compression algorithm.
IEND marks the file end.
If you intend to support palette (color indexed) files you would also need to support the PLTE chunk. When you parse the IHDR chunk you will be able to see what color format is used (type 2 for RGB data, or 6 for RGBA and so on).
Parsing is itself easy so your biggest challenge would be supporting things like ICC profiles (when present in the iCCP chunk) to adjust the image color data. A typical chunk is the gamma chunk (gAMA) which contains a single gamma value you can apply to convert the data to linear format so that it displays correctly when display gamma is applied (there are also other special chunks related to colors).
The second biggest challenge would be the decompression which uses INFLATE. You can use a project such as PAKO zlib port to do this job for you and this port has performance close to native zlib. In addition to that, if you want to do error checking on the data (recommended) CRC-32 checking should also be supported.
For security reason you should always check that fields contain the data they're suppose to as well as that reserved space are initialized with either 0 or the defined data.
Hope this helps!
Example chunk parser: (note: won't run in IE).
function pngParser(buffer) {
var view = new DataView(buffer),
len = buffer.byteLength,
magic1, magic2,
chunks = [],
size, fourCC, crc, offset,
pos = 0; // current offset in buffer ("file")
// check header
magic1 = view.getUint32(pos); pos += 4;
magic2 = view.getUint32(pos); pos += 4;
if (magic1 === 0x89504E47 && magic2 === 0x0D0A1A0A) {
// parse chunks
while (pos < len) {
// chunk header
size = view.getUint32(pos);
fourCC = getFourCC(view.getUint32(pos + 4));
// data offset
offset = pos + 8;
pos = offset + size;
// crc
crc = view.getUint32(pos);
pos += 4;
// store chunk
chunks.push({
fourCC: fourCC,
size: size,
offset: offset,
crc: crc
})
}
return {chunks: chunks}
}
else {
return {error: "Not a PNG file."}
}
function getFourCC(int) {
var c = String.fromCharCode;
return c(int >>> 24) + c(int >>> 16 & 0xff) + c(int >>> 8 & 0xff) + c(int & 0xff);
}
}
// USAGE: ------------------------------------------------
fetch("//i.imgur.com/GP6Q3v8.png")
.then(function(resp) {return resp.arrayBuffer()}).then(function(buffer) {
var info = pngParser(buffer);
// parse each chunk here...
for (var i = 0, chunks = info.chunks, chunk; chunk = chunks[i++];) {
out("CHUNK : " + chunk.fourCC);
out("SIZE : " + chunk.size + " bytes");
out("OFFSET: " + chunk.offset + " bytes");
out("CRC : 0x" + (chunk.crc>>>0).toString(16).toUpperCase());
out("-------------------------------");
}
function out(txt) {document.getElementById("out").innerHTML += txt + "<br>"}
});
body {font: 14px monospace}
<pre id="out"></pre>
From here you can extract the IHDR to find image size and color type, then IDAT chunk(s) to deflate (PNG uses filters per scanline which do complicate things a bit, as well as a interlace mode, see specs) and your almost done ;)