Develop Reference

How to perform math operations with OpenCV Matrix and a point-like value in Javascript?

I want to scale a contour in OpenCV.js. I have a valid contour in cnt variable of type cv.Mat (verified it by using drawContours).
I found a function in Python that does everything I need but I have problems converting it to Javascript.
Python version:
def scale_contour(cnt, scale):
M = cv2.moments(cnt)
cx = int(M['m10']/M['m00'])
cy = int(M['m01']/M['m00'])
cnt_norm = cnt - [cx, cy]
cnt_scaled = cnt_norm * scale
cnt_scaled = cnt_scaled + [cx, cy]
cnt_scaled = cnt_scaled.astype(np.int32)
return cnt_scaled
Here's what I started for Javascript:
function scaleContour(cnt, scale) {
console.log("cnt", cnt.data32S, cnt.rows, cnt.cols, cnt.type());
const M = cv.moments(cnt);
const cx = M['m10']/M['m00'];
const cy = M['m01']/M['m00'];
const offset = [Math.ceil(cx), Math.ceil(cy)];
console.log("Offset", offset);
// cannot use convenient Python arithmetics here,
// have to call functions
// although technically we have 1 row 2 cols for a point, but the cnt type is 2-channel CV_32SC2 (12)
// therefore keeping the size 1,1 and leave the second dimension as a channel to be compatible with the contour format
const pointMat = cv.matFromArray(1, 1, cnt.type(), offset);
console.log("pointMat", pointMat.data32S);
const cntNorm = new cv.Mat(cnt.rows, cnt.cols, cnt.type());
cv.subtract(cnt, pointMat, cntNorm); <-- my app crashes here with an exception that has only some random number - OpenCV seems to always do that when I'm doing something wrong or it's out of memory
console.log("ctnorm", cntNorm.data32S);
Unfortunately, I cannot find a good example on Python-like matrix operations in the official OpenCV.js documentation on basic data structures. It just shows how to create matrices but does not explain how to perform simple math operations with a matrix and a point-like value.
Also, I'm not sure when I need new cv.Mat(cnt.rows, cnt.cols, cnt.type()); and when new cv.Mat() is enough. The documentation has both but does not answer what is the rule of thumb to use an empty Mat and when it must be configured with row/col/type.
And the log output for cnt cols and rows is confusing, it prints 75 rows and 1 col, but the data is Int32Array(150). I found that sometimes the second layer of values are designated by type and not cols/rows. That's confusing. How should we know when to use rows=1,cols=2 and when rows=1,cols=2 and a type with 2 channels?

How to edit objects attributes from array using parallel/multithreading Hamsters.js Javascript lib

When using Hamster.js we have to define the parameter array as
params = {"array":my_array}
my_array is formed by elements with many attributes and I need to change the attributes x and y.
In the function I pass to Hamster.js I defined:
function map_node_hamster() {
params.array.forEach(function (d) {
d.x = some_calculation_here;
d.y = other_calculation_here;
}
}
but after calling the hamster.run() function, the elements of my original array stay intact.
As I need performance, I thought Hamsters could just change the values that my array is pointing. I actually do not know much about Javascript and how it manages arrays.
I'm calling the run function like this:
console.log("Before hamster:");
console.log(network.nodes);
p = {'array': network.nodes, "w": w, "h":h};
hamsters.run(p, map_node_hamster, function(output){console.log(output); return output;}, hamsters.maxThreads, true);
console.log("After hamster:");
console.log(network.nodes);
And the elements of network.nodes are intact after hamsters.run().
How could I change elements of array inside run function? Or.. How would be the right way to do the changes?
As the vector nodes is large, copying, sorting, and things like this will decrease performance and maybe it will be worst than single thread/non-parallel version.

It seems like the answer is to create an index array and change nodes in the callback function (output).
Despite the behavior of my "d3 graphic representation of network" is really not what I expected... maybe the code is actually right in the sense of answering my question (how to change objects from array in parallel computing with Hamsters.js). That is:
p = {'array': network.indexes,
"nodes": network.nodes,
"w": w,
"h": h};
hamsters.run(p, map_node_hamster, function (output) {
output.forEach(function (d) {
network.nodes[d.i].x = d.xy[0];
network.nodes[d.i].y = d.xy[1];
});
return output;
}, cores, true);
And... changing the function to work like this:
function map_node_hamster() {
pfor (var i = 0; i < params.array.length; i++) {
var result;
var d = params.nodes[params.array[i]];
var d = params.nodes[params.array[i]];
result = {x: d.x calculation, y: d.y calculation};
rtn.data.push({"i": params.array[i], "xy": [result.x, result.y]});
}
}

What is the correct way of writing a binding for D3.js animations?

I wonder how to create a d3js binding to simplify animations. I trying to use something similar to a dynamic version of TikZ coordinates, where you simply define a 2d coordinate and refer to it for positioning elements. Here I simply want to define an object M that encapsulate one value and some bindings to objects-attributes, e.g., radius of circle, x-position, y-position, ... . Every change of the value of M should animate the objects-attributes of attached objects at the same time.
I created a jsFiddle:
http://jsfiddle.net/66qxjze9/1/
In this example:
test.addBinding(line,["x1","x2"],[]);
test.addBinding(dot,["cx1"],[]);
test.addBinding(dot2,["cy1"],[function(v){return 2*v;}]);
I wanted to animate the x1, x2 attribute of a line, the cx1 attribute of a dot and the cy1 of dot2 when I call test.setValue(30);.
Basically I thought it should work like
// on value changes
this.setValue = function(x){
// update the value itself
this.v = x;
// update each binded-object
for(var i=0;i<this.entry.length;i++){
var bindObject = this.entry[i];
// update each binded-attribute of this object
for(var j=0;j<bindObject.property.length;j++){
// how to update?
// identify
var modifier = function(x){return x;};
// or a custom function ?
if(typeof bindObject.function[j] !== "undefined"){
modifier = bindObject.function[j];
}
var to = modifier(x);
var attrName = bindObject.property[j];
// update
bindObject.handle
.transition()
.duration(2000)
.attr(attrName, to);
}
}
};
My main idea is to use a wrapper for this value which knows which svg-attributes have to be changed.
I would like to use
// bind some svg elements
var test = new d3jsbinding();
test.addBinding(line,["x1","x2"],[]);
test.addBinding(dot,["cx1"],[]);
test.addBinding(dot2,["cx1"],[function(v){return 2*v;}]);
// update them (at the same time)
test.setValue(30);
Unfortunately I found no way to enqueue the attributes-updates (or better assign the new values of each attribute) before calling .transition().duration(2000)

Have a look at my bl.ocks
Essentially, you need to write the loop into:
d3.transition()
.duration(parent.duration)
.ease(parent.ease)
.each(function() {
... HERE ...
});
disclaimer:

pdf.js: Get the text colour

I have a simple pdf file, containing the words "Hello world", each in a different colour.
I'm loading the PDF, like this:
PDFJS.getDocument('test.pdf').then( onPDF );
function onPDF( pdf )
{
pdf.getPage( 1 ).then( onPage );
}
function onPage( page )
{
page.getTextContent().then( onText );
}
function onText( text )
{
console.log( JSON.stringify( text ) );
}
And I get a JSON output like this:
{
"items" : [{
"str" : "Hello ",
"dir" : "ltr",
"width" : 29.592,
"height" : 12,
"transform" : [12, 0, 0, 12, 56.8, 774.1],
"fontName" : "g_font_1"
}, {
"str" : "world",
"dir" : "ltr",
"width" : 27.983999999999998,
"height" : 12,
"transform" : [12, 0, 0, 12, 86.5, 774.1],
"fontName" : "g_font_1"
}
],
"styles" : {
"g_font_1" : {
"fontFamily" : "serif",
"ascent" : 0.891,
"descent" : 0.216
}
}
}
However, I've not been able to find a way to determine the colour of each word. When I render it, it renders properly, so I know the information is in there somewhere. Is there somewhere I can access this?

As Respawned alluded to, there is no easy answer that will work in all cases. That being said, here are two approaches which seem to work fairly well. Both having upsides and downsides.
Approach 1
Internally, the getTextContent method uses whats called an EvaluatorPreprocessor to parse the PDF operators, and maintain the graphic state. So what we can do is, implement a custom EvaluatorPreprocessor, overwrite the preprocessCommand method, and use it to add the current text color to the graphic state. Once this is in place, anytime a new text chunk is created, we can add a color attribute, and set it to the current color state.
The downsides to this approach are:
Requires modifying the PDFJS source code. It also depends heavily on
the current implementation of PDFJS, and could break if this is
changed.
It will fail in cases where the text is used as a path to be filled with an image. In some PDF creators (such as Photoshop), the way it creates colored text is, it first creates a clipping path from all the given text characters, and then paints a solid image over the path. So the only way to deduce the fill-color is by reading the pixel values from the image, which would require painting it to a canvas. Even hooking into paintChar wont be of much help here, since the fill color will only emerge at a later time.
The upside is, its fairly robust and works irrespective of the page background. It also does not require rendering anything to canvas, so it can be done entirely in the background thread.
Code
All the modifications are made in the core/evaluator.js file.
First you must define the custom evaluator, after the EvaluatorPreprocessor definition.
var CustomEvaluatorPreprocessor = (function() {
function CustomEvaluatorPreprocessor(stream, xref, stateManager, resources) {
EvaluatorPreprocessor.call(this, stream, xref, stateManager);
this.resources = resources;
this.xref = xref;
// set initial color state
var state = this.stateManager.state;
state.textRenderingMode = TextRenderingMode.FILL;
state.fillColorSpace = ColorSpace.singletons.gray;
state.fillColor = [0,0,0];
}
CustomEvaluatorPreprocessor.prototype = Object.create(EvaluatorPreprocessor.prototype);
CustomEvaluatorPreprocessor.prototype.preprocessCommand = function(fn, args) {
EvaluatorPreprocessor.prototype.preprocessCommand.call(this, fn, args);
var state = this.stateManager.state;
switch(fn) {
case OPS.setFillColorSpace:
state.fillColorSpace = ColorSpace.parse(args[0], this.xref, this.resources);
break;
case OPS.setFillColor:
var cs = state.fillColorSpace;
state.fillColor = cs.getRgb(args, 0);
break;
case OPS.setFillGray:
state.fillColorSpace = ColorSpace.singletons.gray;
state.fillColor = ColorSpace.singletons.gray.getRgb(args, 0);
break;
case OPS.setFillCMYKColor:
state.fillColorSpace = ColorSpace.singletons.cmyk;
state.fillColor = ColorSpace.singletons.cmyk.getRgb(args, 0);
break;
case OPS.setFillRGBColor:
state.fillColorSpace = ColorSpace.singletons.rgb;
state.fillColor = ColorSpace.singletons.rgb.getRgb(args, 0);
break;
}
};
return CustomEvaluatorPreprocessor;
})();
Next, you need to modify the getTextContent method to use the new evaluator:
var preprocessor = new CustomEvaluatorPreprocessor(stream, xref, stateManager, resources);
And lastly, in the newTextChunk method, add a color attribute:
color: stateManager.state.fillColor
Approach 2
Another approach would be to extract the text bounding boxes via getTextContent, render the page, and for each text, get the pixel values which reside within its bounds, and take that to be the fill color.
The downsides to this approach are:
The computed text bounding boxes are not always correct, and in some cases may even be off completely (eg: rotated text). If the bounding box does not cover at least partially the actual text on canvas, then this method will fail. We can recover from complete failures, by checking that the text pixels have a color variance greater than a threshold. The rationale being, if bounding box is completely background, it will have little variance, in which case we can fallback to a default text color (or maybe even the color of k nearest-neighbors).
The method assumes the text is darker than the background. Otherwise, the background could be mistaken as the fill color. This wont be a problem is most cases, as most docs have white backgrounds.
The upside is, its simple, and does not require messing with the PDFJS source-code. Also, it will work in cases where the text is used as a clipping path, and filled with an image. Though this can become hazy when you have complex image fills, in which case, the choice of text color becomes ambiguous.
Demo
http://jsfiddle.net/x2rajt5g/
Sample PDF's to test:
https://www.dropbox.com/s/0t5vtu6qqsdm1d4/color-test.pdf?dl=1
https://www.dropbox.com/s/cq0067u80o79o7x/testTextColour.pdf?dl=1
Code
function parseColors(canvasImgData, texts) {
var data = canvasImgData.data,
width = canvasImgData.width,
height = canvasImgData.height,
defaultColor = [0, 0, 0],
minVariance = 20;
texts.forEach(function (t) {
var left = Math.floor(t.transform[4]),
w = Math.round(t.width),
h = Math.round(t.height),
bottom = Math.round(height - t.transform[5]),
top = bottom - h,
start = (left + (top * width)) * 4,
color = [],
best = Infinity,
stat = new ImageStats();
for (var i, v, row = 0; row < h; row++) {
i = start + (row * width * 4);
for (var col = 0; col < w; col++) {
if ((v = data[i] + data[i + 1] + data[i + 2]) < best) { // the darker the "better"
best = v;
color[0] = data[i];
color[1] = data[i + 1];
color[2] = data[i + 2];
}
stat.addPixel(data[i], data[i+1], data[i+2]);
i += 4;
}
}
var stdDev = stat.getStdDev();
t.color = stdDev < minVariance ? defaultColor : color;
});
}
function ImageStats() {
this.pixelCount = 0;
this.pixels = [];
this.rgb = [];
this.mean = 0;
this.stdDev = 0;
}
ImageStats.prototype = {
addPixel: function (r, g, b) {
if (!this.rgb.length) {
this.rgb[0] = r;
this.rgb[1] = g;
this.rgb[2] = b;
} else {
this.rgb[0] += r;
this.rgb[1] += g;
this.rgb[2] += b;
}
this.pixelCount++;
this.pixels.push([r,g,b]);
},
getStdDev: function() {
var mean = [
this.rgb[0] / this.pixelCount,
this.rgb[1] / this.pixelCount,
this.rgb[2] / this.pixelCount
];
var diff = [0,0,0];
this.pixels.forEach(function(p) {
diff[0] += Math.pow(mean[0] - p[0], 2);
diff[1] += Math.pow(mean[1] - p[1], 2);
diff[2] += Math.pow(mean[2] - p[2], 2);
});
diff[0] = Math.sqrt(diff[0] / this.pixelCount);
diff[1] = Math.sqrt(diff[1] / this.pixelCount);
diff[2] = Math.sqrt(diff[2] / this.pixelCount);
return diff[0] + diff[1] + diff[2];
}
};

This question is actually extremely hard if you want to do it to perfection... or it can be relatively easy if you can live with solutions that work only some of the time.
First of all, realize that getTextContent is intended for searchable text extraction and that's all it's intended to do.
It's been suggested in the comments above that you use page.getOperatorList(), but that's basically re-implementing the whole PDF drawing model in your code... which is basically silly because the largest chunk of PDFJS does exactly that... except not for the purpose of text extraction but for the purpose of rendering to canvas. So what you want to do is to hack canvas.js so that instead of just setting its internal knobs it also does some callbacks to your code. Alas, if you go this way, you won't be able to use stock PDFJS, and I rather doubt that your goal of color extraction will be seen as very useful for PDFJS' main purpose, so your changes are likely not going to get accepted upstream, so you'll likely have to maintain your own fork of PDFJS.
After this dire warning, what you'd need to minimally change are the functions where PDFJS has parsed the PDF color operators and sets its own canvas painting color. That happens around line 1566 (of canvas.js) in function setFillColorN. You'll also need to hook the text render... which is rather a character renderer at canvas.js level, namely CanvasGraphics_paintChar around line 1270. With these two hooked, you'll get a stream of callbacks for color changes interspersed between character drawing sequences. So you can reconstruct the color of character sequences reasonably easy from this.. in the simple color cases.
And now I'm getting to the really ugly part: the fact that PDF has an extremely complex color model. First there are two colors for drawing anything, including text: a fill color and stroke (outline) color. So far not too scary, but the color is an index in a ColorSpace... of which there are several, RGB being only one possibility. Then there's also alpha and compositing modes, so the layers (of various alphas) can result in a different final color depending on the compositing mode. And the PDFJS has not a single place where it accumulates color from layers.. it simply [over]paints them as they come. So if you only extract the fill color changes and ignore alpha, compositing etc.. it will work but not for complex documents.
Hope this helps.

There's no need to patch pdfjs, the transform property gives the x and y, so you can go through the operator list and find the setFillColor op that precedes the text op at that point.

Canvas getImageData() For optimal performance. To pull out all data or one at a time?

I need to scan through every pixel in a canvas image and do some fiddling with the colors etc. For optimal performance, should I grab all the data in one go and work on it through the array? Or should I call each pixel as I work on it.
So basically...
data = context.getImageData(x, y, height, width);
VS
data = context.getImageData(x, y, 1, 1); //in a loop height*width times.

You'll get much higher performances by grabbing the image all at once since :
a) a (contiguous) acces to an array is way faster than a function call.
b) especially when this function isa method of a DOM object having some overhead.
c) and there might be buffer refresh issues that might delay response (if canvas is
on sight... or not depending on double buffering implementation).
So go for a one-time grab.
I'll suggest you look into Javascript Typed Arrays to get the most of the
imageData result.
If i may quote myself, look at how you can handle pixels fast in this old post of mine
(look after 2) ):
Nice ellipse on a canvas?
(i quoted the relevant part below : )
You can get a UInt32Array view on your ImageData with :
var myGetImageData = myTempCanvas.getImageData(0,0,sizeX, sizeY);
var sourceBuffer32 = new Uint32Array(myGetImageData.data.buffer);
then sourceBuffer32[i] contains Red, Green, Blue, and transparency packed into one unsigned 32 bit int. Compare it to 0 to know if pixel is non-black ( != (0,0,0,0) )
OR you can be more precise with a Uint8Array view :
var myGetImageData = myTempCanvas.getImageData(0,0,sizeX, sizeY);
var sourceBuffer8 = new Uint8Array(myGetImageData.data.buffer);
If you deal only with shades of grey, then R=G=B, so watch for
sourceBuffer8[4*i]>Threshold
and you can set the i-th pixel to black in one time using the UInt32Array view :
sourceBuffer32[i]=0xff000000;
set to any color/alpha with :
sourceBuffer32[i]= (A<<24) | (B<<16) | (G<<8) | R ;
or just to any color :
sourceBuffer32[i]= 0xff000000 | (B<<16) | (G<<8) | R ;
(be sure R is rounded).
Listening to #Ken's comment, yes endianness can be an issue when you start fighting with bits 32 at a time.
Most computer are using little-endian, so RGBA becomes ABGR when dealing with them 32bits a once.
Since it is the vast majority of systems, if dealing with 32bit integer assume this is the case,
and you can -for compatibility- reverse your computation before writing the 32 bits results on Big endian systems.
Let me share those two functions :
function isLittleEndian() {
// from TooTallNate / endianness.js. https://gist.github.com/TooTallNate/4750953
var b = new ArrayBuffer(4);
var a = new Uint32Array(b);
var c = new Uint8Array(b);
a[0] = 0xdeadbeef;
if (c[0] == 0xef) { isLittleEndian = function() {return true }; return true; }
if (c[0] == 0xde) { isLittleEndian = function() {return false }; return false; }
throw new Error('unknown endianness');
}
function reverseUint32 (uint32) {
var s32 = new Uint32Array(4);
var s8 = new Uint8Array(s32.buffer);
var t32 = new Uint32Array(4);
var t8 = new Uint8Array(t32.buffer);
reverseUint32 = function (x) {
s32[0] = x;
t8[0] = s8[3];
t8[1] = s8[2];
t8[2] = s8[1];
t8[3] = s8[0];
return t32[0];
}
return reverseUint32(uint32);
};

Additionally to what GameAlchemist said, if you want to get or set all the colors of a pixel simultaneously, but you don't want to check endianness, you can use a DataView:
var data = context.getImageData(0, 0, canvas.width, canvas.height);
var view = new DataView(data.data.buffer);
// Read or set pixel (x,y) as #RRGGBBAA (big endian)
view.getUint32(4 * (x + y*canvas.width));
view.setUint32(4 * (x + y*canvas.width), 0xRRGGBBAA);
// Read or set pixel (x,y) as #AABBGGRR (little endian)
view.getUint32(4 * (x + y*canvas.width), true);
view.setUint32(4 * (x + y*canvas.width), 0xAABBGGRR, true);
// Save changes
ctx.putImageData(data, 0, 0);

It depends on what exactly you're doing, but I'd suggest grabbing it all at once, and then looping through it.
Grabbing it all at once is faster than grabbing it pixel by pixel, since searching through an array is a lot faster than searching through a canvas, once for each pixel.
If you're really in need of speed, look into web workers. You can set each one to grab a specific section of the canvas, and since they can run simultaneously, they'll make much better use out of your CPU.
getImageData() isn't really slow enough for you to notice the difference if you were to grab it all at once or individually, in my experiences using the function.