I have a lineCollection of models that represent lines of a graph. Each line crunches through a dataCollection and generates averages once. For one line, its very snappy.
After the dataCollection is fetched, I have a listener to crunch the data for each line.
lineCollection.invoke('setData')
However, this kind of freezes the browser. My next approach was to try to set a timeout to hopefully stop blocking the ui:
lineCollection.each(function(model) {
setTimeout(model.setData, Math.round(Math.random() * 20));
});
This still sorta freezes the whole process. How can I tell the browser to crunch the data in the background?
Edit
"crunch the data" in my case is to zip variable sets of ~3600 length arrays, average each slice, and format ~3600 date objects.
// x axis
var xs = _.map(times, this.formatTime);
// y axis
var values = _.map(allSamples, function(samples) {
return _.pluck(samples, 'value');
});
var avgs = _.map(_.zip.apply(_, values), _.avg);
var ys = _.compact(avgs);
// graph data
var data = { x: xs.slice(xs.length - ys.length), y: ys };
I would definitely look into worker threads.
Related
I want to scale a contour in OpenCV.js. I have a valid contour in cnt variable of type cv.Mat (verified it by using drawContours).
I found a function in Python that does everything I need but I have problems converting it to Javascript.
Python version:
def scale_contour(cnt, scale):
M = cv2.moments(cnt)
cx = int(M['m10']/M['m00'])
cy = int(M['m01']/M['m00'])
cnt_norm = cnt - [cx, cy]
cnt_scaled = cnt_norm * scale
cnt_scaled = cnt_scaled + [cx, cy]
cnt_scaled = cnt_scaled.astype(np.int32)
return cnt_scaled
Here's what I started for Javascript:
function scaleContour(cnt, scale) {
console.log("cnt", cnt.data32S, cnt.rows, cnt.cols, cnt.type());
const M = cv.moments(cnt);
const cx = M['m10']/M['m00'];
const cy = M['m01']/M['m00'];
const offset = [Math.ceil(cx), Math.ceil(cy)];
console.log("Offset", offset);
// cannot use convenient Python arithmetics here,
// have to call functions
// although technically we have 1 row 2 cols for a point, but the cnt type is 2-channel CV_32SC2 (12)
// therefore keeping the size 1,1 and leave the second dimension as a channel to be compatible with the contour format
const pointMat = cv.matFromArray(1, 1, cnt.type(), offset);
console.log("pointMat", pointMat.data32S);
const cntNorm = new cv.Mat(cnt.rows, cnt.cols, cnt.type());
cv.subtract(cnt, pointMat, cntNorm); <-- my app crashes here with an exception that has only some random number - OpenCV seems to always do that when I'm doing something wrong or it's out of memory
console.log("ctnorm", cntNorm.data32S);
Unfortunately, I cannot find a good example on Python-like matrix operations in the official OpenCV.js documentation on basic data structures. It just shows how to create matrices but does not explain how to perform simple math operations with a matrix and a point-like value.
Also, I'm not sure when I need new cv.Mat(cnt.rows, cnt.cols, cnt.type()); and when new cv.Mat() is enough. The documentation has both but does not answer what is the rule of thumb to use an empty Mat and when it must be configured with row/col/type.
And the log output for cnt cols and rows is confusing, it prints 75 rows and 1 col, but the data is Int32Array(150). I found that sometimes the second layer of values are designated by type and not cols/rows. That's confusing. How should we know when to use rows=1,cols=2 and when rows=1,cols=2 and a type with 2 channels?
May I know If I could parse differently, so that performance and load time can be improved. I deal with file size that is in MBs.
My file with multiple objects is parsed as below:
I display multiple graphs in a single page.
'chart1' to 'chart5' arrays are used for each graph's Y-axis.
'xAxisTime' array is used for x-axis (common x-axis for all graphs).
eg: I get 50000 values in each array (chart1, chart2...xAxisTime).
data = fileWithMultipleObjects;
objects= data.split('\n');
for (var i= 0; i < objects.length - 2; i++) {
var obj = JSON.parse(objects[i])
if (obj.type=== "A") {
chart1.push(obj.c1)
chart2.push(obj.c2)
}
else {
chart1.push(null)
chart2.push(null)
}
if (obj.type=== "B") {
chart3.push(obj.c3)
chart4.push(obj.c4)
chart5.push(obj.c5)
}
else {
chart3.push(null)
chart4.push(null)
chart5.push(null)
}
//common for all charts - xAxis data
if (obj.date === undefined) {
obj.date = null
}
if (obj.date!== null) {
var date= obj.date
xAxisTime.push(date)
}
}
Reason for pushing null values when no data:
I need to show all 5 graphs in line with the time the job was run.
Also, this helps me use "this.point.y" (var index = this.point.y), which helps me display tooltips on plotline. Fiddle showing tooltip(when clicked on green line) with "this.point.y" :https://jsfiddle.net/xqb0cn5r/4/
This is how my graphs look. Fiddle: https://jsfiddle.net/xgpL1w3t/. Only charts that meets if condition are rendered. chart1 and chart5 are rendered in this case. Thank you.
Edited:
Below is the data structure of my file:
{"C1":55.77,"C2":11367.25,"type":"A","date":"10/24/2022 12:05:37.236"}
{"C3":55.77,"C4":11367.25,"type":"B","date":"10/24/2022 12:05:37.236","C5":445.21}
{"C1":55.77,"C2":11367.25,"type":"A","date":"10/24/2022 12:05:37.236"}
{"C1":55.77,"C2":11367.25,"type":"A","date":"10/24/2022 12:05:37.240"}
{"C3":55.77,"C4":11367.25,"type":"B","date":"10/24/2022 12:05:37.250","C5:445.25}
{"C3":55.77,"C4":11367.25,"type":"B","date":"10/24/2022 12:05:37.275","C5":445.26}
I display like 8 to 10 charts in real scenario with around 50000 points each. Common x-axis for all. Out of 50000, only 2000 to 4000 are real data, remaining are all null values. I push null, so that I can keep graphs in line with the whole time(x-axis) the job was run.
Right now, it takes time to load charts. As so much data to parse and load. I believe performance can be be improved if my data is parsed differently for the same scenario. I would appreciate any suggestions.
Performing single json parse as below:
var convertStringToJsonFormat= "[" + data.split('\n').join(", ")
JsonFormat = convertStringToJsonFormat.slice(0, -2);
JsonFormat = JsonFormat + "]"
data = JSON.parse(JsonFormat)
TL;DR I want to display a long-running strip chart with Plotly.js. I don't know how to discard old points.
Details
The following updater from my CodePen at https://codepen.io/Michael-F-Ellis/pen/QvXPQr does almost what I want. It shows a set of 20 samples in 2 traces that update continuously at 500 msec intervals. At the end of the demo, it plots all the points to show they still exist.
var cnt = 0;
var interval = setInterval(function() {
// Add next point to each trace
Plotly.extendTraces('graph', {
y: [[rand()], [rand()]]
}, [0, 1])
// Display only 20 most recent points
Plotly.relayout('graph', { 'xaxis.range': [cnt-20, cnt]})
cnt = cnt+1;
if(cnt === 100) {
// Before ending the run, show all points
// to demonstrate they still exist in Plotly.
Plotly.relayout('graph', { 'xaxis.range': [0, cnt]});
clearInterval(interval);
}
}, 500);
The problem is that I do want to delete older points. The real application needs to run essentially forever on a system with limited memory. I'm looking for a Plotly call that will drop the oldest N trace points. It needs to be reasonably efficient as performance of the target system is also limited.
Thanks!
https://codepen.io/Michael-F-Ellis/pen/YxeEwm
The above seems workable from a behavioral standpoint. Here's the revised updating routine:
Plotly.plot('graph', data);
var cnt = 0;
var max = 20;
var interval = setInterval(function() {
// Add next point to each trace
Plotly.extendTraces('graph', {
y: [[rand()], [rand()]]
}, [0, 1])
// Keep only 'max' most recent points
if(cnt > max) {
data[0].y.shift();
data[1].y.shift();
}
cnt = cnt+1;
if(cnt === 100) {
// Before ending the run, show all points
// to demonstrate that only 'max' points
// still exist in Plotly.
Plotly.relayout('graph', { 'xaxis.range': [0, cnt]});
clearInterval(interval);
}
}, 500);
The solution is to keep the data object in a var outside of Plotly and use shift() to drop old points from the beginning of the array as new points are added.
I'm open to another solution, especially if there are known memory or performance problems with this approach.
I'm using highcharts.js to visualize data series from a database. There's lots of data series and they can potantially change from the database they are collected from with ajax. I can't guarantee that they are flawless and sometimes they will have blank gaps in the dates, which is a problem. Highcharts simply draws a line through the entire gap to the next available date, and that's bad in my case.
The series exists in different resolutions. Hours, Days and Weeks. Meaning that a couple of hours, days or weeks can be missing. A chart will only show 1 resolution at a time on draw, and redraw if the resolution is changed.
The 'acutal' question is how to get highcharts to not draw those gaps in an efficient way that works for hous, days and weeks
I know highcharts (line type) can have that behaviour where it doesn't draw a single line over a gap if the gap begins with a null.
What I tried to do is use the resolution (noted as 0, 1, 2 for hour day or week), to loop through the array that contains the values for and detect is "this date + 1 != (what this date + 1 should be)
The code where I need to work this out is here. Filled with psudo
for (var k in data.values) {
//help start, psudo code.
if(object-after-k != k + resolution){ //The date after "this date" is not as expected
data.values.push(null after k)
}
//help end
HC_datamap.push({ //this is what I use to fill the highchart later, so not important
x: Date.parse(k),
y: data.values[k]
});
}
the k objects in data.values look like this
2015-05-19T00:00:00
2015-05-20T00:00:00
2015-05-21T00:00:00
...and more dates
as strings. They can number in thousands, and I don't want the user to have to wait forever. So performance is an issue and I'm not an expert here either
Please ask away for clarifications.
I wrote this loop.
In my case my data is always keyed to a date (12am) and it moves either in intervals of 1 day, 1 week or 1 month. Its designed to work on an already prepared array of points ({x,y}). Thats what dataPoints is, these are mapped to finalDataPoints which also gets the nulls. finalDataPoints is what is ultimately used as the series data. This is using momentjs, forwardUnit is the interval (d, w, or M).
It assumes that the data points are already ordered from earliest x to foremost x.
dataPoints.forEach(function (point, index) {
var plotDate = moment(point.x);
finalDataPoints.push(point);
var nextPoint = dataPoints[index+1];
if (!nextPoint) {
return;
}
var nextDate = moment(nextPoint.x);
while (plotDate.add(1, forwardUnit).isBefore(nextDate)) {
finalDataPoints.push({x: plotDate.toDate(), y: null});
}
});
Personally, object with property names as dates may be a bit problematic, I think. Instead I would create an array of data. Then simple loop to fill gaps shouldn't be very slow. Example: http://jsfiddle.net/4mxtvotv/ (note: I'm changing format to array, as suggested).
var origData = {
"2015-05-19T00:00:00": 20,
"2015-05-20T00:00:00": 30,
"2015-05-21T00:00:00": 50,
"2015-06-21T00:00:00": 50,
"2015-06-22T00:00:00": 50
};
// let's change to array format
var data = (function () {
var d = [];
for (var k in origData) {
d.push([k, origData[k]]);
}
return d;
})();
var interval = 'Date'; //or Hour or Month or Year etc.
function fillData(data, interval) {
var d = [],
now = new Date(data[0][0]), // first x-point
len = data.length,
last = new Date(data[len - 1][0]), // last x-point
iterator = 0,
y;
while (now <= last) { // loop over all items
y = null;
if (now.getTime() == new Date(data[iterator][0]).getTime()) { //compare times
y = data[iterator][1]; // get y-value
iterator++; // jump to next date in the data
}
d.push([now.getTime(), y]); // set point
now["set" + interval](now.getDate() + 1); // jump to the next period
}
return d;
}
var chart = new Highcharts.StockChart({
chart: {
renderTo: 'container'
},
series: [{
data: fillData(data, interval)
}]
});
Second note: I'm using Date.setDay() or Date.setMonth(), of course if your data is UTC-based, then should be: now["setUTC" + interval].
I need to scan through every pixel in a canvas image and do some fiddling with the colors etc. For optimal performance, should I grab all the data in one go and work on it through the array? Or should I call each pixel as I work on it.
So basically...
data = context.getImageData(x, y, height, width);
VS
data = context.getImageData(x, y, 1, 1); //in a loop height*width times.
You'll get much higher performances by grabbing the image all at once since :
a) a (contiguous) acces to an array is way faster than a function call.
b) especially when this function isa method of a DOM object having some overhead.
c) and there might be buffer refresh issues that might delay response (if canvas is
on sight... or not depending on double buffering implementation).
So go for a one-time grab.
I'll suggest you look into Javascript Typed Arrays to get the most of the
imageData result.
If i may quote myself, look at how you can handle pixels fast in this old post of mine
(look after 2) ):
Nice ellipse on a canvas?
(i quoted the relevant part below : )
You can get a UInt32Array view on your ImageData with :
var myGetImageData = myTempCanvas.getImageData(0,0,sizeX, sizeY);
var sourceBuffer32 = new Uint32Array(myGetImageData.data.buffer);
then sourceBuffer32[i] contains Red, Green, Blue, and transparency packed into one unsigned 32 bit int. Compare it to 0 to know if pixel is non-black ( != (0,0,0,0) )
OR you can be more precise with a Uint8Array view :
var myGetImageData = myTempCanvas.getImageData(0,0,sizeX, sizeY);
var sourceBuffer8 = new Uint8Array(myGetImageData.data.buffer);
If you deal only with shades of grey, then R=G=B, so watch for
sourceBuffer8[4*i]>Threshold
and you can set the i-th pixel to black in one time using the UInt32Array view :
sourceBuffer32[i]=0xff000000;
set to any color/alpha with :
sourceBuffer32[i]= (A<<24) | (B<<16) | (G<<8) | R ;
or just to any color :
sourceBuffer32[i]= 0xff000000 | (B<<16) | (G<<8) | R ;
(be sure R is rounded).
Listening to #Ken's comment, yes endianness can be an issue when you start fighting with bits 32 at a time.
Most computer are using little-endian, so RGBA becomes ABGR when dealing with them 32bits a once.
Since it is the vast majority of systems, if dealing with 32bit integer assume this is the case,
and you can -for compatibility- reverse your computation before writing the 32 bits results on Big endian systems.
Let me share those two functions :
function isLittleEndian() {
// from TooTallNate / endianness.js. https://gist.github.com/TooTallNate/4750953
var b = new ArrayBuffer(4);
var a = new Uint32Array(b);
var c = new Uint8Array(b);
a[0] = 0xdeadbeef;
if (c[0] == 0xef) { isLittleEndian = function() {return true }; return true; }
if (c[0] == 0xde) { isLittleEndian = function() {return false }; return false; }
throw new Error('unknown endianness');
}
function reverseUint32 (uint32) {
var s32 = new Uint32Array(4);
var s8 = new Uint8Array(s32.buffer);
var t32 = new Uint32Array(4);
var t8 = new Uint8Array(t32.buffer);
reverseUint32 = function (x) {
s32[0] = x;
t8[0] = s8[3];
t8[1] = s8[2];
t8[2] = s8[1];
t8[3] = s8[0];
return t32[0];
}
return reverseUint32(uint32);
};
Additionally to what GameAlchemist said, if you want to get or set all the colors of a pixel simultaneously, but you don't want to check endianness, you can use a DataView:
var data = context.getImageData(0, 0, canvas.width, canvas.height);
var view = new DataView(data.data.buffer);
// Read or set pixel (x,y) as #RRGGBBAA (big endian)
view.getUint32(4 * (x + y*canvas.width));
view.setUint32(4 * (x + y*canvas.width), 0xRRGGBBAA);
// Read or set pixel (x,y) as #AABBGGRR (little endian)
view.getUint32(4 * (x + y*canvas.width), true);
view.setUint32(4 * (x + y*canvas.width), 0xAABBGGRR, true);
// Save changes
ctx.putImageData(data, 0, 0);
It depends on what exactly you're doing, but I'd suggest grabbing it all at once, and then looping through it.
Grabbing it all at once is faster than grabbing it pixel by pixel, since searching through an array is a lot faster than searching through a canvas, once for each pixel.
If you're really in need of speed, look into web workers. You can set each one to grab a specific section of the canvas, and since they can run simultaneously, they'll make much better use out of your CPU.
getImageData() isn't really slow enough for you to notice the difference if you were to grab it all at once or individually, in my experiences using the function.