Parsing a large JSON array in Javascript - javascript

I'm supposed to parse a very large JSON array in Javascipt. It looks like:
mydata = [
{'a':5, 'b':7, ... },
{'a':2, 'b':3, ... },
.
.
.
]
Now the thing is, if I pass this entire object to my parsing function parseJSON(), then of course it works, but it blocks the tab's process for 30-40 seconds (in case of an array with 160000 objects).
During this entire process of requesting this JSON from a server and parsing it, I'm displaying a 'loading' gif to the user. Of course, after I call the parse function, the gif freezes too, leading to bad user experience. I guess there's no way to get around this time, is there a way to somehow (at least) keep the loading gif from freezing?
Something like calling parseJSON() on chunks of my JSON every few milliseconds? I'm unable to implement that though being a noob in javascript.
Thanks a lot, I'd really appreciate if you could help me out here.

You might want to check this link. It's about multithreading.
Basically :
var url = 'http://bigcontentprovider.com/hugejsonfile';
var f = '(function() {
send = function(e) {
postMessage(e);
self.close();
};
importScripts("' + url + '?format=json&callback=send");
})();';
var _blob = new Blob([f], { type: 'text/javascript' });
_worker = new Worker(window.URL.createObjectURL(_blob));
_worker.onmessage = function(e) {
//Do what you want with your JSON
}
_worker.postMessage();
Haven't tried it myself to be honest...
EDIT about portability: Sebastien D. posted a comment with a link to mdn. I just added a ref to the compatibility section id.

I have never encountered a complete page lock down of 30-40 seconds, I'm almost impressed! Restructuring your data to be much smaller or splitting it into many files on the server side is the real answer. Do you actually need every little byte of the data?
Alternatively if you can't change the file #Cyrill_DD's answer of a worker thread will be able to able parse data for you and send it to your primary JS. This is not a perfect fix as you would guess though. Passing data between the 2 threads requires the information to be serialised and reinterpreted, so you could find a significant slow down when the data is passed between the threads and be back to square one again if you try to pass all the data across at once. Building a query system into your worker thread for requesting chunks of the data when you need them and using the message callback will prevent slow down from parsing on the main thread and allow you complete access to the data without loading it all into your main context.
I should add that worker threads are relatively new, main browser support is good but mobile is terrible... just a heads up!

Related

Send message from WeChat mini-program to web-view

I'm building WeChat Mini-Program that on one of it's pages has web-view control. For example:
page.wxml
<web-view src="https://..." bindmessage="onWebViewMessage"></web-view>
page.js
const app = getApp();
Page({
onWebViewMessage:function(e) {
console.log(e);
},
onLoad:function() {
}
});
In web-view an HTML page is loaded (index.html), that includes jweixin-1.3.2.js lib from WeChat, for connecting with WeChat API as well as connect to parent Mini-program. Page is empty, no DOM elements, just javascript that will execute when document is loaded.
It has it's javascript something like this:
index.js
document.addEventListener('DOMContentLoaded',function(){
wx.miniProgram.postMessage({data:'test'});
});
I am able to post messages from this document to mini-program without issues. Also can send some mini-program navigation commands such as wx.miniProgram.navigateTo({url:'path/to/page'}); so all seems fine. I can also get callback in Mini-program when web-view has completed loading.
Question:
How can I post message from Mini-program to web-view? For example, to pass a string or an Object to the web-view.
I have been googling for hours and can't seem to find anyone doing it, but I can't believe it's just one-way communication possible.
Any help or idea is appreciated!
I have found an effective way to pass data from mini-program to web-view content, and it seems at this moment in time, this is the only possible way to do it.
Mini-program
1. Base64 module
You will need to be able to convert normal String into Base64 string. Mini-program API has a method for converting byte array into base64 string, but that won't be usable for this purpose. So, create your own module that does that:
File: lib/b64.js
var string2base64 = function(str) {
.... here put your js code for making b64 string ....
return result;
};
module.exports = {
string2base64
};
2. Page with Web-View
In the page that has web-view control, prepare DOM element in wxml file like this:
File: pages/xxx/index.wxml
<web-view src="{{webURL}}" bindload="onWebLoad" binderror="onWebError"></web-view>
Notice that src parameter is now bound to page's webURL property. Whenever page sets value to this property, will automatically be applied to the DOM elemenet.
In file pages/xxx/index.js you will need to add base64 module:
const b64 = require('../../lib/b64.js')
note that require path may vary depending how you have setup your project
and in page's data object, add webURL and webBaseURL properties, like this:
Page({
data: {
webURL:'',
webBaseURL:'https://your/web/app/url',
messageQueue:[],
messageQueueSize:0,
.... other page properties go here ....
},
..... rest of your page code goes here .....
})
Notice that webURL is set to be empty. This means that when page loads, an empty string will be set to DOM object by default.
webBaseURL will explain just in a bit.
messageQueue is an Array that will store pending messages to be sent to web-view.
messageQueueSize is just Array length. Used for better performance, to avoid reading Array.length.
3. Start Message Queue
In onShow callback of the page, set webURL and start interval that will read messageQueue Array every 250ms. You can change the way this is done if you dislike using intervals, this was just simplest way to do theory test.
onShow: function(){
// This will start loading of the content in web-view
this.setData({webURL: this.data.webBaseURL } );
// Sends message from message queue to web-view
let _this = this;
setInterval(function(e) {
if( _this.data.messageQueueSize < 1 ) return;
_this.data.messageQueueSize --;
let msg = _this.data.messageQueue.splice(0,1);
_this.setData({webURL: _this.data.webBaseURL+"#"+msg});
},250);
}
You can see that message is appended to web-view source (url) as a hash.
webBaseURL is used to generate final URL with hash, that is then send to web-view.
4. Add a Message to the Queue
To create a message in message queue, just define following method in your page:
addMessageToQueue: function(obj) {
obj.unique = Math.round(Math.random()*100000);
let msg = b64.string2base64(JSON.stringify(obj));
this.data.messageQueue.push(msg);
this.data.messageQueueSize++;
}
Whenever you call this method, just pass an Object with whatever properties you need it to have, and it will be converted into JSON string, then to base64 string, and finally appended to the message queue.
unique property is added to make generated base64 result always different even if the rest of object properties are the same - I just needed this for the purpose of my project. You can ignore it / remove it if you do not need it.
Since there's interval running and checking on the message queue, all messages added like this will be sent to web-view in the same order they were added to the queue.
Now there's only one thing left - to add hash change listening in the HTML page we have loaded into the web-view:
HTML Web-app
1. Listen to hash change
window.addEventListener("hashchange",function(e){
let messageBase64 = window.location.hash.substr(1);
let json = window.atob( messageBase64 );
let data = JSON.parse(json);
console.log("Received data from mini-program:",data);
});
Tested on Xiaomi Mi8 Pro. I am yet to test on other devices sold in China.
Cheers!

Non blocking Javascript and concurrency

I have code on a web-worker and because i can't post to it an object with methods(functions) , i dont know how to stop blocking the UI with this code:
if (data != 'null') {
obj['backupData'] = obj.tbl.data().toArray();
obj['backupAllData'] = data[0];
}
obj.tbl.clear();
obj.tbl.rows.add(obj['backupAllData']);
var ext = config.extension.substring(1);
$.fn.dataTable.ext.buttons[ext + 'Html5'].action(e, dt, button, config);
obj.tbl.clear();
obj.tbl.rows.add(obj['backupData'])
This code exports records from an html table. Data is an array and is returned from a web worker and sometimes can have 50k or more objects.
As obj and all the methods that it contains are not transferable to we-worker, when data length 30k ,40k or 50k or even more, the UI blocks.
which is the best way to do this?
Thanks in advance.
you could try wrapping the heavy work in an async function like a timeout to allow the engine to queue the whole logic and elaborate it as soon as it has time
setTimeout(function(){
if (data != 'null') {
obj['backupData'] = obj.tbl.data().toArray();
obj['backupAllData'] = data[0];
}
//heavy stuff
}, 0)
or , if the code is extremely long, you can try figure it out a strategy to split your code into chunk of operation and execute each chunk in a separate async function (timeout)
Best way to iterate over an array without blocking the UI
Update:
Sadly, ImmutableJS doesn't work at the moment across webworkers. You should be able to transfer the ArrayBuffer so you don't need to parse it back into an array. Also read this article. If your workload is that heavy, it would be best to actually send back one item at a time from the worker.
Previously:
The code is converting all the data into an array, which is immediately costly. Try returning an immutable data structure from web worker if possible. This will guarantee that it doesn't change when the references change and you can continue iterating over it slowly in batches.
The next thing you can do is to use requestIdleCallback to schedule small batches of items to be processed.
This way you should be able to make the UI breathe a bit.

How to save multiple objects to an array in a chrome extension?

I'm building my first chrome extension and I want it to track the TV series I watch and I'm currently trying to get it to save metadata on the series that I am following.
I have a content script that returns the title, the newest episode (and the URL of this episode) as well as the URL of the cover image of the series. I am currently trying to save it with some code on my background script (I have made sure to include "storage" under the permissions section of the manifest file).
So far my script looks like this (This was developed with help from Trying to save and fetch a Javascript object using chrome.storage API?):
var bkg = chrome.extension.getBackgroundPage();
response.aID = new Series(response.aTitle,response.aNewEp,response.aNewEpURL,response.aImage);
chrome.storage.sync.set(response.aID, function(){
chrome.storage.sync.get(function(val){
bkg.console.log("The saved title is: ", val.anTitle);
bkg.console.log("The saved newEp is: ", val.anNewEp);
bkg.console.log("The saved newEpURL is: ", val.anNewEpURL);
bkg.console.log("The saved imageURL is: ", val.anImage);
});
});
Problem is, the script only seems to store one response.aID at a time, so I can never store data for more than 1 TV series. Every time I try, the script seems to overwrite my previous entry. So I would like to ask whether there's any way to store more than 1 TV series at a time?
I have looked at storing an array and then pushing each new object into that array (Store an array with chrome.storage.local), but I don't quite understand the syntax involved so I'm not sure if this would work for me.
Unfortunately you didn't include the piece of code where you save your data, but i think you dont store your data with indices for the different TV series so the stored one gets overwritten everytime you store another one.
Anyway I would prefer storing your data in a JSON element (basically every javascript element can by converted to one but continue reading) because js provides several functions for this format which make it quite easy to use.
When opening your extension, load the data and call
var data = JSON.parse (yourloadedstring);
so the string (which should look like {"TVShows": [{"title": "How i met your mother", "url": ...}, {...}]} (look here for an explenation how JSON works) gets "translated" to an element from which you can read simply by calling
data.TVShows[0].title
or
data.TVShows[1].imageURL
You can edit this data JSON element when you add a new show for example by saying
data.TVShows[2].title = "The Big Bang Theory";
data.TVShows[2].URL= ...;
data.TVShows[2].imageURL= ...;
and save this element to chromes storage by calling
var dataToSave = JSON.stringify(data);
You have a string in your storage then, containing all information you need and you can simply parse it later like explained above :)
I hope everything is clearly to understand, if not pls ask me!
Cheers

Store and Retrieve WinJS.Binding.List in Application Data

I'm developing a Windows 8 Store app using HTML/JavaScript and I've run into an issue storing and retrieving a WinJS.Binding.List into Windows.Storage.ApplicationData.current.roamingSettings.
I DID get this to work by hand rolling my own method of converting the binding list into XML string and storing that, then on retrieval parsing it back out into a list. But, this seems crazy inefficient and I'm trying to find a better way. I've tried JSON.stringify() and JSON.parse() which seem to store and retrieve the right data but as soon as I bind the data to the winControl the application crashes with a 0 (no error message at all).
Here's a bit of my code to demonstrate what I'm attempting (list is a binding list):
function onSaveData() {
if (list) {
Windows.Storage.ApplicationData.current.roamingSettings.values["data"] = JSON.stringify(list);
}
}
function onLoadData() {
var data = Windows.Storage.ApplicationData.current.roamingSettings.values["data"];
if (data) {
list = JSON.parse(data);
var listview = element.querySelector("#mylistview").winControl;
listview.itemsSource = new WinJS.Binding.List(list);
}
}
I know I can get this working the long way, so I'm not looking for any solution... I'm really just hoping there's an easy way to store/retrieve these data objects that I'm missing. If I can find an easier way to do this it will eliminate about 40 lines of code and I can stop using an entire library. Also, as I go forward I plan to have more binding lists that will need to be stored as well. Thanks!
You need to bind to the List's dataSource property, not to the List itself:
listview.itemsSource = new WinJS.Binding.List(list).dataSource;
The dataSource property is specifically the IListDataSource that the ListView requires for a data source. The ListView doesn't understand anything about the WinJS.Binding.List directly, only through that particular interface. (I discuss this in Chapter 7, section "The Structure of Data Sources", in my free ebook, Programming Windows Store Apps with HTML, CSS, and JavaScript, 2nd Edition.)
Your saving and reloading the list with JSON is completely fine.

Recieving a stream from rails 4.0 in JS callback

I'm trying transmit an image file from the server to the client, but my javascript callback becomes active before the stream closes I doing this because sending it in a traditional render json: times out and takes way to long anyway. The stream takes much less time, but i keep can't get all the data before the callback fires up.
controller code
def mytest
image=ImageList.new(AssistMe.get_url(image_url))
response.stream.write image.export_pixels(0, 0, image.columns, image.rows, 'RGBA').to_s
response.stream.close
end
javascript
var getStream, runTest;
runTest = function() {
return $.post('/dotest', getStream);};
getStream = function(params) {
return document.getElementById('whatsup2').innerHTML =
"stream is here " + params.length;};
the response is an array, I can make it an array of arrays by adding a "[" at the front and a "],['finish'] at the end to be able to detect the end of the data, but I haven't been able to figure out how to get javascript to wait until the end of stream to run. I assume i need to set up some kind of pole to check for the end, but how do I attach it to the callback?
Okay, here's a blog that describes this pretty well
blog
But i decided to forgo a stream and use .to_s. Since you can pipe several actions tougher
render object.method.method.to_s you get all the server side benefits of using a stream without the complexity. If you have a slow process where you need to overlap the client and server actions, then go to the blog and do it. Otherwise to_s covers it pretty well

Categories

Resources