Develop Reference

How to compare two base64 strings in Javascript and ignore some differences

I'm getting one base64 string from API response and other one I'm converted image (which is in test data file) to base64 using cypress readfile method.
When I'm using below command the assertion is failing because there is tracking number difference which will be always new with every call.
And I'm getting 2 different base64.
//This base64 is from API response
var base64FromAPI =
res.body.completedShipments[0].completedPackages[0].documents[0].image;
//Image is picked from Test Data file and converts to base64
cy.readFile(`cypress/e2e/Testdata/Canada Post 02StoreId.pdf`, "base64").should(
"eq",
base64FromAPI
);
Because there is tracking number on the label(image) which will be generated from API response is always different.
Is there any other way to compare base64 strings and to ignore some % of difference while comparing in cypress or javascript.
Or is there any other way to do this.
Thanks in advance.

Essentially you can't do this at the base64 level. The differences in a raw bitstream like base64 are totally meaningless. The differences can only become apparent through rendering that image. Actually, what you need to do is pretty complex! I'm assuming it's not possible or a good idea in your use case to change away from having the server add the text to the image, to for example, using DOM to overlay it instead.
If that's the case, the only thing you could do is utilise visual regression testing. With this, you can set a threshold on which a % similarity is defined.
Since the base64 comes from the API. This would probably mean also having test code that injects an img tag with the base64 as the source, so you can allow the visual snapshot to take place.
This works at the level of image analysis rather than on the actual bitstream. Internally it will render and compare the images.
Another way I can think of, though this is quite complex and I wouldn't pursue it unless the above did not work is to:
Use image manipulation libraries to load the base64 into an actual rendered image in memory.
Try to cut away/crop the superimposed text using image manipulation libraries in order to reliably remove areas of difference.
Base 64 that.
Compare that to a known stable base64 of the "rest" of the image.

What is the best way to include huge json into node.js script?

How to include huge json file (550MB) into node.js script?
I tried:
var json = require('./huge-data-set.json')
The script was executed with increased --max-old-space-size parameter
node --max-old-space-size=4096 diff.js
it failed with:
buffer.js:500
throw new Error('"toString()" failed');
^
Error: "toString()" failed
at Buffer.toString (buffer.js:500:11)
at Object.fs.readFileSync (fs.js:552:33)
at Object.Module._extensions..json (module.js:556:20)

Behind Node is v8 engine implementing JS implementation. V8 is written in c++ and as a result it uses types to save its characters in strings (sarcasm used with caution here).
As specified by ECMA JavaScript stores every character on two bytes (utf16 encoded).
The String type is the set of all ordered sequences of zero or more
16-bit unsigned integer values (“elements”) up to a maximum length of
2^53-1 elements
so what you are experiencing is not an out of memory limit, but instead a failed operation since the string is larger than the bytes size of the string type.
if you try to write the json object in javascript instead of reading it from fs (by requiring it), you will have an out of memory exception then, since the limit is set to prevent such cases from happening.

The error arises because you're trying to create a (single) string that is too large.
Some possible solutions:
Use a different backing store (e.g. sqlite db) and query data as you need instead of loading it all into memory at once
If your data is an array at the top level, consider having each array element on a separate line, that way you can just buffer a line and JSON.parse() that, then continue to the next line. However, you could still run into memory/GC issues if the parsed JavaScript values eat up too much space.

How can I save a very large in-memory object to file?

I have a very large array with thousands of items
I tried this solution:
Create a file in memory for user to download, not through server
of creating an anchor
text file
~~JSON.stringify on the array caused the tab to freeze~~ Correction: Trying to log out the result caused the tab to freeze, stringify by itself works fine
The data was originally in string form but creating an anchor with that data resulted in a no-op, I'm assuming also because the data was too big, because using dummy data successfully resulted in a file download being triggered
How can I get this item onto my filesystem?
edit/clarification:
There is a very large array that I can only access via the the browser inspector/console. I can't access it via any other language

Javascript does not allow you to read or write files, except for cookies, and I think the amount of data you are using exceeds the size limit for cookies. This is for security reasons.
However languages such as php, python and ruby allow the reading and writing of files. It appears you are using binary data, so use binary files and write functions.
As to the choice of language : if you already know one use that, or whichever you can get help with. Writing a file is a very basic operation and all three languages are equally good. If you don't know any of these languages you can literally copy and paste the code from their websites.

What is the maximum length that $.parseJSON() can handle?

I have a long json array that needs to be sent to an html5 mobile app and parsed. The whole array has around 700kb (gziped to 150kb) and it's 554976 characters long at the moment. But it will increase on time.
Using jquery to parse the json, my app crashes while trying to parse it. And so does jsonlint, json parser.fr and any other online json validator I try, so I'm guessing eval() is not an option either.
Might be a broad question but what is the maximum "acceptable" length for a json array?
I have already removed as much data as I can from the array, the only option I can think of is to split the array in 3-4 server calls and parse it separately in the app. Is there any other option?
EDIT
Thanks to #fabien for pointing that if jsonlint crashes there is a problem on the json. There was a hidden "space" character in one of the nodes. It was parsed correctly on the server but not on the client.

I've parsed way bigger arrays with jquery.
My first guess is that there's an error in your json.
You have many ways to find it (sublime text could highlight the error but some time, it's a bit long). Try to paste it in a web tool like http://www.jsoneditoronline.org/. and use any of the buttons (to format or to send to the right view). It'll tell you where the error is.

What is the most efficient way in JavaScript to parse huge amounts of data from a file

What is the most efficient way in JavaScript to parse huge amounts of data from a file?
Currently I use JSON parse to serialize an uncompressed 250MB file, which is really slow. Is there a simple and fast way to read a lot of data in JavaScript from a file without looping through every character? The data stored in the file are only a few floating point arrays?
UPDATE:
The file contains a 3d mesh, 6 buffers (vert, uv etc). Also the buffers need to be presented as typed arrays. streaming is not a option because the file has to be fully loaded before the graphics engine can continue. Maybe a better question is how to transfer huge typed arrays from a file to javascript in the most efficient way.

I would recommend a SAX based parser for these kind of JavaScript or a stream parser.
DOM parsing would load the whole thing in memory and this is not the way to go by for large files like you mentioned.
For Javascript based SAX Parsing (in XML) you might refer to
https://code.google.com/p/jssaxparser/
and
for JSON you might write your own, the following link demonstrates how to write a basic SAX based parser in Javascript
http://ajaxian.com/archives/javascript-sax-based-parser

Have you tried encoding it to a binary and transferring it as a blob?
https://developer.mozilla.org/en-US/docs/DOM/XMLHttpRequest/Sending_and_Receiving_Binary_Data
http://www.htmlgoodies.com/html5/tutorials/working-with-binary-files-using-the-javascript-filereader-.html#fbid=LLhCrL0KEb6

There isn't a really good way of doing that, because the whole file is going to be loaded into memory and we all know that all of them have big memory leaks. Can you not instead add some paging for viewing the contents of that file?
Check if there are any plugins that allow you to read the file as a stream, that will improve this greatly.
UPDATE
http://www.html5rocks.com/en/tutorials/file/dndfiles/
You might want to read about the new HTML5 API's to read local files. You will have the issue with downloading 250mb of data still tho.

I can think of 1 solution and 1 hack
SOLUTION:
Extending the split the data in chunks: it boils down to http protocol. REST parts on the notion that http has enough "language" for most client-server scenarios.
You can setup on the client a request header Content-len to establish how much data you need per request
Then on the backend have some options http://httpstatus.es
Reply a 413 if the server is simply unable to get that much data from the db
417 if the server is able to reply but not under the requested header (Content-len)
206 with the provided chunk, letting know the client "there is more from where that came from"
HACK:
Use Websocket and get the binary file. Then use the html5 FileAPI to load it into memory.
This is likely to fail though because its not the download causing the problem, but the parsing of an almost-endless JS object

You're out of luck on the browser. Not only do you have to download the file, but you'll have to parse the json regardless. Parse it on the server, break it into smaller chunks, store that data into the db, and query for what you need.