I have some numbers in json which overflow the Number type, so I want it to be bigint, but how?
{"foo":[[0],[64],[89],[97]],"bar":[[2323866757078990912,144636906343245838,441695983932742154,163402272522524744],[2477006750808014916,78818525534420994],[18577623609266200],[9008333127155712]]}
TLDR;
You may employ JSON.parse() reviver parameter
Detailed Solution
To control JSON.parse() behavior that way, you can make use of the second parameter of JSON.parse (reviver) - the function that pre-processes key-value pairs (and may potentially pass desired values to BigInt()).
Yet, the values recognized as numbers will still be coerced (the credit for pinpointing this issue goes to #YohanesGultom).
To get around this, you may enquote your big numbers (to turn them into strings) in your source JSON string, so that their values are preserved upon converting to bigint.
As long as you wish to convert to bigint only certain numbers, you would need to pick up appropriate criteria (e.g. to check whether the value exceeds Number.MAX_SAFE_INTEGER with Number.isSafeInteger(), as #PeterSeliger has suggested).
Thus, your problem may be solved with something, like this:
// source JSON string
const input = `{"foo":[[0],[64],[89],[97]],"bar":[[2323866757078990912,144636906343245838,441695983932742154,163402272522524744],[2477006750808014916,78818525534420994],[18577623609266200],[9008333127155712]]}`
// function that implements desired criteria
// to separate *big numbers* from *small* ones
//
// (works for input parameter num of type number/string)
const isBigNumber = num => !Number.isSafeInteger(+num)
// function that enquotes *big numbers* matching
// desired criteria into double quotes inside
// JSON string
//
// (function checking for *big numbers* may be
// passed as a second parameter for flexibility)
const enquoteBigNumber = (jsonString, bigNumChecker) =>
jsonString
.replaceAll(
/([:\s\[,]*)(\d+)([\s,\]]*)/g,
(matchingSubstr, prefix, bigNum, suffix) =>
bigNumChecker(bigNum)
? `${prefix}"${bigNum}"${suffix}`
: matchingSubstr
)
// parser that turns matching *big numbers* in
// source JSON string to bigint
const parseWithBigInt = (jsonString, bigNumChecker) =>
JSON.parse(
enquoteBigNumber(jsonString, bigNumChecker),
(key, value) =>
!isNaN(value) && bigNumChecker(value)
? BigInt(value)
: value
)
// resulting output
const output = parseWithBigInt(input, isBigNumber)
console.log("output.foo[1][0]: \n", output.foo[1][0], `(type: ${typeof output.foo[1][0]})`)
console.log("output.bar[0][0]: \n", output.bar[0][0].toString(), `(type: ${typeof output.bar[0][0]})`)
.as-console-wrapper{min-height: 100% !important;}
Note: you may find RegExp pattern to match strings of digits among JSON values not quite robust, so feel free to come up with yours (as mine was the quickest I managed to pick off the top of my head for demo purposes)
Note: you may still opt in for some library, as it was suggested by #YohanesGultom, yet adding 10k to your client bundle or 37k to your server-side dependencies (possibly, to docker image size) for that sole purpose may not be quite reasonable.
Related
I've scraped this allergy data string from a public website:
Cedar 679 gr/m3 High, Grass 20 gr/m3 Medium, Trees 80 gr/m3 Medium, Molds Low.
Sometimes the number of items is fewer, but the general format for trees and grasses is always like this, with commas separating each type of allergen:
TYPE AMOUNT g/m3 LEVEL
Molds is the exception; assume it will always be a string of text. Assume we don't require the molds data at all.
What library or technique would you use to parse this into a neat JSON object, for example:
{
"Cedar": "679",
"Grass": "20",
"Trees": "80"
}
As Sam stated in the comments, it'd be ideal to utilize npmjs.com/package/csv-parser
However, if you want to use vanilla JS, I wrote a basic script that works given your input:
//function takes a csv string and returns a list of Objects
//and only includes values with 4 parts
function parseCsv(csvString) {
let out = {};
let spacedValues = csvString.split(/,\s*/);
let values = spacedValues.map(str => str.split(" "));
values.forEach((value, index) => {
if (value.length === 4) {
//you can change the value from an object to value[1] if you only need the amount
out[`${value[0]}`] = {
AllergenAmount: value[1],
AllergenUnits: value[2],
AllergenLevel: value[3]
}
});
}
// add an else if here if you want to keep values with more/less than 4 parts of the string
});
return out;
}
//wrapper that implements the builtin JSON.stringify method
const csvToJSONString = csvString => JSON.stringify(parseCsv(csvString));
To use it, just pass the csv string into the csvToJSONString function, and it will return a JSON string. You can also change the properties from an object to value[1] if you only needed the amount (commented in code).
I worked with "csvtojsn" module before in a similar situation and it helped a lot.
https://www.npmjs.com/package/csvtojson
You should try csv-parse I'm using it in my current project and it works like a charm.
I have seen other related questions, but they did not solve my problem, or may be I somehow missed the exactly same resolved queries.
Here is the problem. The service that I call up returns a JSON response with some keys having large numbers as values, and I want to pass them on to my view and display. The issue is that they are getting rounded off, which I don't want. Actually its coming inside a buffer from which I am doing now:
JSON.parse(res.body.toString()) // res.body is a Buffer
and sending to view. How can I retain the whole number in the form of a string and send this to view so exactly the same is made available to UI.
I thought may be a replacer will help, but it does not works too.
const replacer = (key, value) => {
if (typeof value === 'number') {
return JSON.stringify(value);
}
return value;
};
//78787878977987787897897897123456786747398
const obj = {
name: 'John',
income: 78787878977987787897897897123456786747398,
car: null
};
var buf = Buffer.from(JSON.stringify(obj));
console.log(buf.toString());
// console.log(JSON.stringify(buf.toString()))
// console.log('func res: ', replacer('key', 78787878977987787897897897123456786747398))
// console.log(obj.income.toString())
console.log(JSON.stringify(obj, replacer));
You can recommend some external trusted library, or better, suggest me the solution through direct code only.
Edit:
The outcome in short is: Convert the response to String before returning from the server. Once it gets into JS (Buffer in my case), the conversion already occurred meaning that from the application side, nothing can be done to retrieve it.
Please let me know if there's a real solution to this without modifying server response.
Unfortunately, the number is higher than max_safe_integer, so if it ever gets parsed as a number, even if it's converted back to a string later (such as with the reviver function, the second parameter to JSON.parse), it won't be reliable. But luckily, since you have a JSON string, you can replace numeric values with string values before JSON.parseing it. For example:
const resBody = '{"foo":"bar", "objs":[{"name":"John", "income": 78787878977987787897897897123456786747398}]}';
const resBodyReplaced = resBody.replace(/: *(\d+)/g, ':"$1"');
console.log(JSON.parse(resBodyReplaced).objs[0].income);
I have 3 objects with the keys as it looks like this:
They are in format of YYYYMMDD. I am trying to get data of a month. But I am not getting the desired output.
When I query it like this:
var ref = db.child("-KPXECP6a1pXaM4gEYe0");
ref.orderByKey().startAt("20160901").once("value", function (snapshot) {
console.log("objects: " + snapshot.numChildren());
snapshot.forEach(function(childSnapshot) {
console.log(childSnapshot.key);
});
});
I get the following output:
objects: 3
20160822-KPl446bbdlaiQx6BOPL
20160901-KPl48ID2FuT3tAVf4DW
20160902-KPl4Fr4O28VpsIkB70Z
When I query this along with endAt:
ref.orderByKey().startAt("20160901").endAt("20160932").once("value", function (snapshot) {
console.log("objects: " + snapshot.numChildren());
snapshot.forEach(function(childSnapshot) {
console.log(childSnapshot.key);
});
});
I get this:
objects: 0
If I use ~ sign at the end,
ref.orderByKey().startAt("20160901").endAt("20160932~").once("value", function (snapshot) {
console.log("objects: " + snapshot.numChildren());
snapshot.forEach(function(childSnapshot) {
console.log(childSnapshot.key);
});
});
I get the output:
objects: 3
20160822-KPl446bbdlaiQx6BOPL
20160901-KPl48ID2FuT3tAVf4DW
20160902-KPl4Fr4O28VpsIkB70Z
Is there anything I am missing here?
Wow... this took some time to dig up. Thanks for the jsfiddle, that helped a lot.
TL;DR: ensure that you always have a non-numeric character in your search criteria, e.g. ref.orderByKey().startAt("20160901-").endAt("20160931~").
Longer explanation
In Firebase all keys are stored as strings. But we make it possible for developers to store arrays in the database. In order to allow that we store the array indices as string properties. So ref.set(["First", "Second", "Third"]) is actually stored as:
"0": "First"
"1": "Second"
"2": "Third"
When you get the data back from Firebase, it'll convert this into an array again. But it is important for your current use-case to understand that it is stored as key-value pairs with string keys.
When you execute a query, Firebase tries to detect whether you're querying a numeric range. When it thinks that is your intent, it converts the arguments into numbers and queries against the numeric conversion of the keys on the server.
In your case since you are querying on only a numeric value, it will switch to this numeric query mode. But since your keys are actually all strings, nothing will match.
For this reason I'd recommend that you prefix keys with a constant string. Any valid character will do, I used a - in my tests. This will fool our "is it an array?" check and everything will work the way you want it.
The quicker fix is to ensure that your conditions are non-convertible to a number. In the first snippet I did this by adding a very low range ASCII character to the startAt() and a very high ASCII character to endAt().
Both of these are workarounds for the way Firebase deals with arrays. Unfortunately the API doesn't have a simple way to handle it and requires such a workaround.
I am fetching data from bigquery which I need to store in MongoDB as integer, so that I can perform operations on that data in Mongo. Even though the data types of columns in bigquery is Integer, its nodejs api is returning string in its Javascript object. E.g. I'm getting results that look like [{row1:'3',row2:'4',row3:'5'},{row1:'13',row2:'14',row3:'15'}...]
typeof gives string on each element of object. I can run a loop and convert each element to integer, but that is not scalable on the data set. Also, I don't want all strings to be converted to integers, only ones which are stored as integer in bigquery. I'm using gcloud module in nodejs to fetch data.
assuming you know where the type property is on the response, something like this would work.
var response = [{type: 'Integer', value: '13'} /* other objects.. */];
var mappedResponse = response.map(function(item) {
// Put your logic here
// This implementation just bails
if (item.type != 'Integer') return item;
// This just converts the value to an integer, but beware
// it returns NaN if the value isn't actually a number
item.value = parseInt(item.value);
// you MUST return the item after modifying it.
return item;
});
This still loops over each item, but immediately bails out if it's not what we're looking for. Could also compose multiple maps and filters to generalize this out.
The only way to get by this is by first applying a filter, but this basically achieves the same thing as our initial type check
var mappedResponse = response
// Now we only deal with integers in the map function
.filter(x => x.type == 'Integer)
.map(function(item) {
// This just converts the value to an integer, but beware
// it returns NaN if the value isn't actually a number
item.value = parseInt(item.value);
// you MUST return the item after modifying it.
return item;
});
BigQuery deliberately encodes integers as strings when returning them via API to avoid loss of precision for large values. For now, the only option is to parse them on the client side.
if I have a large javascript string array that has over 10,000 elements,
how do I quickly search through it?
Right now I have a javascript string array that stores the description of a job,
and I"m allowing the user to dynamic filter the returned list as they type into an input box.
So say I have an string array like so:
var descArr = {"flipping burgers", "pumping gas", "delivering mail"};
and the user wants to search for: "p"
How would I be able to search a string array that has 10000+ descriptions in it quickly?
Obviously I can't sort the description array since they're descriptions, so binary search is out. And since the user can search by "p" or "pi" or any combination of letters, this partial search means that I can't use associative arrays (i.e. searchDescArray["pumping gas"] )
to speed up the search.
Any ideas anyone?
As regular expression engines in actual browsers are going nuts in terms of speed, how about doing it that way? Instead of an array pass a gigantic string and separate the words with an identifer.
Example:
String "flipping burgers""pumping gas""delivering mail"
Regex: "([^"]*ping[^"]*)"
With the switch /g for global you get all the matches. Make sure the user does not search for your string separator.
You can even add an id into the string with something like:
String "11 flipping burgers""12 pumping gas""13 delivering mail"
Regex: "(\d+) ([^"]*ping[^"]*)"
Example: http://jsfiddle.net/RnabN/4/ (30000 strings, limit results to 100)
There's no way to speed up an initial array lookup without making some changes. You can speed up consequtive lookups by caching results and mapping them to patterns dynamically.
1.) Adjust your data format. This makes initial lookups somewhat speedier. Basically, you precache.
var data = {
a : ['Ant farm', 'Ant massage parlor'],
b : ['Bat farm', 'Bat massage parlor']
// etc
}
2.) Setup cache mechanics.
var searchFor = function(str, list, caseSensitive, reduce){
str = str.replace(/(?:^\s*|\s*$)/g, ''); // trim whitespace
var found = [];
var reg = new RegExp('^\\s?'+str, 'g' + caseSensitive ? '':'i');
var i = list.length;
while(i--){
if(reg.test(list[i])) found.push(list[i]);
reduce && list.splice(i, 1);
}
}
var lookUp = function(str, caseSensitive){
str = str.replace(/(?:^\s*|\s*$)/g, ''); // trim whitespace
if(data[str]) return cache[str];
var firstChar = caseSensitive ? str[0] : str[0].toLowerCase();
var list = data[firstChar];
if(!list) return (data[str] = []);
// we cache on data since it's already a caching object.
return (data[str] = searchFor(str, list, caseSensitive));
}
3.) Use the following script to create a precache object. I suggest you run this once and use JSON.stringify to create a static cache object. (or do this on the backend)
// we need lookUp function from above, this might take a while
var preCache = function(arr){
var chars = "abcdefghijklmnopqrstuvwxyz".split('');
var cache = {};
var i = chars.length;
while(i--){
// reduce is true, so we're destroying the original list here.
cache[chars[i]] = searchFor(chars[i], arr, false, true);
}
return cache;
}
Probably a bit more code then you expected, but optimalisation and performance doesn't come for free.
This may not be an answer for you, as I'm making some assumptions about your setup, but if you have server side code and a database, you'd be far better off making an AJAX call back to get the cut down list of results, and using a database to do the filtering (as they're very good at this sort of thing).
As well as the database benefit, you'd also benefit from not outputting this much data (10000 variables) to a web based front end - if you only return those you require, then you'll save a fair bit of bandwidth.
I can't reproduce the problem, I created a naive implementation, and most browsers do the search across 10000 15 char strings in a single digit number of milliseconds. I can't test in IE6, but I wouldn't believe it to more than 100 times slower than the fastest browsers, which would still be virtually instant.
Try it yourself: http://ebusiness.hopto.org/test/stacktest8.htm (Note that the creation time is not relevant to the issue, that is just there to get some data to work on.)
One thing you could do wrong is trying to render all results, that would be quite a huge job when the user has only entered a single letter, or a common letter combination.
I suggest trying a ready made JS function, for example the autocomplete from jQuery. It's fast and it has many options to configure.
Check out the jQuery autocomplete demo
Using a Set for large datasets (1M+) is around 3500 times faster than Array .includes()
You must use a Set if you want speed.
I just wrote a node script that needs to look up a string in a 1.3M array.
Using Array's .includes for 10K lookups:
39.27 seconds
Using Set .has for 10K lookups:
0.01084 seconds
Use a Set.