I am fetching data from bigquery which I need to store in MongoDB as integer, so that I can perform operations on that data in Mongo. Even though the data types of columns in bigquery is Integer, its nodejs api is returning string in its Javascript object. E.g. I'm getting results that look like [{row1:'3',row2:'4',row3:'5'},{row1:'13',row2:'14',row3:'15'}...]
typeof gives string on each element of object. I can run a loop and convert each element to integer, but that is not scalable on the data set. Also, I don't want all strings to be converted to integers, only ones which are stored as integer in bigquery. I'm using gcloud module in nodejs to fetch data.
assuming you know where the type property is on the response, something like this would work.
var response = [{type: 'Integer', value: '13'} /* other objects.. */];
var mappedResponse = response.map(function(item) {
// Put your logic here
// This implementation just bails
if (item.type != 'Integer') return item;
// This just converts the value to an integer, but beware
// it returns NaN if the value isn't actually a number
item.value = parseInt(item.value);
// you MUST return the item after modifying it.
return item;
});
This still loops over each item, but immediately bails out if it's not what we're looking for. Could also compose multiple maps and filters to generalize this out.
The only way to get by this is by first applying a filter, but this basically achieves the same thing as our initial type check
var mappedResponse = response
// Now we only deal with integers in the map function
.filter(x => x.type == 'Integer)
.map(function(item) {
// This just converts the value to an integer, but beware
// it returns NaN if the value isn't actually a number
item.value = parseInt(item.value);
// you MUST return the item after modifying it.
return item;
});
BigQuery deliberately encodes integers as strings when returning them via API to avoid loss of precision for large values. For now, the only option is to parse them on the client side.
Related
I have some numbers in json which overflow the Number type, so I want it to be bigint, but how?
{"foo":[[0],[64],[89],[97]],"bar":[[2323866757078990912,144636906343245838,441695983932742154,163402272522524744],[2477006750808014916,78818525534420994],[18577623609266200],[9008333127155712]]}
TLDR;
You may employ JSON.parse() reviver parameter
Detailed Solution
To control JSON.parse() behavior that way, you can make use of the second parameter of JSON.parse (reviver) - the function that pre-processes key-value pairs (and may potentially pass desired values to BigInt()).
Yet, the values recognized as numbers will still be coerced (the credit for pinpointing this issue goes to #YohanesGultom).
To get around this, you may enquote your big numbers (to turn them into strings) in your source JSON string, so that their values are preserved upon converting to bigint.
As long as you wish to convert to bigint only certain numbers, you would need to pick up appropriate criteria (e.g. to check whether the value exceeds Number.MAX_SAFE_INTEGER with Number.isSafeInteger(), as #PeterSeliger has suggested).
Thus, your problem may be solved with something, like this:
// source JSON string
const input = `{"foo":[[0],[64],[89],[97]],"bar":[[2323866757078990912,144636906343245838,441695983932742154,163402272522524744],[2477006750808014916,78818525534420994],[18577623609266200],[9008333127155712]]}`
// function that implements desired criteria
// to separate *big numbers* from *small* ones
//
// (works for input parameter num of type number/string)
const isBigNumber = num => !Number.isSafeInteger(+num)
// function that enquotes *big numbers* matching
// desired criteria into double quotes inside
// JSON string
//
// (function checking for *big numbers* may be
// passed as a second parameter for flexibility)
const enquoteBigNumber = (jsonString, bigNumChecker) =>
jsonString
.replaceAll(
/([:\s\[,]*)(\d+)([\s,\]]*)/g,
(matchingSubstr, prefix, bigNum, suffix) =>
bigNumChecker(bigNum)
? `${prefix}"${bigNum}"${suffix}`
: matchingSubstr
)
// parser that turns matching *big numbers* in
// source JSON string to bigint
const parseWithBigInt = (jsonString, bigNumChecker) =>
JSON.parse(
enquoteBigNumber(jsonString, bigNumChecker),
(key, value) =>
!isNaN(value) && bigNumChecker(value)
? BigInt(value)
: value
)
// resulting output
const output = parseWithBigInt(input, isBigNumber)
console.log("output.foo[1][0]: \n", output.foo[1][0], `(type: ${typeof output.foo[1][0]})`)
console.log("output.bar[0][0]: \n", output.bar[0][0].toString(), `(type: ${typeof output.bar[0][0]})`)
.as-console-wrapper{min-height: 100% !important;}
Note: you may find RegExp pattern to match strings of digits among JSON values not quite robust, so feel free to come up with yours (as mine was the quickest I managed to pick off the top of my head for demo purposes)
Note: you may still opt in for some library, as it was suggested by #YohanesGultom, yet adding 10k to your client bundle or 37k to your server-side dependencies (possibly, to docker image size) for that sole purpose may not be quite reasonable.
I have a map of id => value that I want to sort by value.
But no matter what I do, it always gets sorted by id.
Basically I have a sorted map on server side that I send to javascript via json.
{"3":"Apple","2":"Banana","1":"Orange"}
After de-serialization I get
{
1:"Orange",
2:"Banana",
3:"Apple"
}
And no matter what I try, it seems to stay in this order. Is it possible in javascript to force a non ascending sort order with interger keys?
var json = '{"3":"Apple", "2":"Banana", "1":"Orange"}';
var data = $.parseJSON(json);
for (var ix in data) {
console.log(ix + ": " + data[ix]);
}
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
You should not rely on objects key order for these reasons.
I personally would recommend you to either use a Map, or to build an Array instead.
Below is an example to build an array from your source: for simplicity, I've added a key property to make the sorting easier.
Note: I'm using Array.from to build the array, which is taking the length from the parsed object keys length, and using the callback to init the object inline.
var json = '{"3":"Apple", "2":"Banana", "1":"Orange"}';
// Parse the json string.
const parsed = JSON.parse(json);
// Acquire the keys length
const length = Object.keys(parsed).length;
// Build an array of objects ordered in the same way it came.
const result = Array.from({length}, (_, i) => ({key: length - i, [length - i]: parsed[length - i]}));
// Log a copy of the result.
console.log(JSON.parse(JSON.stringify(result)));
// Sort ascending:
result.sort((a,b) => a.key - b.key);
// Log a copy of the sorted result.
console.log(JSON.parse(JSON.stringify(result)));
// Sort descending:
result.sort((a,b) => b.key - a.key);
// log the sorted array
console.log(result);
If you really want to rely on key orders, you can (of course), but using an array is slightly cleverer and gives no chance to have something which is not ordered as expected, unless (of course) the sorting algorithm is wrong or fails for some reason (like if key is undefined or null or not numeric in the above case).
As a final note, I'm aware that the question is about sorting an object, but because of the above reasons, I think the correct answer is just to DON'T use an object at all in that scenario specifically.
I have seen other related questions, but they did not solve my problem, or may be I somehow missed the exactly same resolved queries.
Here is the problem. The service that I call up returns a JSON response with some keys having large numbers as values, and I want to pass them on to my view and display. The issue is that they are getting rounded off, which I don't want. Actually its coming inside a buffer from which I am doing now:
JSON.parse(res.body.toString()) // res.body is a Buffer
and sending to view. How can I retain the whole number in the form of a string and send this to view so exactly the same is made available to UI.
I thought may be a replacer will help, but it does not works too.
const replacer = (key, value) => {
if (typeof value === 'number') {
return JSON.stringify(value);
}
return value;
};
//78787878977987787897897897123456786747398
const obj = {
name: 'John',
income: 78787878977987787897897897123456786747398,
car: null
};
var buf = Buffer.from(JSON.stringify(obj));
console.log(buf.toString());
// console.log(JSON.stringify(buf.toString()))
// console.log('func res: ', replacer('key', 78787878977987787897897897123456786747398))
// console.log(obj.income.toString())
console.log(JSON.stringify(obj, replacer));
You can recommend some external trusted library, or better, suggest me the solution through direct code only.
Edit:
The outcome in short is: Convert the response to String before returning from the server. Once it gets into JS (Buffer in my case), the conversion already occurred meaning that from the application side, nothing can be done to retrieve it.
Please let me know if there's a real solution to this without modifying server response.
Unfortunately, the number is higher than max_safe_integer, so if it ever gets parsed as a number, even if it's converted back to a string later (such as with the reviver function, the second parameter to JSON.parse), it won't be reliable. But luckily, since you have a JSON string, you can replace numeric values with string values before JSON.parseing it. For example:
const resBody = '{"foo":"bar", "objs":[{"name":"John", "income": 78787878977987787897897897123456786747398}]}';
const resBodyReplaced = resBody.replace(/: *(\d+)/g, ':"$1"');
console.log(JSON.parse(resBodyReplaced).objs[0].income);
I have 3 objects with the keys as it looks like this:
They are in format of YYYYMMDD. I am trying to get data of a month. But I am not getting the desired output.
When I query it like this:
var ref = db.child("-KPXECP6a1pXaM4gEYe0");
ref.orderByKey().startAt("20160901").once("value", function (snapshot) {
console.log("objects: " + snapshot.numChildren());
snapshot.forEach(function(childSnapshot) {
console.log(childSnapshot.key);
});
});
I get the following output:
objects: 3
20160822-KPl446bbdlaiQx6BOPL
20160901-KPl48ID2FuT3tAVf4DW
20160902-KPl4Fr4O28VpsIkB70Z
When I query this along with endAt:
ref.orderByKey().startAt("20160901").endAt("20160932").once("value", function (snapshot) {
console.log("objects: " + snapshot.numChildren());
snapshot.forEach(function(childSnapshot) {
console.log(childSnapshot.key);
});
});
I get this:
objects: 0
If I use ~ sign at the end,
ref.orderByKey().startAt("20160901").endAt("20160932~").once("value", function (snapshot) {
console.log("objects: " + snapshot.numChildren());
snapshot.forEach(function(childSnapshot) {
console.log(childSnapshot.key);
});
});
I get the output:
objects: 3
20160822-KPl446bbdlaiQx6BOPL
20160901-KPl48ID2FuT3tAVf4DW
20160902-KPl4Fr4O28VpsIkB70Z
Is there anything I am missing here?
Wow... this took some time to dig up. Thanks for the jsfiddle, that helped a lot.
TL;DR: ensure that you always have a non-numeric character in your search criteria, e.g. ref.orderByKey().startAt("20160901-").endAt("20160931~").
Longer explanation
In Firebase all keys are stored as strings. But we make it possible for developers to store arrays in the database. In order to allow that we store the array indices as string properties. So ref.set(["First", "Second", "Third"]) is actually stored as:
"0": "First"
"1": "Second"
"2": "Third"
When you get the data back from Firebase, it'll convert this into an array again. But it is important for your current use-case to understand that it is stored as key-value pairs with string keys.
When you execute a query, Firebase tries to detect whether you're querying a numeric range. When it thinks that is your intent, it converts the arguments into numbers and queries against the numeric conversion of the keys on the server.
In your case since you are querying on only a numeric value, it will switch to this numeric query mode. But since your keys are actually all strings, nothing will match.
For this reason I'd recommend that you prefix keys with a constant string. Any valid character will do, I used a - in my tests. This will fool our "is it an array?" check and everything will work the way you want it.
The quicker fix is to ensure that your conditions are non-convertible to a number. In the first snippet I did this by adding a very low range ASCII character to the startAt() and a very high ASCII character to endAt().
Both of these are workarounds for the way Firebase deals with arrays. Unfortunately the API doesn't have a simple way to handle it and requires such a workaround.
I'm trying to search a Parse.com field which is an array for a partial string.
When the field is in String format I can do the following:
// Update the filtered array based on the search text and scope.
// Remove all objects from the filtered search array
[self.searchResults removeAllObjects];
// Filter the array using NSPredicate
NSPredicate *predicate = [NSPredicate predicateWithFormat:#"SELF.busnumber contains[c] %#", searchText];
self.searchResults = [NSMutableArray arrayWithArray:[self.objects filteredArrayUsingPredicate:predicate]];
This works, however the new field I want to search in is an Array.
It works when I change the it to the following:
PFQuery * query = [PFQuery queryWithClassName:#"Bus"];
[query whereKey:#"route" equalTo:[NSString stringWithFormat:#"%#", searchText]];
[query findObjectsInBackgroundWithBlock:^(NSArray *objects, NSError *error) {
NSLog(#"Objects: %#", objects);
if (error)
{
NSLog(#"ERROR: %#", error.localizedDescription);
}
else
{
[self.searchResults removeAllObjects];
[self.searchResults addObjectsFromArray:objects];
[self.searchDisplayController.searchResultsTableView reloadData];
}}];
However I need the exact String for this.
I want to be able to search for parts of a string though, but when I change it to:
[query whereKey:#"route" containsString:[NSString stringWithFormat:#"%#", searchText]];
I get:
[Error]: $regex only works on string fields (Code: 102, Version: 1.7.4)
Any ideas? Thanks :)
What you've attempted is rational, but the string qualifiers on PFQuery work only on strings.
I've seen this theme frequently on SO: PFQuery provides only basic comparisons for simple attributes. To do anything more, one must query for a superset and do app level computation to reduce the superset to the desired set. Doing so is expensive for two reasons: app-level compute speed/space, and network transmission of the superset.
The first expense is mitigated and the second expense is eliminated by using a cloud function to do the app level reduction of the superset. Unless you need the superset records on the client anyway, consider moving this query to the cloud.
Specific to this question, here's what I think the cloud function would resemble:
// very handy to have underscore available
var _ = require('underscore');
// return Bus objects whose route array contains strings which contain
// the passed routeSubstring (request.params.routeSubstring)
Parse.Cloud.define("busWithRouteContaining", function(request, response) {
// for now, don't deal with counts > 1k
// there's a simple adjustment (using step recursively) to get > 1k results
var query = new Parse.Query("Bus");
query.find().then(function(buses) {
var matching = _.select(buses, function(bus) {
var route = bus.get("route");
var routeSubstring = request.params.routeSubstring;
return _.contains(route, function(routeString) {
return routeString.includes(routeSubstring);
});
});
response.success(matching);
}, function(error) {
response.error(error);
});
});
If you do decide to perform the reduction on the client and need help with the code, I can edit that in. It will be a pretty simple switch to predicateWithBlock: with a block that iterates the array attribute and checks rangeOfString: on each.