Develop Reference

What's the proper way of collecting all the data from API with paging?

I'm trying to collect all the data at once from this specific API: https://docs.nuapay.com/v1/#list-mandates using JavaScript, node.js and Loopback framework.
However they are using paging and each page with response for mandates include only 20 results, and our API include over 700 pages.
In this case I'll have to do over 700 API calls to build the total result.
What's the best practice to recover results for all the pages without slowing down the process ?
Typical response for this API will be:
{
"uri" : "/schemes/46pkx7o9n5/mandates",
"data" : [ {
"id" : "46pkxn8e9n",
"uri" : "/schemes/8g3o2yyk2w/mandates/46pkxn8e9n",
"mandateId" : "BEL000049995012C",
"debtorName" : "Debtor Name",
"debtorIBAN" : "GB94SELN00999976543215",
"debtorMobileNumber" : "0360321312312",
"mandateStatus" : "PENDING",
"creationDate" : "2015-07-21"
}, {
"id" : "nx9kndaejm",
"uri" : "/schemes/8g3o2yyk2w/mandates/nx9kndaejm",
"mandateId" : "BEL000049995012S",
"debtorName" : "Debtor Name",
"debtorIBAN" : "GB94SELN00999976543215",
"debtorMobileNumber" : "0360321312312",
"mandateStatus" : "ACTIVE",
"creationDate" : "2015-07-21"
}, {
"id" : "vw7kj9gey5",
"uri" : "/schemes/8g3o2yyk2w/mandates/vw7kj9gey5",
"mandateId" : "BEL000049995012X",
"debtorName" : "Debtor Name",
"debtorIBAN" : "GB94SELN00999976543215",
"debtorMobileNumber" : "0360321312312",
"mandateStatus" : "ACTIVE",
"creationDate" : "2015-07-21"
} ],
"page" : {
"pageNumber" : 1,
"pageSize" : 20,
"totalElements" : 3,
"totalPages" : 1
},
"sort" : [ ]
}

Not getting the object key value using javascript

I am trying to read some key-value from one object using Javascript but one key value is not reading at all. I am explaining my code below.
customer={
"_id" : ObjectId("5e21bdb921a1de5a8885c368"),
"IsActive" : true,
"GroupName" : "ashok",
"CustomerId" : "7008980097Saikantassss#okedqart",
"GroupCode" : "ashokgrp",
"GroupType" : "Group",
"GroupDescription" : "test description",
"GroupImage" : "",
"StoreCode" : "DKWF",
"GroupID" : "1673e78b-8d35-4c5b-9bda-35d8788a64ad",
"CreatedAt" : ISODate("2020-01-17T13:59:21.373Z"),
"UpdatedAt" : ISODate("2020-01-17T13:59:21.373Z"),
"__v" : 0
}
Here I have the above fetched value from mongodb but when I am printing the customerId value like below.
console.log(customer['CustomerId']);
//undefined
The above value is coming undefined but same time console.log(customer['GroupCode']); giving the value as ashokgrp. Here I need to print also customer['CustomerId'] value.

var customer={
"_id" : "5e21bdb921a1de5a8885c368",
"IsActive" : true,
"GroupName" : "ashok",
"CustomerId" : "7008980097Saikantassss#okedqart",
"GroupCode" : "ashokgrp",
"GroupType" : "Group",
"GroupDescription" : "test description",
"GroupImage" : "",
"StoreCode" : "DKWF",
"GroupID" : "1673e78b-8d35-4c5b-9bda-35d8788a64ad",
"CreatedAt" : "2020-01-17T13:59:21.373Z",
"UpdatedAt" : "2020-01-17T13:59:21.373Z",
"__v" : 0
}
console.log(customer.CustomerId);
It Will work .

How to filter firebase data based on timestamp using startAt & endAt

I am trying to filter firebase data using startAt and/or endAt.
My data is structured as below.
{
"notes" : {
"-LOs0Ikx4ydM5RatREM1" : {
"data" : {
"dueDate" : 1561629600000,
"description" : "Korewa nan desuka?!",
"createdAt" : 1539611900000,
"title" : "First "
},
"members" : {
"author" : "1212121212121212121212121212"
}
},
"-LOs0Ikx4ydM5RatREM2" : {
"data" : {
"dueDate" : 4004870448000,
"description": "Test"
"createdAt" : 1539611900000,
"title" : "Second"
},
"members" : {
"author" : "1212121212121212121212121212"
}
},
"-LhBt9msLFKqUQ-koI9W" : {
"data" : {
"dueDate" : 1564653600000,
"description" : "abc",
"createdAt" : 1560363158279,
"title" : "August 1"
},
"members" : {
"author" : "3434343434343434343434343434"
}
},
"-LhBtKdDrQv9eKuYdfCi" : {
"data" : {
"dueDate" : 1564653600000,
"description" : "abcdef",
"createdAt" : 1560363158279,
"title" : "August 2"
},
"members" : {
"author" : "3434343434343434343434343434"
}
}
}
}
What I wish is to fetch all "notes" where dueDate has passed.
const now = moment().valueOf() //Eg. 1561629500000
database.ref('notes/')
.orderByChild("dueDate")
.endAt(now)
.once("value", (snapshot) => {
console.log('Process expired notes')
snapshot.forEach( (data) => {
const obj = data.val()
console.log('Date comparison:', (now >= obj.data.alertDate))
...
The code above does not work, it returns all the objects from the example JSON. The console.log logs "False" for three out of four returned objects.
I could do a comparison and only process the objects that meets my criteria, but that would defeat the purpose.
I have indexed the database on ["notes\data\alertDate"].
What am I missing? I must have misinterpreted the documentation somehow. :)

Your dueDate property is nested under data, so you need to address is as data/dueDate:
database.ref('notes')
.orderByChild("data/dueDate")
You might want to include both a startAt() and endAt() clause, with just a reasonable value for startAt() and the specific value you're already using for endAt().

Find and replace only certain fields in a document

I have a mongo document like so:
{
"_id" : "EimL8Hf5SsCQKM7WP",
"enable" : "no",
"collector" : [
{
"name" : "prod",
"enable" : "no",
"transport" : {
"address" : {},
"port" : "8000",
"protocol" : "http-secure"
},
"authentication" : {},
"notify-syslog" : "yes"
},
{
"name" : "test",
"enable" : "no",
"transport" : {
"address" : {},
"port" : "8000",
"protocol" : "http-secure"
},
"authentication" : {},
"notify-syslog" : "yes"
}
],
"misc" : {
"createdBy" : "someuser",
"updatedBy" : "someuser",
}
}
I display certain fields of the mongo document as an autoform to the user so that they can be edited. Since the other fields are hidden, the outcome of the form will only contain the fields that were exposed.
I end up with a modified document object like so: (keep an eye on the "name" fields)
{
"enable" : "no",
"collector" : [
{
"name" : "someohername",
},
{
"name" : "anothername",
}
],
"misc" : {
"createdBy" : "someuser",
"updatedBy" : "anotheruser",
}
}
My current requirement is to somehow merge both the documents together such that the initial document has all fields of the modified document, along with it's fields.
Outcome:
{
"_id" : "EimL8Hf5SsCQKM7WP",
"enable" : "no",
"collector" : [
{
"name" : "someothername",
"enable" : "no",
"transport" : {
"address" : {},
"port" : "8000",
"protocol" : "http-secure"
},
"authentication" : {},
"notify-syslog" : "yes"
},
{
"name" : "anothername",
"enable" : "no",
"transport" : {
"address" : {},
"port" : "8000",
"protocol" : "http-secure"
},
"authentication" : {},
"notify-syslog" : "yes"
}
],
"misc" : {
"createdBy" : "someuser",
"updatedBy" : "anotheruser",
}
}
If you notice above, only the name field has been updated whilst retaining the remaining fields.
I tried to directly update the document using collection.update({id}, $set:{<<modified_docuemnt}}); but it is replacing the entire collector field with what is passed in the $set
How exactly do I go about this? It does not matter if I'm doing the changes directly in mongodb or as a JSON object using javascript.

MongoDB - Updating only $ref from DBRef field type

I need to update a field of multiple collection documents. The field is a DBRef and I just need to change the $ref field value.
One of the documents is like this:
{ "_id" : { "$oid" : "50ab682bd3155502a75c7cf6"} , "codeId" : { "$ref"
: "version" , "$id" : { "$oid" : "511cb7d5696bdbaf4c85ebb1"}}}
The final result I want is this:
{ "_id" : { "$oid" : "50ab682bd3155502a75c7cf6"} , "codeId" : { "$ref"
: "code" , "$id" : { "$oid" : "511cb7d5696bdbaf4c85ebb1"}}}
I've tried it like this:
db.collection.update(
{},
{$set:{"codeId":{$ref:"code"}}},
false,
true
);
The problem is that the $id is lost (set as null)
{ "_id" : { "$oid" : "50ab682bd3155502a75c7cf6"} , "codeId" : { "$ref"
: "code" , "$id" : { "$oid" : null}}}
How do I keep the $id?
Thanks

You can use dot notation in order to update the nested document. Note that, whenever the dot notation is used, it should be put between quotes.
db.collection.update(
{},
{ $set:{
"codeId.$ref":"code"
}
},
false,
true
);
You can read more about Database References.