I need to call the oData service for the HTTP POST operation. Currently I send them as batches of 1000 by calling all the batches at a time. But there is a problem ,because of concurrent batches that the table gets locked. So I would want to send using $batch with the change set. But I am not sure how to call the Odata service with Changeset in Node.
Current Code for the batches.
const parallel = async(headers, input) => {
let payload = [],
items = [],
header ;
const sendreq = async (headers, request, i) => {
//sending request
}
for (var i = 0, len = input.ITEM.length; i < len; i += 1000) {
items.push(input.ITEM.slice(i, i + 1000));
}
for (i = 0; i < items.length; i++) {
header.ITEM= items[i];
payload.push(header)
header = {}
}
let j = 0;
const batches = Array(Math.ceil(input.ITEM.length / 1000)).fill(Array(1).fill(sendreq))
for (const batch of batches) {
await Promise.all(batch.map(f => f(headers, payload, j++)))
}
}
Related
The following loop to call an async function, here a smart contract interaction using web3. I want to get the balance of an array of token by calling balanceOf() and convert it subsequently with the attached usdrate. For parallel processing I am using Promise.all. Obviously, the function below Promise.all() with access [I % currency.length] does not work, since sorted result is not guaranteed.
My question is, how can I multiply the amounts with the correct usdrates attached to the tokens and still use Promise.all?
currencies = [{
contract: token1,
usdrate: 0.5
},
{
contract: token2,
usdrate: 1.0
},
{
contract: token3,
usdrate: 1.05
},
{
contract: token4,
usdrate: 1.10
},
{
contract: token5,
usdrate: 1.40
},
{
contract: token6,
usdrate: 1.0
},
{
contract: token7,
usdrate: 1.0
}
];
}
async function getUsdWealthAsync(addresses) {
var totalWealth = 0;
var amountPromises = [];
for (var j = 0; j < currencies.length; j++) {
for (var i = 0; i < addresses.length; i++) {
amountPromises.push(currencies[j].contract.methods.balanceOf(addresses[i]).call());
}
}
await Promise.all(amountPromises).then(function(amounts) {
for (var i = 0; i < amounts.length; i++) {
amounts[i] = Number.parseInt(amounts[i]);
totalWealth += (amounts[i] / 100) * currencies[i % currencies.length].usdrate;
}
})
return totalWealth;
}
async functions always return a Promise, you can define an async function that receives an address and a currency and returns a Promise that has the calculation done already, so you don't have index troubles. Something like
async function getAmount(currency, address) {
const amount = await currency.contract.methods.balanceOf(address).call();
return amount * currency.usdrate;
}
async function getUsdWealthAsync(addresses) {
const amountPromises = [];
for (const currency of currencies) {
for (const address of addresses) {
amountPromises.push(getAmount(currency,address)/*Remember, calling this funciton returns a Promise*/);
}
}
const realAmounts = await Promise.all(amountPromises)
return realAmounts.reduce((total,current) => total+current, 0);
}
Where the last line with the reduce call is supposed to sum all the amounts you have
Why not use a nested Promise.all() to bundle all of the asynchronous calls for a particular currency under a single Promise? By doing this, you also retain the index alignment for processing the response.
async function getUsdWealthAsync(addresses) {
let totalWealth = 0;
let amountPromises = [];
// For each of the currencies...
for (var j = 0; j < currencies.length; j++) {
// Create a set that will hold balance promises for this currency.
const balancePromisesForCurrency = [];
for (var i = 0; i < addresses.length; i++) {
// Create those promises and add them to the set.
balancePromisesForCurrency.push(
currencies[j].contract.methods.balanceOf(addresses[i]).call()
);
}
// Create a new promise that resolves to the list of balance results,
// index-aligned to the addresses, for this currency. Add that Promise
// to the set of per-currency Promises, index-aligned to the currencies
// array.
amountPromises.push(Promise.all(balancePromisesForCurrency));
}
// Create a new cumulative promise from the `amountPromises` array.
await Promise.all(amountPromises).then(function (amountsForCurrency) {
// For each of the balance lists received...
amountsForCurrency.forEach((amounts, amountsIndex) => {
// Get the corresponding currency.
const currency = currencies[amountIndex];
// Total up the balances scaled by the currency's USD rate.
amounts.forEach((amount, idx) => {
totalWealth += (+amount / 100) * currency.usdrate;
});
});
})
return totalWealth;
}```
You have other great answers.
Another way could be that, you can attach the USD rate along with the result from the balanceOf in the promise itself, And then while resolving the promises, you can access the USD rate directly.
Maybe something like this:
async function getUsdWealthAsync(addresses) {
var totalWealth = 0;
var amountPromises = [];
for (var j = 0; j < currencies.length; j++) {
for (var i = 0; i < addresses.length; i++) {
const { usdrate, contract } = currencies[j];
amountPromises.push(
contract.methods.balanceOf(addresses[i]).call()
.then((amount) => ({ amount, usdrate }))
);
}
}
const amounts = await Promise.all(amountPromises);
for (var i = 0; i < amounts.length; i++) {
const { amount, usdrate } = amounts[i];
amount = Number.parseInt(amount);
totalWealth += (amount / 100) * usdrate;
}
return totalWealth;
}
I'm trying to retrieve an array of cards for a project. However, in my function, the final contacts array returns an empty array.
I know that, because I have an async call to another funcion inside the forEach loop, the loop doesn't execute as intended. However, I'm very newbie when it comes to deal with this issues, so I want to ask you what's the best approach to deal with this.
This is my code:
export const extractsIDSForUser = async (currentUser: User) : Promise <Object> => {
let contactCards = currentUser.contacts;
const contacts = [];
const usersRef = await firebase.firestore().collection('Users').get();
const usersSnapshot = usersRef.docs.map(doc => doc.data());
contactCards.forEach(async folder => {
const ids = [];
folder.forEach(contact => {
ids.push(contact);
});
for (let i = 0; i < ids.length; i +=1) {
const contact = ids[i];
for (let j = 0; j < usersSnapshot.length; j += 1) {
const userId = usersSnapshot[j].id;
// Async call to function
const cardsFromUser = await extractCardsFromUser(userId);
const arrayCards = Object.values(cardsFromUser);
if (arrayCards.length > 0) {
for (let j = 0; j < arrayCards.length; j += 1) {
const arrayId = arrayCards[j].id;
const sameCardId = arrayId === contact;
if (sameCardId) {
// Where I insert the values into the array
contacts.push(arrayCards[j]);
}
}
}
}
}
});
// this is empty
return contacts;
}
What will be the best approach to deal with this?
I think you have already found a solution, but I had a similar problem and found this article quite helpful.
You could use a traditional for (const contactCard of contactCards) and it will work, but it will be less efficient than using a Promise.all approach.
I am hitting an endpoint using certain guids that I am receiving from a for loop. I would like to exclude some of the guids from hitting the endpoint.
async function main() {
const url = validateURL(process.argv[2]);
const origin = validatePath(process.argv[3]);
const debug = validateDebug(process.argv[4]);
let guid =
data = await getCategories(url, origin);
result = data["results"];
for (i = 0; i < result.length; i++) {
guid = result[i]["guid"];
await legacyEndpoint(url,guid);
}
}
main();
Is it possible to have a file with the list of guids I would like to ignore so when the loop is run it just leaves those guids ?
Thanks
async function main() {
const url = validateURL(process.argv[2]);
const origin = validatePath(process.argv[3]);
const debug = validateDebug(process.argv[4]);
let guid =
data = await getCategories(url, origin);
result = data["results"];
for (i = 0; i < result.length; i++) {
guid = result[i]["guid"];
if (guid !== "32666424" || "489429571658" || "6803523795" || "489429571658"){
await legacyEndpoint(url,guid);
} else
console.log("excluded");
}
}
When checking your guid against the unallowed list, you can use Array.includes JavaScript Array includes() Method to ensure your guid is not in the unallowed list.
Something like so
async function main() {
var unallowedGuids = ["32666424","489429571658","6803523795","489429571658"];
const url = validateURL(process.argv[2]);
const origin = validatePath(process.argv[3]);
const debug = validateDebug(process.argv[4]);
var data = await getCategories(url, origin);
var result = data["results"];
for (i = 0; i < result.length; i++) {
let guid = result[i]["guid"];
if (!unallowedGuids.includes(guid)){
await legacyEndpoint(url,guid);
}
else
console.log("excluded");
}
}
I'm new to JS asynchronous and I have a question about: how to start working on created array only if all queries are done. I fetch pages in for loop. That's my code:
var allOrgReposData = [];
var repoContributorsUrls = [];
for (var i=1; i <= orgPageIterations; i++) {
var orgReposUrl = 'https://api.github.com/orgs/angular/repos?page='+i;
fetch(orgReposUrl)
.then(response => response.json())
.then(orgReposData => {
allOrgReposData = allOrgReposData.concat(orgReposData);
console.log(allOrgReposData);
})
}
As You can see the allOrgReposData array is created on for loop, but If I try to do something on this array, script do It on every iteration so my operations are multipicated instead execution single time for exapmle (30 item on page): 30; 60; 90; 120; 150; 171 = 621 instead 171.
Is It possible to "wait" for finish fetching and get access to array without this. "multiplication"?
Greetings!
You can use Promise.all which will wait until all promises are complete:
var allOrgReposData = [];
var repoContributorsUrls = [];
var promises = [];
let orgPageIterations = 1;
for (var i = 1; i <= orgPageIterations; i++) {
let orgReposUrl = 'https://api.github.com/orgs/angular/repos?page=' + i;
promises.push(fetch(orgReposUrl).then(response => response.json()));
}
Promise.all(promises)
.then(data => {
allOrgReposData = data;
console.log(allOrgReposData);
})
.catch(err => console.error(err));
Please note that I've also changed var orgReposUrl to let orgReposUrl to make us of block scoping.
You could keep track of the number of calls you did with a variable :
var allOrgReposData = [];
var repoContributorsUrls = [];
var callSuccess = 1; //Variable keeping track of your ajax calls
for (var i=1; i <= orgPageIterations; i++) {
var orgReposUrl = 'https://api.github.com/orgs/angular/repos?page='+i;
fetch(orgReposUrl)
.then(response => response.json())
.then(orgReposData => {
allOrgReposData = allOrgReposData.concat(orgReposData);
console.log(allOrgReposData);
callSuccess++; //Increment your var for each call
if(callSuccess == orgPageIterations){ //If every call has already been made, then continue
//DO YOUR THING HERE
}
})
}
I have a ObservableArray.Using ajax async calls I am fetching data for binding.There will be 1000's of data. In each calls will fetch 100's of data. Problem is due to ajax async call order will not be from 1 - 1000. Cannot make it as sync call because browser will not respond. How can i sort the async data in knock out?
var DataVM = ko.observableArray([]);
ko.applyBindings(DataVM, document.getElementById("ControlBlock"));
for (var i = 0; i < totalAjaxCall; i++) {
GetData(guid, start, end, self.DataCallback);
start = start + 100;
end = end +100;
}
DataCallback= function (result) {
var temp = JSON.parse(result.d);
var data = [];
var data = temp.Data;
for (var j = 0; j < data.length; j++) {
var tempItem_ = new Item();
tempItem_.Number = data[j].Number;
// Other codes
DataVM.push(tempItem_ );
}
};
You can remember the blocks as they come in, then reassemble things when you've received them all. See comments:
var DataVM = ko.observableArray([]);
ko.applyBindings(DataVM, document.getElementById("ControlBlock"));
// Remember the results in a temporary array of arrays
var received = 0;
var receivedBlocks = [];
for (var i = 0; i < totalAjaxCall; i++) {
// Tell `DataCallback` which block it's going to get
GetData(guid, start, end, self.DataCallback.bind(null, i));
// No need for `self` ----^^^^^
start = start + 100;
end = end +100;
}
DataCallback = function (i, result) {
// Create and remember the items for this block
receivedBlocks[i] = JSON.parse(result.d).map(function(e) {
var tempItem_ = new Item();
tempItem_.Number = num;
return tempItem_;
});
++received;
// Do we have them all?
if (received == totalAjaxCall) {
// Flatten our array of arrays, now we have all the pieces
var receivedItems = []
receivedBlocks.forEach(function(block) {
receivedItems.push.apply(result, block);
});
// Push all of those onto DataVM as a single operation
// Note: You were using `DataVM.push`, so I used that here,
// but if you wanted to *replace* the contents of `DataVM`,
// (or if you know it's empty), you'd just do:
// DataVM(receivedItems);
// instead.
DataVM.push.apply(DataVM, receivedItems);
}
};
I'm not sure how far you want to deviate from your current code, but I'd like to advertise some of knockout's additional features :)
If you create a small "in-between" model for your requests, you can make use of computed values to automatically keep track of a correctly sorted list of data.
For example, if you define a new Request() like so:
var Request = function(start, end) {
this.completed = ko.observable(false);
this.data = [];
getData(start, end, this.onLoad.bind(this));
};
Request.prototype.onLoad = function(data) {
this.data = data;
this.completed(true);
};
You can change your for loop to create those "in-between" models. This creates a Request for 0 to 100, 101 to 201, etc. Each of these models is stored in an array, in the order of creation.
function getDataRequests(start, end, chunkSize) {
var requests = [];
for (var i = start; i < end; i += chunkSize) {
requests.push(new Request(i, Math.min(i + chunkSize, end)));
}
return requests;
};
Now that you can create an ordered array, you can compute another ordered array of data by merging all completed requests together:
var DataVM = function(start, end, chunkSize) {
// We keep track of a list of requests
var requests = ko.observableArray(
getDataRequests(start, end, chunkSize)
);
// Because requests have an observable completed prop,
// we can automatically keep track of a list of completed
// requests
var completedRequests = ko.pureComputed(() =>
requests().filter(r => r.completed()));
// Now, whenever a requests completes, we flatten the
// `data` parts for `completed` requests
this.data = ko.pureComputed(() => completedRequests()
.reduce((items, r) => items.concat(r.data), []));
};
Because you have the requests array, you can easily compute UI properties. For example: firstLoaded is a computed that returns the completed value of your first request.
Here's a complete example (ES2015):
var DataVM = function(start, end, chunkSize) {
// We keep track of a list of requests
var requests = ko.observableArray(
getDataRequests(start, end, chunkSize)
);
// Because requests have an observable completed prop,
// we can automatically keep track of a list of completed
// requests
var completedRequests = ko.pureComputed(() =>
requests().filter(r => r.completed()));
// Now, whenever a requests completes, we flatten the
// `data` parts for `completed` requests
this.data = ko.pureComputed(() => completedRequests()
.reduce((items, r) => items.concat(r.data), []));
// Shows progress
this.loadingMsg = ko.pureComputed(() => {
var completedCount = completedRequests().length,
allCount = requests().length;
return completedCount === allCount
? `Done loading ${end - start} items in ${allCount} steps`
: `Loading... (${completedCount}/${allCount})`;
});
// Check if the first (if any) request has completed loading
this.firstCompleted = ko.pureComputed(() =>
requests().length && requests()[0].completed());
};
var Request = function(start, end) {
this.completed = ko.observable(false);
this.data = [];
getData(start, end, this.onLoad.bind(this));
};
Request.prototype.onLoad = function(data) {
this.data = data;
this.completed(true);
};
var vm = new DataVM(0, 50, 5);
ko.applyBindings(vm);
// Mock async ajax stuff and data getters
function getDataRequests(start, end, chunkSize) {
var requests = [];
for (var i = start; i < end; i += chunkSize) {
requests.push(new Request(i, Math.min(i + chunkSize, end)));
}
return requests;
};
function getData(start, end, cb) {
setTimeout(function() {
cb(mockData(start, end));
}, Math.random() * 3000 + 500);
}
function mockData(from, to) {
return Array(to - from).fill(from).map(function(_, i) {
return from + i;
});
}
<script src="https://cdnjs.cloudflare.com/ajax/libs/knockout/3.2.0/knockout-min.js"></script>
<div data-bind="text: loadingMsg"></div>
<ul data-bind="foreach: data, visible: firstCompleted" style="border: 1px solid black;">
<li data-bind="text: $data"></li>
</ul>