Develop Reference

How do I refactor this series of API calls to execute faster?

I have a series of API calls I need to make in order to render a grid of image tiles for selection by the user. Right now it takes 3-5 seconds for the page to load and I think it's because I've accidentally added some extra loops, but I'm struggling to discern where the wasted flops are. This is technically a question about NFT data, but the problem is algorithmic not crypto related.
The call sequence is:
Call "Wallet" API to get all assets associated with an address - API doc
On success, call "Asset Metadata" API to get further info about each asset API Doc
Loop step 2 until all assets have a metadata response
This is my code that works (unless there is no assets associated with a wallet), but is just very slow. I'm sure there is a better way to handle this, but I'm struggling to see how. Thanks for your time!
// API Request
var myHeaders = new Headers();
myHeaders.append("X-API-Key", CENTER_API_KEY); //API Key in constants file
var requestOptions = {
method: 'GET',
headers: myHeaders,
redirect: 'follow'
};
const [nftData, updatenftData] = useState();
const [apiState, updateapiState] = useState("init");
const [renderNFT, updaterenderNFT] = useState([]);
useEffect(() => {
const getData = async () => {
let resp = await fetch(walletAPICall, requestOptions);
let json = await resp.json()
updatenftData(json.items);
updateapiState("walletSuccess");
}
const getRender = async () => {
let nftTemp = [];
for (let i=0;i<nftData.length;i++) {
let tempAddress = nftData[i].address;
let tempTokenId = nftData[i].tokenId;
let resp = await fetch(`https://api.center.dev/v1/ethereum-mainnet/${tempAddress}/${tempTokenId}`, requestOptions)
let json = await resp.json()
// console.log(json);
nftTemp.push(json);
}
updaterenderNFT(nftTemp);
updateapiState("NftDataSuccess");
}
if (apiState=="init") {
getData();
}
else if (apiState=="walletSuccess") {
getRender();
}
}, [requestOptions]);

getRender fetches data items sequentially.
You should do it in parallel using Promise.all or Promise.allSettled
Something like this...
function fetchItem(item) {
const res = await fetch(item.url);
return res.json();
}
await Promise.all[...data.map(fetchItem)]

How to migrate request-promise to axios or fetch

I want to code a app with React-Native which loads JSON-files from a website with cookie-authentication.
For testing I tried it in a normal JS-file without React-native and with request-promise.
const fs = require("fs");
const request = require("request-promise").defaults({ jar: true });
async function main() {
var incodeHeader = "";
var incodeToken = "";
try {
const loginResult = await request.post("https://somepage/login.php", {
form: {
client: "XXX",
login: "username",
password: "password",
},
});
} catch (err) {
console.log(err);
}
incodeHeader = getIncodeHeader();
incodeToken = getIncodeToken();
const data = await request.post("https://somepage/load.json", {
headers: {
[incodeHeader]: incodeToken,
},
form: {
max: "10",
},
});
fs.writeFileSync("data.json", data);
}
main();
This worked well, so I wanted to use this method in my App, but I couldn't find a way to use request-promise in React-Native so I decided to use axios.
const axios = require("axios");
const qs = require("qs");
axios.defaults.withCredentials = true;
async function main() {
const data = {
client: "XXX",
login: "username",
password: "password",
};
await axios
.post("https://somepage/login.php", qs.stringify(data))
.catch((err) => console.log(err));
const incodeHeader = getIncodeHeader();
const incodeToken = getIncodetoken();
await axios
.get(
"https://somepage/load.json",
{ data: { max: "5" } },
{
headers: {
[incodeHeader]: incodeToken,
},
}
)
.then((respone) => console.log(respone))
.catch((err) => console.log(err));
}
main();
But in this code not even the login works and I really don't know why. Can somebody tell me how to do this right, or can tell me another solution which works in React-Native?

First, I don't know why you're stringifying the request body in the first request, axios already handle this, you can pass just the data object, maybe it's the solution for your problem.
Second (just a tip). Create a helper object to make http requests and do not instance axios directly, so then, you can change the http request handler in an easy way instead changing it on each file, one day you probably will need to do this if you want to keep your app updated.
Third, don't mix await and then, choose:
try {
const result = await action();
// ...
} catch (err) {
// ...
}
or
action()
.then((result) => {
// ...
})
.catch((err) => {
// ...
});

change await axios.get to await axios.post

Using promises in async function

I'm trying to listen for a Stripe webhook call, then carry out some actions such as sending an email. My site is on Netlify and I've adapted some code I found in a tutorial:
This works locally, but not when I run it as a Netlify function (basically a lambda). Basically, the part from "client.getSpace.." doesn't appear to run at all. I suspect this is something to do with using these .then promises within an async function, but I'm not sure.
require('dotenv').config();
const stripe = require('stripe')(process.env.STRIPE_SECRET_KEY);
const contentful = require('contentful-management');
const client = contentful.createClient({
accessToken:
process.env.CONTENTFUL_CONTENT_MANAGEMENT_ACCESS_TOKEN
});
var postmark = require("postmark");
var serverToken = process.env.POSTMARK_SERVER_TOKEN;
var postmark_client = new postmark.ServerClient(serverToken);
exports.handler = async ({ body, headers }) => {
try {
const stripeEvent = stripe.webhooks.constructEvent(
body,
headers['stripe-signature'],
process.env.STRIPE_WEBHOOK_SECRET
);
if (stripeEvent.type === 'checkout.session.completed') {
console.log('confirm checkout session completed');
const eventObject = stripeEvent.data.object;
const entryId = eventObject.client_reference_id;
let email = "";
let secret = "";
client.getSpace(process.env.WEBSITE_CONTENTFUL_SPACE_ID)
.then(space => space.getEnvironment('master'))
.then(environment => environment.getEntry(entryId))
.then(entry => {
entry.fields.paymentStatus['en-GB'] = 'Paid';
email = entry.fields.email['en-GB'];
return entry.update();
})
.then(entry => entry.publish())
.then(entry => postmark_client.sendEmailWithTemplate({
"From": "x#test.com",
"To": email,
"TemplateId": 12345678,
"TemplateModel": {
"abc": "xyz"
}
}))
.catch(console.error)
}
return {
statusCode: 200,
body: JSON.stringify({ received: true }),
};
} catch (err) {
console.log(`Stripe webhook failed with ${err}`);
return {
statusCode: 400,
body: `Webhook Error: ${err.message}`,
};
}
};

For what it's worth to you and anyone else who comes across this question. I had a similar issue using NextJS on Vercel. I rewrote the .then syntax using async/await and the problem seems to be solved, so far. I'm no expert, but I think in this case you would begin by replacing
client.getSpace(process.env.WEBSITE_CONTENTFUL_SPACE_ID)
.then(space => space.getEnvironment('master'))
with something like
const send = await client.getSpace(process.env.WEBSITE_CONTENTFUL_SPACE_ID)
const getEnvironment = await space.getEnvironment('master')
so on and so forth. I'm not sure how you would rewrite everything else, or if this will even help, but it put me back on track.

How to proxy specific request in Puppeteer

is there a way to proxy / redirect specific urls to others?
for example when Puppeteer page goes to "mydomain.com" i'd like all calls to "mydomain.com/styles/.css" be proxied to "localhost:8080/styles/.css".
I don't want all request to be rediret through a proxy. but something similar to what https://chrome.google.com/webstore/detail/resource-override/pkoacgokdfckfpndoffpifphamojphii?hl=en chrome extension does.

as #Hellonearthis linked, the soulution seems to be
const page = await browser.newPage();
await page.setRequestInterception(true);
page.on('request', (request) => {
if (request.url().indexOf("mydomain.com") !== -1) {
// simply replace with another url
request.continue((request.url().replace("mydomain.com", "http://localhost:8080/styles"));
}
else {
request.continue();
}
});

For puppeteersharp, it took me a while to figure it out.
page = await ChromeDriver.NewPageAsync();
await page.SetRequestInterceptionAsync(true);
page.Request += new EventHandler<RequestEventArgs>(async delegate (object o, RequestEventArgs a)
{
if (a.Request.Url.StartsWith("mydomain.com"))
{
await a.Request.ContinueAsync(new Payload
{
Headers = a.Request.Headers,
Method = a.Request.Method,
PostData = a.Request.PostData != null ? a.Request.PostData.ToString() : "",
Url = a.Request.Url.Replace("mydomain.com", "http://localhost:8080")
});
}
else
{
await a.Request.ContinueAsync();
}
});

how to use Firebase Cloud Functions with promises and forEach?

I am trying to do TWO things here.
1)Send a notification to all employees. 2)Copy a Specific ref to the
Employees id ref. if no Special ref exists i will copy General ref.
The program runs without errors. Infact its perfect. But sometimes i get a Timed out error with the Notifications code part.
Error: fcm.googleapis.com network timeout. Please try again.
The code that copys one reference to another, always works, never ever received an error there.
I feel the error is due to not correctly handling promises with forEach. Could you help me get this code to excecute flawlessly, with correctly placed Promises?
exports.myFunc = functions.https.onRequest( (request, response) => {
admin.database().ref('/Employees').once('value').then(function(snap) {
snap.forEach(function (snapshot) {
var obj = snapshot.val();
if(obj.department){//only go ahead if dept is present
console.log(' : ' + obj.department);
var id, tkid, dept;
id = obj.empId; tkid = obj.tokenId; dept = obj.department;
var welcomeStr="hello! Welcom to our Department";
//================================================================notifications
var payload = {
data: {
greeting: welcomeStr,
to_who: id
}
};
admin.messaging().sendToDevice(tkid,payload)
.then(function(response){
console.log("Successfully sent message: ", response);
})
.catch(function(error){
console.log("Error sending message: ", error);
})
//===================================================Ref copying
var destinationRef = admin.database().ref('/Employees/' + id);//final destination
var option2Ref = admin.database().ref('/Company/General');//when special doesnt exist
var option1Ref = admin.database().ref('/Company/Special');//if special exists
option1.once('value', function(snapshot1){
if (snapshot1.exists()){//copy from straing from option11 to Employees/id
option1.once('value', function(snap) {
destinationRef.set( snap.val(), function(error) {
if( error && typeof(console) !== 'undefined' && console.error ) { console.error(error); }
console.log('DONE .... ' + id);
});
});
}
else{//we need to copy from option2 to Employees/id
option2Ref.once('value', function(snap) {
newRef.set( snap.val(), function(error) {
if( error && typeof(console) !== 'undefined' && console.error ) { console.error(error); }
console.log('DONE .... ' + id);
});
});
}
});
}
else{
console.log('No Department: ' + obj.dept);
return;
}
});
});
response.send("WOKAY!");
});

here i've rewritten your code in hopes to clean up the complicated promise chains
dropped promises are one of the most common and difficult problems to debug, i've seen my fair share
important changes to your code:
modern async syntax
so that the promises are cleaner to organize
use Promise.all instead of forEach
this way every promise is awaited without being forgotten
(hopefully) all of the promises are returned properly
all snapshot operations are run concurrently, and the onRequest handler should wait until they're all finished before terminating
using promises for once and set instead of callbacks
i'm not really sure what libraries these are
it looks like they accept promise-based usage
so i eliminated callback usage in favor of promises
please review the TODO mark
not really sure what's intended for that else block, so be sure to patch that up
async function handleSnapshot(snapshot) {
const {empId, tokenId, department} = snapshot.val()
// only go ahead if dept is present
if (!department) throw new Error("no department")
console.log("department:", department)
//================================================================notifications
const payload = {
data: {
greeting: "Hello! Welcome to our Department",
to_who: empId
}
}
const response = await admin.messaging().sendToDevice(tokenId, payload)
console.log("successfully sent message", response)
//===================================================Ref copying
const destinationRef = admin.database().ref('/Employees/' + empId) // final destination
const option2Ref = admin.database().ref('/Company/General') // when special doesnt exist
const option1Ref = admin.database().ref('/Company/Special') // if special exists
const snapshot1 = await option1Ref.once("value")
// copy from string from option1 to Employees/id
if (snapshot1.exists()) {
await destinationRef.set(snapshot1.val())
console.log("DONE1...", empId)
}
// TODO review this block
// we need to copy from option2 to Employees/id
else {
const snapshot2 = await option2Ref.once("value")
await destinationRef.set(snapshot2.val())
console.log("DONE2...", empId)
}
}
exports.myFunc = functions.https.onRequest(async(request, response) => {
const snapshots = await admin.database().ref('/Employees').once('value')
await Promise.all(snapshots.map(handleSnapshot))
response.send("WOKAY!")
})

To add one very important step to #ChaseMoskal answer.
For those using TypeScript with Firebase, since firebase server is not running v8+ in NodeJs, theres a great chance you might get this error:
"TypeError: snapshots.map is not a function"... on the line: await Promise.all(snapshots.map(handleSnapshot)).
Thats cause in your tsconfig.json its possibly "lib": ["es6"]. In that case just add this small snippet to the accepted answer, to get the Firebase Datasnapshot into an array that could be used with .map(...)
Longer Version:
exports.myFunc = functions.https.onRequest(async(request, response) => {
const snapshots = await admin.database().ref('/Employees').once('value')
var data_snap_arr = [];
snapshots.forEach(function(child_Snapshot) {
var stuff = child_Snapshot.val();
stuff.key = child_Snapshot.key;
data_snap_arr.push(stuff);
await Promise.all(data_snap_arr.map(handleSnapshot))
response.send("WOKAY!")
})
Shorter Version:
exports.myFunc = functions.https.onRequest(async(request, response) => {
const snapshots = await admin.database().ref('/Employees').once('value')
let data_snap_arr = Object.keys(snapshots.val() || {}) .map(k => snapshots.val()[k]);
await Promise.all(data_snap_arr.map(handleSnapshot))
response.send("WOKAY!")
})