1
I have route like this:
router.get('/test', async(req, res)=>{
});
i used to use request module to perform the http calls.
But now, I want to call multiple http calls and combine the responses into single JSON object
How to do it?
With a method marked as async you can use the await keyword to make the http calls synchronous, ie wait for the resonse.
router.get('/test', async(req, res)=>{
// first call
var result1 = await fetch('https://jsonplaceholder.typicode.com/users');
var resultJson1 = await result1.json();
var resultObject1 = JSON.parse(resultJson1);
// second call
var result2 = await fetch('https://jsonplaceholder.typicode.com/posts');
var resultJson1 = await result1.json();
var resultObject2 = JSON.parse(resultJson2);
var combinedResult = resultObject1.fieldName + resultObject2.anotherFieldName
res.send(JSON.stringify(combinedResult));
});
Related
I'm currently looking to test an async function, a scraper that scrapes Ryan Air's website for a price on a given route to be exact. And I want to test that the scraped price is actually what the price should be. When trying to run it with jest to test, I cannot seem to make it work properly... I've looked on Google and various other sites and they all seem to have solutions for async functions that have callbacks, promises, etc. and NOT async functions that don't have those.
My function takes as a parameter the URL of a given route on Ryan Air.
Here is my async function (file is named scraperProduct.js):
const puppeteer = require('puppeteer');
async function scraperProduct(url){
console.log('Starting scraper...');
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto(url);
await page.waitFor(500);
//Price departure
const [el1] = await page.$x('/html/body/flights-root/div/div/div/div/flights-summary-container/flights-summary/div/div[1]/journey-container/journey/div/div[2]/carousel-container/carousel/div/ul/li[3]/carousel-item/button/div[2]/ry-price/span[2]');
const txt = await el1.getProperty('textContent');
const Price = await txt.jsonValue();
//Price return
const [el6] = await page.$x('/html/body/flights-root/div/div/div/div/flights-summary-container/flights-summary/div/div[2]/journey-container/journey/div/div[2]/carousel-container/carousel/div/ul/li[3]/carousel-item/button/div[2]/ry-price/span[2]');
const txt6 = await el6.getProperty('textContent');
const Price2 = await txt6.jsonValue();
return Price + Price2;
}
module.exports = scraperProduct;
And this is my test file (named scraperProduct.test.js):
const scraperProduct = require('./scraperProduct');
test("Testing that scraper retrieves correct price from Ryan Air", async () => {
expect(
scraperProduct('https://www.ryanair.com/dk/da/trip/flights/select?adults=1&teens=0&children=0&infants=0&dateOut=2020-07-13&dateIn=2020-07-20&originIata=CPH&destinationIata=STN&isConnectedFlight=false&isReturn=true&discount=0')
).toBe(698);
});
'toBe(698)' is 698 since that is what the price should be in the test.
I appreciate any help I can get with this - it's my first time using jest, so I'm a bit of a noob atm.
Since you are trying to test an async function, you need to wait for the result of that function i.e use await.
This is one of the possible solutions when testing asynchronous code. Wait for the result and then test it.
const scraperProduct = require('./scraperProduct');
test("Testing that scraper retrieves correct price from Ryan Air", async () => {
const result = await scraperProduct('https://www.ryanair.com/dk/da/trip/flights/select?adults=1&teens=0&children=0&infants=0&dateOut=2020-07-13&dateIn=2020-07-20&originIata=CPH&destinationIata=STN&isConnectedFlight=false&isReturn=true&discount=0')
expect(result).toBe(698);
});
I am trying to set a global variable within a function, but the code continues before the variable has been updated.
e.g.
var username = 'Example';
const fetch = require('node-fetch');
var num = 1234;
var uuidP;
const request = async () => {
const response = await fetch(`https://api.mojang.com/users/profiles/minecraft/${username}`);
const json = await response.json();
uuidP = json.id;
}
request();
console.log(num); //returns 1234
console.log(uuidP); //returns udefined
Javascript is heavily optimised. You need to declare the update() function is asynchronous, and then use a Promise to await the response of the update. Have a look at this example.
Will this code send more request
const pr1 = request1();
const pr2 = request2();
const pr3 = request3();
await Promise.all([pr1, pr2, pr3])
const res1 = await pr1
const res2 = await pr2
const res3 = await pr3
than
const pr1 = request1();
const pr2 = request2();
const pr3 = request3();
const [res1, res2, res3] = await Promise.all([pr1, pr2, pr3])
What is the most efficient way to send multiple async requests using nodejs?
Will this code send more request
No.
It sends three requests.
Then it waits for all of them to resolve.
Then it waits for each of them to resolve (which takes no time because they have resolved already) assigning them to variables as it goes.
Waiting for something to resolve does not start it from scratch.
What is most efficient way to send multiple async requests using nodejs?
I would be astonished if there were any practical performance differences between the two versions of the code.
The second version, which uses destructuring, is (subjectively) easier to read.
Write for the people maintaining your code (which is will often be you in 6 months time). Giving them code that is easy to maintain is going to save you more than trying to deal with microoptimisations.
If every request is independent of other, you can use await Promise.all([]) as above mentioned. But if you need result of the previous request, then you need to go await for each request.
const [res1, res2, res3] = await Promise.all([pr1, pr2, pr3])
With Promises:
var promise1 = Promise.resolve(3);
var promise2 = 42;
var promise3 = new Promise(function(resolve, reject) {
setTimeout(resolve, 100, 'foo');
});
Promise.all([promise1, promise2, promise3]).then(function(values) {
console.log(values);
});
With Async/ Await:
let [foo, bar] = await Promise.all([getFoo(), getBar()]);
const pr1 = request1();
const pr2 = request2();
const pr3 = request3();
const [res1, res2, res3] = await Promise.all([pr1, pr2, pr3])
This is the best way, this will trigger 3 parallel requests instead of sequential flow.
I am trying to make a script that :
Grabs all urls from a sitemap
Takes a screenshot of it with puppeteer
I am currently trying to understand how to code asynchronously but I still have troubles with finding the right coding pattern for this problem.
Here is the code I currently have :
// const spider = require('./spider');
const Promise = require('bluebird');
const puppeteer = require('puppeteer');
const SpiderConstructor = require('sitemapper');
async function crawl(url, timeout) {
const results = await spider(url, timeout);
await Promise.each(results, async (result, index) => {
await screen(result, index);
});
}
async function screen(result, index) {
const browser = await puppeteer.launch();
console.log('doing', index);
const page = await browser.newPage();
await page.goto(result);
const path = await 'screenshots/' + index + page.title() + '.png';
await page.screenshot({path});
browser.close();
}
async function spider(url, timeout) {
const spider = await new SpiderConstructor({
url: url,
timeout: timeout
});
const data = await spider.fetch();
console.log(data.sites.length);
return data.sites;
};
crawl('https://www.google.com/sitemap.xml', 15000)
.catch(err => {
console.error(err);
});
I am having the following problems :
The length of the results array is not a constant, it varies every time I launch the script, which I guess resides in the fact it is not fully resolved when I display it, but I thought the whole point of await was so that we are guarantied that on next line the promise is resolved.
The actual screenshotting action part of the script doesn't work half the time and I am pretty sure I have unresolved promises but I have no of the actual pattern for looping over an async function, right now it seems like it does a screenshot after the other (linear and incremental) but I get alot of duplicates.
Any help is appreciated. Thank you for your time
To optimize the response delay, it is necessary to perform work after are response has been sent back to the client. However, the only way I can seem to get code to run after the response is sent is by using setTimeout. Is there a better way? Perhaps somewhere to plug in code after the response is sent, or somewhere to run code asynchronously?
Here's some code.
koa = require 'koa'
router = require 'koa-router'
app = koa()
# routing
app.use router app
app
.get '/mypath', (next) ->
# ...
console.log 'Sending response'
yield next
# send response???
console.log 'Do some more work that the response shouldn\'t wait for'
Do NOT call ctx.res.end(), it is hacky and circumvents koa's response/middleware mechanism, which means you might aswell just use express.
Here is the proper solution, which I also posted to https://github.com/koajs/koa/issues/474#issuecomment-153394277
app.use(function *(next) {
// execute next middleware
yield next
// note that this promise is NOT yielded so it doesn't delay the response
// this means this middleware will return before the async operation is finished
// because of that, you also will not get a 500 if an error occurs, so better log it manually.
db.queryAsync('INSERT INTO bodies (?)', ['body']).catch(console.log)
})
app.use(function *() {
this.body = 'Hello World'
})
No need for ctx.end()
So in short, do
function *process(next) {
yield next;
processData(this.request.body);
}
NOT
function *process(next) {
yield next;
yield processData(this.request.body);
}
I have the same problem.
koa will end response only when all middleware finish(In application.js, respond is a response middleware, it end the response.)
app.callback = function(){
var mw = [respond].concat(this.middleware);
var gen = compose(mw);
var fn = co.wrap(gen);
var self = this;
if (!this.listeners('error').length) this.on('error', this.onerror);
return function(req, res){
res.statusCode = 404;
var ctx = self.createContext(req, res);
onFinished(res, ctx.onerror);
fn.call(ctx).catch(ctx.onerror);
}
};
But, we can make problem solved by calling response.end function which is node's api:
exports.endResponseEarly = function*(next){
var res = this.res;
var body = this.body;
if(res && body){
body = JSON.stringify(body);
this.length = Buffer.byteLength(body);
res.end(body);
}
yield* next;
};
you can run code in async task by use setTimeout, just like:
exports.invoke = function*() {
setTimeout(function(){
co(function*(){
yield doSomeTask();
});
},100);
this.body = 'ok';
};