I am creating an HTTP-callable function that makes contact with an API. However, I am getting an error with sending back a 200. I think it has something to do with a mistake I made in using asynchronous functions. Here is my code.
exports.organisationDataToTemp = functions.region('europe-west3').https.onRequest((req, res) => {
res.set('Access-Control-Allow-Origin', '*');
const GETparam = req.query.kvk;
const KvK = GETparam.toString();
//Test if KvK number is already in temp collection
const snapshot = db.collection('temp').where('information.kvkNumber', '==', KvK).get()
.then(function(querySnapshot) {
querySnapshot.forEach(function(doc) {
//This is where it needs to send the header and it fails. I do get here only when I need this code to run, but it still fails
console.log('kvk number already in temp collection');
res.status(200).send(doc.id);
return;
});
});
//Irrelevant code
//Make API call using the provided KvK number
const keyName = 'VitozFMIS';
const API_key = 'sdfghjqwertyuiopdfghytrdcvbjftyujnbvc';
//Call the first JSON
const _EXTERNAL_URL = 'https://api.kvk.nl/api/v2/testprofile/companies?kvkNumber=' + KvK + '&' + keyName + '=' + API_key + '&startPage=1';
fetch(_EXTERNAL_URL)
.then(response => response.json())
.then(data => {
const total = data.data.totalItems;
for(n=1;n<=total;n++){
const API = 'https://api.kvk.nl/api/v2/testprofile/companies?kvkNumber=' + KvK + '&' + keyName + '=' + API_key + '&startPage=' + n;
//irrelevant code
//Make the API call
fetch(API)
.then(resp => resp.json())
.then(data => {
//irrelevant code
});
}
});
//Return 200 if no errors occured
res.status(200).send(cleanupID);
return;
});
Normally, the code runs exactly as needed, but when the kvk number is already in the collection, it needs to send the document ID back in a 200. I am not sure, but I think it is because it sends another code before this one, but I do not understand why it fails. Does someone know what fails?
This line of code is always going to immediately return a 200 response, before anything else happens:
res.status(200).send(cleanupID);
That's because the Firestore and fetch APIs you're using are asynchronous and return immediately. The callbacks you provide execute some time later, after the results are available. Calling then does not block your code from continuing - it just establishes a callback to be run when the promise resolves.
Cloud Functions requires that sending the response must be the very last thing your function does, so you should wait until all of the work fully succeeds or fails before sending the response.
You will need to structure your code to use the callbacks from the promises to decide what to do. All of the logic much be inside the callback, or deferred to another callback by returning another promise. I strongly suggest finding a tutorial to learn how this works. It's essential for writing effective JavaScript. It's also highly valuable to learn async/await syntax to make promises easier to work with.
Related
The following function, when consumed, fetches and returns an array of up to 50 comments of a single post. It first fetches a single post by ID, and this post object has an array of comments ID's, which it will fetch.
My goal is to early abort this task in my React class component using componentWillUnmount, by calling abort on the signal on the class instance. The signal is passed as abortSignal.
The question is, I have a case of nested fetch requests. How should I approach this so I can make sure any on-going fetch requests are aborted when the component is unmounting? As I seee it, passing the signal to the outer fetch won't suffice if this stage was already completed. Should I create another signal inside of this function, and pass to to individual fetch?
const fetchComments = async (type, abortSignal) => {
const res = await fetch(endpoints[`${type}Stories`]);
const post = await res.json();
return Promise.all(post.slice(0, 50).map(async id => {
const url = endpoints.singleStory.replace('[id]', id);
const comment = await fetch(url);
return comment.json();
}));
}
I don't see any reason you can't reuse the abortSignal across all the fetch calls. You probably also want to check its aborted flag after awaiting so you bail proactively:
const fetchComments = async (type, signal) => {
const res = await fetch(endpoints[`${type}Stories`], {signal});
if (signal.aborted) {
return; // Or whatever
}
if (!res.ok) {
throw new Error("HTTP error " + res.status);
}
const post = await res.json();
if (signal.aborted) {
return; // Or whatever
}
return Promise.all(post.slice(0, 50).map(async id => {
// Probably not much point to checking `signal.aborted` here, you *just* checked it above
const url = endpoints.singleStory.replace('[id]', id);
const comment = await fetch(url, {signal});
if (!comment.ok) {
throw new Error("HTTP error " + comment.status);
}
return comment.json();
}));
}
Two notes on that:
I changed abortSignal to signal so I could use shorthand property notation when passing it to fetch, because I'm lazy. :-)
Your code is falling prey to the fetch footgun — you need to check for HTTP success, it only rejects on network error, not HTTP error. I've inserted checks above.
Note: If you can modify the API, I'd strongly recommend making it possible to ask for the 50 comments as part of the initial fetch, or at least to be able to ask for a batch of comments, rather than loading each comment individually with its own HTTP request (although HTTP/2 helps a lot).
Im' trying to understand using promises with Google Cloud Functions a bit better. I just learned about the 'finally' method on promises, which is called after all promises in the chain are fully resolved or rejected. In a http function is it good practice to put response.send() inside of the finally method?
The below code uses request-promise-native for the http request. In the first .then() I call parseSchedule, which uses the cheerio web scraping api to loop through some data and on a website, and add it to the scheduledGames array (synchronously, I think).
I return from that and the then log that data to the console in writeDB, but one thing I noticed is that I see response.send() log 'execution finished' before I see the data from scheduleGames in the log. Is that correct?
Should I be using the 'finally' block like this?
Thanks,
const options = {
uri: 'https://www.cbssports.com/nba/schedule/' + urlDate,
Connection: 'keep-alive',
transform: function (body) {
return cheerio.load(body);
}
};
return request(options)
.then(parseSchedule)
.then(writeSchedule)
.catch((err) => console.log("there was an error: " + err))
.finally(res.send("execution finished"));
function parseSchedule($){
const scheduledGames = [];
$('tbody').children('tr').each((i, element) => {
const gameTime = $(element).children('td').eq(2).find('a').text()
const scheduledGame = { gameTime: gameTime};
scheduledGames.push(scheduledGame);
});
return scheduledGames;
}
function writeDB(scheduledGames){
console.log(scheduledGames);
}
}
It typically makes more sense to send a success response at the time in the promise chain when everything is successful, or send an error response in a catch handler. If you do these two things, it doesn't make sense to use finally at all, since success and error are the only two cases you really need to handle. Unless you have some special case, stick to just success and error.
I am running in to an issue with Firebase cloud functions. I have an onWrite cloud function that triggers a sequence of events. I have a path for requests that the onWrite cloud function is tied to. When that cloud function executes, it deletes the new request for the requests path and pushes the request in to a render path/que that will be used client side for rendering UI elements/data. Once the data has been written to the render path, I call a vanilla javascript function that is not tied to any cloud events. The vanilla javascript function is supposed to reach out to an external API and fetch some data to later be updated on the render object that was pushed in to the render path.
The problem is that the vanilla javascript function never executes. I have been looking all over the web to figure out why this happening but can't seem to figure out why. I am on the Flame plan so outbound api requests should be allowed to my knowledge. Here an example of my code:
const functions = require('firebase-functions');
const admin = require('firebase-admin');
const request = require('request');
admin.initializeApp();
exports.requestModule = functions.database.ref('/requests').onWrite((change, context) => {
// Create reference to database
let db = admin.database();
if (context && context.auth && context.auth.uid) {
const afterData = change.after.val();
let uid = context.auth.uid;
let cleanData = afterData[uid];
cleanData.status = "loading";
// Remove the requested module from the requests path
let cleansePath = db.ref('/requests/' + uid);
cleansePath.remove().then((snapshot) => {
return true;
}).catch((error) => {
console.log(error);
return false;
});
// Add requested module to the render path
let renderPath = db.ref('/render/' + uid);
renderPath.push(cleanData).then((snapshot) => {
let val = snapshot.val();
let key = snapshot.key;
// Trigger the get weather api call
getWeather(uid, key, val);
return true;
}).catch((error) => {
console.log(error);
return false;
});
}
});
// Fetches data from external api
function getWeather (uid, key, obj) {
console.log('Fetching weather!');
let db = admin.database();
request('https://api.someweathersite.net/forecast/', (error, response, body) => {
if (!error && Number(response.statusCode) === 200) {
console.log('error:', error);
console.log('statusCode:', response && response.statusCode);
console.log('body:', body);
obj.data = body;
obj.status = 'loaded';
// Set data from api response in render object to be shown client side
let render = db.ref('/render/' + uid + '/' + key );
render.set(obj).then(() => {
return true;
}).catch((error) => {
console.log(error)
return false;
});
}
});
}
The console.log message at the top of the "getWeather" function never executes. I don't think that the "getWeather" function is ever executing.
If I put the api call directly in the onWrite "requestModule" function, the api call will work. However, when it calls an external function it never gets called/works. I basically want to have the "requestModule" function handle all requests and plan to have a module dispatcher that handles which module function/api data should be fetched from. That's why I don't want to keep the api call in the "requestModule" function. Any idea of why this happening or how I can get this working?
getWeather is performing asynchronous work to fetch some data, but it's not returning a promise to indicate when that work is complete. In fact, none of the async work you're performing here is correctly using the promises returned by the various API calls. It's not sufficient to simply use then() on each promise.
You need to keep track of all of the async work, and return a single promise that resolves only after all the work is complete. Otherwise, Cloud Functions may terminate and clean up your function before the work is complete. (Note that it's not deterministic which work may or may not actually complete before forced termination, but the only way to ensure that all work completes is through that single promise you return.)
You may want to watch my tutorials on using promises in Cloud Functions to get a better handle on what you're required to do make your functions work correctly: https://firebase.google.com/docs/functions/video-series/
I'm trying to validate certain URLs based on their status code. I tried to do something like this:
const http = require('http');
let link = 'http://source.unsplash.com/random/100x100';
const getStatusCode = (link, callback) => {
http.get(link, res => {
callback(res.statusCode);
})
}
const codeCallback = code => {
console.log('Status code inside callback: ', code); //works, displays 301
return code
}
let statusCode = getStatusCode(link, codeCallback)
console.log(statusCode) //undefined
if (statusCode) {...} //do something
Is there a way to store status code or should I implement validation logic inside callback?
Welcome to the callback hell ;)
The way you're using it - yes, you need to do your validation logic inside the callback.
If you decide to explore this area further, I'd suggest looking promises and async/await in JavaScript.
Good luck!
Your code is a good example of a typical problem that arises when asynchronous calls happen in javascript. Specifically, the problem is that you are trying to print a variable before its value has been made available.
To explain more, all the http requests are asynchronous in nature due to the fact that they should not be blocking your application while you are waiting for a response from a server.
So when you are calling console.log(statusCode), this line gets executed before the http.get callback has been executed, so at the moment of execution the statusCode is undefined.
Indeed, the solution would be to handle the statusCode inside the callback. Something like this.
const http = require('http');
let link = 'http://source.unsplash.com/random/100x100';
const getStatusCode = (link, callback) => {
http.get(link, res => {
if (res.statusCode){
callback(res.statusCode);
}
})
}
const codeCallback = code => {
console.log('Status code inside callback: ', code); //works, displays 301
return code
}
getStatusCode(link, codeCallback);
If you are interested in understanding how the whole asynchronous logic execution works here is a good intro https://www.youtube.com/watch?v=8aGhZQkoFbQ
So i've been doing some reading and I think I have a general grasp on this subject but could use some insight from someone more experienced. I've been trying to write a simple RSS reader in Meteor and have been facing some issues with calling the Meteor method asynchronously. I currently define the method on the server(synchronously) and call it on the client(asynchronously). What I don't understand is that when I try to make the HTTP.call on the server, I return an undefined value passed to my client if I pass a callback into the request. But when I make the API request synchronously everything seems to work fine. Is this the normal behavior I should expect/the way I should be making the API call?
Meteor.methods({
getSubReddit(subreddit) {
this.unblock();
const url = 'http://www.reddit.com/r/' + subreddit + '/.rss';
const response = HTTP.get(url, {}, (err, res) => {
if(!err) {
//console.log(res.content);
return res;
} else {
return err;
}
});
}
});
Here's the method defined on the server side. Note that logging res.content shows that I'm actually getting the right content back from the call. I've tried reading some other answers on the topic and seen some things about using Future/wrapAsync, but I'm not sure I get it. Any help would be greatly appreciated!
The HTTP.get is doing async work, so callback passed to it will be called out of this meteor method call context.
To get desired result you should do it like this:
Meteor.methods({
getSubReddit(subreddit) {
// IMPORTANT: unblock methods call queue
this.unblock();
const url = 'http://www.reddit.com/r/' + subreddit + '/.rss';
const httpGetSync = Meteor.wrapAsync(HTTP.get);
try {
const response = httpGetSync(url, {});
//console.log(response.content);
return response.content;
} catch (err) {
// pass error to client
throw new Meteor.Error(...);
}
}
});