I'm using firebase functions on a server for API calls. Everything works fine 70% of the time, but all of a sudden some of my function calls start failing to execute, giving my API a 404, and don't work for the next few hours.
In my StackDriver I can see the function isn't called again when I try. My API just gives me a 404 without ever reaching the server.
Below is one of the calls that fails once in a while. Going to the URL i'm fetching, the GET result always shows up, so I have no clue what the issue is.
API call:
const getCreators = () => {
return window
.fetch(url + '/get-creators', {
method: 'GET',
headers: {
'Content-Type': 'application/json',
},
})
.then((res) => {
console.log(res);
if (res.status === 200) {
return res.json();
} else {
return null;
}
})
.then((data) => {
if (!data || data.error) {
return null;
} else {
return data;
}
});
};
Server code:
const app = express();
app.get('/get-creators', async (req, res) => {
console.log('creators: ');
creators
.find()
.toArray()
.then((result) => {
console.log(result);
res.status(200).send(result);
})
.catch(() => {
console.log('error');
res.send('error');
});
});
app.listen(4242, () => console.log(`Node server listening at https ${4242}!`));
exports.app = functions.https.onRequest(app);
Found it. You don't want the below code on your server:
app.listen(4242, () => console.log(`Node server listening at https ${4242}!`));
I commented this code out, republished, and all is well.
I thought having this didn't make a difference, but apparently once in a blue moon it can and will try to make the server listen locally, which gave me a 404.
Related
I've been spending nearly all day just trying to implement a rather simple feature in my React code. The basic idea is checking if a server is reachable, and if it isn't, return a console.log() indicating so. Here's what I have so far:
Relevant Code
const handleLinkRegex = () => {
fetch(LinkInput, { mode: "no-cors" })
.then((response) => {
if (response.ok || response.status === 0) {
console.log("yessir");
} else if (response.status === 404) {
return Promise.reject("error 404");
} else {
return Promise.reject("some other error: " + response.status);
}
})
.then((data) => console.log("data is", data))
.catch((error) => console.log("error is", error));
};
Output
If the link is valid, such as https://mantine.dev/core/input/, the result is yessir, followed with data is undefined.
If the link is invalid and returns a 404, such as https://mantine.dev/core/input/invalidurl, the result is a console 404 error, and yessir, followed with data is undefined, which is the same as if it didn't fail.
What I tried
Using the url-exist library only resulted in a CORS error
Attempted to use a different solution from a stackoverflow question:
const handleLinkVerify = async () => {
fetch(LinkInput, { mode: "no-cors" })
.then((r) => {
console.log("Is reachable");
})
.catch((e) => {
console.log("Is not there");
});
};
Which resulted in every url, no matter if valid or not, to return as Is not there.
Overall, I'm waving the white flag with dealing with a simple issue. It's taking me hours just to catch this 404 error and handle it, and no matter what green checkmark answer I read their solution doesn't work for me, for some reason. I feel like I'm missing something obvious, but I don't know what. Thanks for any help.
Since it is not possible to distinguish a CORS-Error from any other Error, let's say Network-Error, and you can't even read the Status-Code, so you can't tell if the website sent a 404 or any other code, the approach you want to go (checking it on the front-end) is technically impossible. CORS was specifically designed to behave that way. If you want to read more on that: Trying to use fetch and pass in mode: no-cors
Your best bet here would be to do this sort of thing on the backend, since you can just ignore the cors header and just read the data. You could do something like that:
I used express and axios, but you can use whatever you want to.
const express = require("express");
const app = express();
const axios = require("axios");
app.use(express.json());
app.post("/checkStatusCode", async (req, res) => {
const { url } = req.body;
if (url == undefined || typeof url != "string") {
return res.status(400).json({ status: 400, msg: "URL required" });
}
try {
const request = await axios.get(url);
res.status(request.status).json({ url, status: request.status, msg: request.statusText });
} catch (err) {
if (err.response != undefined) {
res.status(err.response.status).json({ url, status: err.response.status, msg: err.response.statusText });
} else {
console.log(err);
res.status(500).json({ status: 500, msg: "Internal Server Error" });
}
}
});
app.listen(5000);
Then you would just call that on your frontend, and check for the Statuscodes:
var myHeaders = new Headers();
myHeaders.append("Content-Type", "application/json");
var raw = JSON.stringify({
"url": "https://google.com"
});
var requestOptions = {
method: 'POST',
headers: myHeaders,
body: raw,
redirect: 'follow'
};
fetch("http://localhost:5000/checkStatusCode", requestOptions)
.then(response => response.json())
.then(result => console.log(result))
.catch(error => console.log('error', error));
If you have trouble with CORS on your backend, there is a npm package for that: https://www.npmjs.com/package/cors
Just require and use it like this:
const cors = require("cors");
app.use(cors());
I am making a Discord bot, and I want it to be able to use the YouTube API to fetch new uploads from a specific channel.
I have searched elsewhere, but they all say how to upload videos, not how to track uploads.
Is this possible, and how can I do it?
Edit: Tried PubSubHubbub but it was very confusing and I couldn't get it to work
Here an example built on top of Node.js (v12) and Fastify and published with ngrok:
I wrote some comments explaining what it is happening:
const fastify = require('fastify')({ logger: true })
const xmlParser = require('fast-xml-parser')
const { URLSearchParams } = require('url')
const fetch = require('node-fetch')
// add an xml parser
fastify.addContentTypeParser('application/atom+xml', { parseAs: 'string' }, function (req, xmlString, done) {
try {
const body = xmlParser.parse(xmlString, {
attributeNamePrefix: '',
ignoreAttributes: false
})
done(null, body)
} catch (error) {
done(error)
}
})
// this endpoint needs for authentication
fastify.get('/', (request, reply) => {
reply.send(request.query['hub.challenge'])
})
// this endpoint will get the updates
fastify.post('/', (request, reply) => {
console.log(JSON.stringify(request.body, null, 2))
reply.code(204)
reply.send('ok')
})
fastify.listen(8080)
.then(() => {
// after the server has started, subscribe to the hub
// Parameter list: https://pubsubhubbub.github.io/PubSubHubbub/pubsubhubbub-core-0.4.html#rfc.section.5.1
const params = new URLSearchParams()
params.append('hub.callback', 'https://1f3dd0c63e78.ngrok.io') // you must have a public endpoint. get it with "ngrok http 8080"
params.append('hub.mode', 'subscribe')
params.append('hub.topic', 'https://www.youtube.com/xml/feeds/videos.xml?channel_id=UCfWbGF64qBSVM2Wq9fwrfrg')
params.append('hub.lease_seconds', '')
params.append('hub.secret', '')
params.append('hub.verify', 'sync')
params.append('hub.verify_token', '')
return fetch('https://pubsubhubbub.appspot.com/subscribe', {
headers: { 'content-type': 'application/x-www-form-urlencoded' },
body: params,
method: 'POST'
})
})
.then((res) => {
console.log(`The status must be 204. Received ${res.status}`)
// shows the error if something went wrong
if (res.status !== 204) {
return res.text().then(txt => console.log(txt))
}
})
I used my channel id to do some testing, consider that the notification is not in real-time, the POSTs are triggered after several minutes usually.
Firstly, please note that I am very very new to JS and coding as a general :)
Desired behaviour:
I have written the following JS HTTPS Firebase function which, which takes in a query parameter locationId, it performs a GET API call and saves the response back to Firebase. The code correctly saves the data to Firebase as desired. I have come across similar issues but i'm struggling to adapt those solutions to my specific issue below. From what I see, I'm only sending the response once.
Specific error: The following is the console output
Cannot set headers after they are sent to the client
Unhandled rejection
My function:
exports.doshiiGetMenuForOnboardedVenue = functions.https.onRequest((req, res) => {
// Forbidding PUT requests.
if (req.method === 'PUT') {
return res.status(403).send('Forbidden!');
}
cors(req, res, () => {
const locationId = req.query.locationId;
console.log('locationId', locationId);
if (locationId){
console.log('locationId', locationId);
var token = jwttoken();
const options = {
headers: {
'content-type': 'application/json',
'authorization': 'Bearer ' + token}
};
const uri = 'https://sandbox.doshii.co/partner/v3/locations/' + locationId + '/menu?lastVersion=:lastVersion&filtered=true'
axios.get(uri, options)
.then(response => {
return admin.database().ref(`/venue-menus/${locationId}/`).set(response.data)
.then(response => {
return res.status(200).send(locationId)
})
.catch(err => {
return res.status(500).send({error: err})
})
})
.then(response => {
return res.status(200).send(locationId)
})
.catch(err => {
return res.status(500).send({error: err})
})//end axios
} else {
return res.status(500).send({error: 'locationId missing'})
}//end if-else (!locationId)
})//end cors
});
By flattening out your nested promises, you can see that your code is performing the following instructions (when the axios call doesn't throw an error):
admin.database().ref(`/venue-menus/${locationId}/`).set(response.data))
.then(response => res.status(200).send(locationId))
.catch(err => res.status(500).send({error: err})
.then(response => res.status(200).send(locationId)) // this line is always called after either of the above.
.catch(err => res.status(500).send({error: err})
As general practice, unless required, you should not nest promises with their own then() and catch() handlers as it will lead to bizarre effects like this.
Furthermore, if your code calls for using //end axios or //end cors messages, you should flatten out your code so it makes sense without those messages.
Adapting your code to "fail fast", correcting your API responses and appropriately hiding error stack traces gives:
const cors = require('cors')({
origin: true,
methods: ["GET"]
});
exports.doshiiGetMenuForOnboardedVenue = functions.https.onRequest((req, res) => {
cors(req, res, (err) => { // note: cors will handle OPTIONS method
if (err) {
// note: log full error at ERROR message level
console.error('Internal CORS error:', err);
// note: return only generic status message to client
return res.status(500).json({error: 'Internal Server Error'});
}
// Forbidding anything but GET requests.
if (req.method !== 'GET') {
// 405 METHOD_NOT_ALLOWED
return res.status(405)
.set('Allow', 'GET')
.json({error: 'Not Allowed!'});
}
const locationId = req.query.locationId;
console.log('locationId', locationId);
if (!locationId) {
// 400 BAD_REQUEST
return res.status(400).json({error: 'locationId missing'})
}
var token = jwttoken();
const options = {
headers: {
'content-type': 'application/json',
'authorization': 'Bearer ' + token
}
};
// note: Don't forget to enable billing for third-party APIs!
const uri = 'https://sandbox.doshii.co/partner/v3/locations/' + locationId + '/menu?lastVersion=:lastVersion&filtered=true'
axios.get(uri, options)
.then(response => admin.database().ref(`/venue-menus/${locationId}/`).set(response.data))
.then(() => {
// note: as locationId was already sent by the client, send new/useful
// information back or nothing but the right status code
res.status(200).json({ ref: `/venue-menus/${locationId}/` });
})
.catch(err => {
// note: log full error at ERROR message level
console.error('Failed to retrieve/save API data:', err);
// note: return only message to client
res.status(500).json({error: err.message || 'Internal Server Error'});
});
});
});
Using interceptBufferProtocol, I can successfully intercept the loadURL event to https://google.com mainWindow.loadURL("https://google.com/"); and replace it with my custom HTML code. The HTML code has an iframe which I am trying to proxy. This can usually be achieved by setting the electron browserWindow proxy but in my case, it fails to work. I set the proxy with the following code:
mainWindow.webContents.session.setProxy({
proxyRules: "http://" + proxy
}, () => {
console.log('Proxy: http://' + proxy)
})
Intercept url code:
ses.protocol.interceptBufferProtocol('https', (req, callback) => {
ses.resolveProxy(req.url, (x) => {
console.log(x)
})
if (req.url == "https://google.com/") {
fs.readFile(path.join(__dirname, "/../../path/stuff.html"), 'utf8', function(err, html) {
callback(Buffer.from(html, 'utf8'));
});
} else {
const request = net.request(req)
request.on('response', res => {
const chunks = []
res.on('data', chunk => {
chunks.push(Buffer.from(chunk))
})
res.on('end', async () => {
const file = Buffer.concat(chunks)
callback(file)
})
})
if (req.uploadData) {
req.uploadData.forEach(part => {
if (part.bytes) {
request.write(part.bytes)
} else if (part.file) {
request.write(fs.readFileSync(part.file))
}
})
}
request.end()
}
})
However, no matter what I do, it appears to use my local IP instead of a proxy. Do I have any options?
The code runs fine without a proxy. I'm trying to run it with one. The problem lies within the .interceptBufferProtocol() function. Any help would be appreciated!
Platform:
I have an api in sails.js and a frontend in react. The calls between front and back end are being made with fetch api.
More information:
In the course of some api endpoints I have to execute an external file, at this point I am using the execfile() function of node.js, and I have to wait for it to be executed to respond to the frontend.
What is the problem?
If the file is executed in a short time, for example less than 1 minute everything runs well and the behavior occurs as expected on both sides, but if (in this case) the file takes more than 1 minute to execute, there is something to trigger a second call to api (I do not know where this is being called, but I tested the same endpoint with postman and I did not have this problem so I suspect the react / fetch-api) and the api call with the same data from the first call is redone. This causes the file to run twice.
Something that is even stranger is that if you have the DevTools Network inspector turned on this second call does not appear, but nothing in the documentation of sailjs points to this behavior.
Example of an endpoint in sails.js:
/**
* FooController
*/
const execFile = require("child_process").execFile;
module.exports = {
foo: async (req, res) => {
let result = await module.exports._executeScript(req.body).catch(() => {
res.status(500).json({ error: "something has occurred" });
});
res.json(result);
},
_executeScript: body => {
return new Promise((resolve, reject) => {
let args = [process.cwd() + "/scripts/" + "myExternalFile.js", body];
let elem = await module.exports
._execScript(args)
.catch(err => reject(err));
resolve(elem);
});
},
_execScript: args => {
return new Promise((resolve, reject) => {
try {
execFile("node", args, { timeout: 150000 }, (error, stdout, stderr) => {
if (error || (stderr != null && stderr !== "")) {
console.error(error);
} else {
console.log(stdout);
}
let output = { stdout: stdout, stderr: stderr };
resolve(output);
});
} catch (err) {
reject(err);
}
});
}
};
Example of component react with fetch call:
import React from "react";
import { notification } from "antd";
import "./../App.css";
import Oauth from "./../helper/Oauth";
import Config from "./../config.json";
class App extends React.Component {
constructor(props) {
super(props);
this.state = {
syncInAction: false,
data: null
};
}
componentDidMount() {
this.handleSync();
}
async handleSync() {
let response = await fetch(Config.apiLink + "/foo/foo", {
method: "POST",
mode: "cors",
headers: {
Authorization: Oauth.isLoggedIn(),
Accept: "application/json",
"Content-Type": "application/json"
},
body: JSON.stringify(this.state.myData)
}).catch(err => {
notification.error({
key: "catch-ApiFail",
message: "Erro"
});
});
let json = await response.json();
this.setState({
syncInAction: false,
data: json
});
}
render() {
return <div>{this.state.data}</div>;
}
}
export default App;
What is my expected goal / behavior:
It does not matter if the call takes 1 minute or 10 hours, the file can only be called once and when it finishes, then yes, it can return to the frontend.
Note that the examples do not have the original code and have not been tested. Is a simplified version of the code to explain the behavior
I ended up solving the problem, apparently nodejs has a default timing of 2 minutes on the server, and can be rewritten to miss this timout or increase it.
This was just adding a line of code at the beginning of the foo() endpoint and the problem was solved.
The behavior of redoing the call is that it is not documented, and it is strange not to have this behavior when using the postman, but here is the solution for whoever needs it.
Final result:
foo: async (req, res) => {
req.setTimeout(0);
let result = await module.exports._executeScript(req.body).catch(() => {
res.status(500).json({ error: "something has occurred" });
});
res.json(result);
};