Platform:
I have an api in sails.js and a frontend in react. The calls between front and back end are being made with fetch api.
More information:
In the course of some api endpoints I have to execute an external file, at this point I am using the execfile() function of node.js, and I have to wait for it to be executed to respond to the frontend.
What is the problem?
If the file is executed in a short time, for example less than 1 minute everything runs well and the behavior occurs as expected on both sides, but if (in this case) the file takes more than 1 minute to execute, there is something to trigger a second call to api (I do not know where this is being called, but I tested the same endpoint with postman and I did not have this problem so I suspect the react / fetch-api) and the api call with the same data from the first call is redone. This causes the file to run twice.
Something that is even stranger is that if you have the DevTools Network inspector turned on this second call does not appear, but nothing in the documentation of sailjs points to this behavior.
Example of an endpoint in sails.js:
/**
* FooController
*/
const execFile = require("child_process").execFile;
module.exports = {
foo: async (req, res) => {
let result = await module.exports._executeScript(req.body).catch(() => {
res.status(500).json({ error: "something has occurred" });
});
res.json(result);
},
_executeScript: body => {
return new Promise((resolve, reject) => {
let args = [process.cwd() + "/scripts/" + "myExternalFile.js", body];
let elem = await module.exports
._execScript(args)
.catch(err => reject(err));
resolve(elem);
});
},
_execScript: args => {
return new Promise((resolve, reject) => {
try {
execFile("node", args, { timeout: 150000 }, (error, stdout, stderr) => {
if (error || (stderr != null && stderr !== "")) {
console.error(error);
} else {
console.log(stdout);
}
let output = { stdout: stdout, stderr: stderr };
resolve(output);
});
} catch (err) {
reject(err);
}
});
}
};
Example of component react with fetch call:
import React from "react";
import { notification } from "antd";
import "./../App.css";
import Oauth from "./../helper/Oauth";
import Config from "./../config.json";
class App extends React.Component {
constructor(props) {
super(props);
this.state = {
syncInAction: false,
data: null
};
}
componentDidMount() {
this.handleSync();
}
async handleSync() {
let response = await fetch(Config.apiLink + "/foo/foo", {
method: "POST",
mode: "cors",
headers: {
Authorization: Oauth.isLoggedIn(),
Accept: "application/json",
"Content-Type": "application/json"
},
body: JSON.stringify(this.state.myData)
}).catch(err => {
notification.error({
key: "catch-ApiFail",
message: "Erro"
});
});
let json = await response.json();
this.setState({
syncInAction: false,
data: json
});
}
render() {
return <div>{this.state.data}</div>;
}
}
export default App;
What is my expected goal / behavior:
It does not matter if the call takes 1 minute or 10 hours, the file can only be called once and when it finishes, then yes, it can return to the frontend.
Note that the examples do not have the original code and have not been tested. Is a simplified version of the code to explain the behavior
I ended up solving the problem, apparently nodejs has a default timing of 2 minutes on the server, and can be rewritten to miss this timout or increase it.
This was just adding a line of code at the beginning of the foo() endpoint and the problem was solved.
The behavior of redoing the call is that it is not documented, and it is strange not to have this behavior when using the postman, but here is the solution for whoever needs it.
Final result:
foo: async (req, res) => {
req.setTimeout(0);
let result = await module.exports._executeScript(req.body).catch(() => {
res.status(500).json({ error: "something has occurred" });
});
res.json(result);
};
Related
Im in the process of moving an app from Vue 2 -> 3
I decided to take a moment to really upgrade and refactor all my repo and that led to using Vue3 recs on new tech, one being vite
My problem is I don't totally understand how the backend API process works so im struggling to move my api route from vue-cli to vite.
I would like to keep using the logic that call functions from api/users to remain in place but im open to a better option
Ultimately I get 404 - Not Found as my response which means its cant find the route
Heres my api/user.js
import request from '../utils/request'
export function login(data) {
return request({
url: '/user/login',
method: 'post',
data
})
}
export function getInfo(token) {
return request({
url: '/user/info',
method: 'get',
params: { token }
})
}
export function logout() {
return request({
url: '/user/logout',
method: 'post'
})
}
utils/Request.js
import axios from 'axios'
import { ElMessageBox, ElMessage } from 'element-plus'
import { userStore } from '../stores/user'
import { getToken } from '../utils/auth'
// create an axios instance
const service = axios.create({
baseURL: import.meta.env.VUE_APP_BASE_API, // url = base url + request url
// withCredentials: true, // send cookies when cross-domain requests
timeout: 5000 // request timeout
})
// request interceptor
service.interceptors.request.use(
config => {
// do something before request is sent
const useStore = userStore;
console.log("Req", "Req Init");
if (useStore.token) {
// let each request carry token
// ['X-Token'] is a custom headers key
// please modify it according to the actual situation
config.headers['X-Token'] = getToken()
}
return config
},
error => {
// do something with request error
console.log(error) // for debug
console.log("failed") // for debug
return Promise.reject(error)
}
)
// response interceptor
service.interceptors.response.use(
/**
* If you want to get http information such as headers or status
* Please return response => response
*/
/**
* Determine the request status by custom code
* Here is just an example
* You can also judge the status by HTTP Status Code
*/
response => {
const res = response.data
console.log("Res", "Res Init");
// if the custom code is not 20000, it is judged as an error.
if (res.code !== 20000) {
ElMessage({
message: res.message || 'Error',
type: 'error',
duration: 5 * 1000
})
// 50008: Illegal token; 50012: Other clients logged in; 50014: Token expired;
if (res.code === 50008 || res.code === 50012 || res.code === 50014) {
// to re-login
ElMessageBox.confirm('You have been logged out, you can cancel to stay on this page, or log in again', 'Confirm logout', {
confirmButtonText: 'Re-Login',
cancelButtonText: 'Cancel',
type: 'warning'
}).then(() => {
store.dispatch('user/resetToken').then(() => {
location.reload()
})
})
}
return Promise.reject(new Error(res.message || 'Error'))
} else {
return res
}
},
error => {
console.log('err' + error) // for debug
ElMessage({
message: error.message,
type: 'error',
duration: 5 * 1000
})
return Promise.reject(error)
}
)
export default service
And a peek at my store that actually calls the endpoint
import { login, logout, getInfo } from '../api/user'
actions: { // user login
login({ commit }, userInfo) {
const { username, password } = userInfo
// **Call is made here to 'login'**
return new Promise((resolve, reject) => {
login({ username: username.trim(), password: password }).then(response => {
const { data } = response
commit('SET_TOKEN', data.token)
setToken(data.token)
resolve()
}).catch(error => {
reject(error)
})
})
},
Lastly where the call originates from: this.store.login in my login.vue component
this.store.login('user/login', this.loginForm).then(() => {
this.$router.push({ path: this.redirect || '/' })
this.loading = false
}).catch(() => {
this.loading = false
})
This is my first StackOverflow post, so be kind if I need to include something else and thanks to any ideas or tips. Thank you
I've tried to search on different ports but the route still comes back as undefined.
I tried to change the vite config to include the server option but it still doesn't seem to help.
Not sure If I need to modify the config or not but I did have some settings related to the server mock on my old webpack config
I'm using firebase functions on a server for API calls. Everything works fine 70% of the time, but all of a sudden some of my function calls start failing to execute, giving my API a 404, and don't work for the next few hours.
In my StackDriver I can see the function isn't called again when I try. My API just gives me a 404 without ever reaching the server.
Below is one of the calls that fails once in a while. Going to the URL i'm fetching, the GET result always shows up, so I have no clue what the issue is.
API call:
const getCreators = () => {
return window
.fetch(url + '/get-creators', {
method: 'GET',
headers: {
'Content-Type': 'application/json',
},
})
.then((res) => {
console.log(res);
if (res.status === 200) {
return res.json();
} else {
return null;
}
})
.then((data) => {
if (!data || data.error) {
return null;
} else {
return data;
}
});
};
Server code:
const app = express();
app.get('/get-creators', async (req, res) => {
console.log('creators: ');
creators
.find()
.toArray()
.then((result) => {
console.log(result);
res.status(200).send(result);
})
.catch(() => {
console.log('error');
res.send('error');
});
});
app.listen(4242, () => console.log(`Node server listening at https ${4242}!`));
exports.app = functions.https.onRequest(app);
Found it. You don't want the below code on your server:
app.listen(4242, () => console.log(`Node server listening at https ${4242}!`));
I commented this code out, republished, and all is well.
I thought having this didn't make a difference, but apparently once in a blue moon it can and will try to make the server listen locally, which gave me a 404.
I am making a Discord bot, and I want it to be able to use the YouTube API to fetch new uploads from a specific channel.
I have searched elsewhere, but they all say how to upload videos, not how to track uploads.
Is this possible, and how can I do it?
Edit: Tried PubSubHubbub but it was very confusing and I couldn't get it to work
Here an example built on top of Node.js (v12) and Fastify and published with ngrok:
I wrote some comments explaining what it is happening:
const fastify = require('fastify')({ logger: true })
const xmlParser = require('fast-xml-parser')
const { URLSearchParams } = require('url')
const fetch = require('node-fetch')
// add an xml parser
fastify.addContentTypeParser('application/atom+xml', { parseAs: 'string' }, function (req, xmlString, done) {
try {
const body = xmlParser.parse(xmlString, {
attributeNamePrefix: '',
ignoreAttributes: false
})
done(null, body)
} catch (error) {
done(error)
}
})
// this endpoint needs for authentication
fastify.get('/', (request, reply) => {
reply.send(request.query['hub.challenge'])
})
// this endpoint will get the updates
fastify.post('/', (request, reply) => {
console.log(JSON.stringify(request.body, null, 2))
reply.code(204)
reply.send('ok')
})
fastify.listen(8080)
.then(() => {
// after the server has started, subscribe to the hub
// Parameter list: https://pubsubhubbub.github.io/PubSubHubbub/pubsubhubbub-core-0.4.html#rfc.section.5.1
const params = new URLSearchParams()
params.append('hub.callback', 'https://1f3dd0c63e78.ngrok.io') // you must have a public endpoint. get it with "ngrok http 8080"
params.append('hub.mode', 'subscribe')
params.append('hub.topic', 'https://www.youtube.com/xml/feeds/videos.xml?channel_id=UCfWbGF64qBSVM2Wq9fwrfrg')
params.append('hub.lease_seconds', '')
params.append('hub.secret', '')
params.append('hub.verify', 'sync')
params.append('hub.verify_token', '')
return fetch('https://pubsubhubbub.appspot.com/subscribe', {
headers: { 'content-type': 'application/x-www-form-urlencoded' },
body: params,
method: 'POST'
})
})
.then((res) => {
console.log(`The status must be 204. Received ${res.status}`)
// shows the error if something went wrong
if (res.status !== 204) {
return res.text().then(txt => console.log(txt))
}
})
I used my channel id to do some testing, consider that the notification is not in real-time, the POSTs are triggered after several minutes usually.
I'm raising the white flag and asking for suggestions even though I feel the answer is probably right in front of my face.
I have a login form that I am submitting to an api (AWS) and acting on the result. The issue I am having is once the handleSubmit method is called, I am immediately getting into the console.log statement... which to no surprise returns dispatch result: undefined
I realize this is likely not a direct function of vue.js, but how I have the javascript set executing.
Here is my login component:
// SignInForm.vue
handleSubmit() {
try {
const {username, password} = this.form;
this.$store.dispatch('user/authenticate', this.form).then(res => {
console.log('dispatch result: ', res);
});
} catch (error) {
console.log("Error: SignInForm.handleSubmit", error);
}
},
...
Here is what my store is doing. I'm sending it to a UserService I've created. Everything is working great. I am getting the correct response(s) and can log everything out I need. The UserService is making an axios request (AWS Amplify) and returning the response.
// user.js (vuex store)
authenticate({state, commit, dispatch}, credentials) {
dispatch('toggleLoadingStatus', true);
UserService.authenticate(credentials)
.then(response => {
dispatch('toggleLoadingStatus', false);
if (response.code) {
dispatch("setAuthErrors", response.message);
dispatch('toggleAuthenticated', false);
dispatch('setUser', undefined);
// send error message back to login component
} else {
dispatch('toggleAuthenticated', true);
dispatch('setUser', response);
AmplifyEventBus.$emit("authState", "authenticated");
// Need to move this back to the component somehow
// this.$router.push({
// name: 'dashboard',
// });
}
return response;
});
},
...
Where I'm getting stuck at is, if I have error(s) I can set the errors in the state, but I'm not sure how to access them in the other component. I've tried setting the data property to a computed method that looks at the store, but I get errors.
I'm also struggling to use vue-router if I'm successfully authenticated. From what I've read I really don't want to be doing that in the state anyway -- so that means I need to return the success response back to the SignInForm component so I can use vue-router to redirect the user to the dashboard.
Yep. Just took me ~6 hours, posting to SO and then re-evaluating everything (again) to figure it out. It was in fact, somewhat of a silly mistake. But to help anyone else here's what I was doing wrong...
// SignInForm.vue
async handleSubmit() {
try {
await this.$store.dispatch("user/authenticate", this.form)
.then(response => {
console.log('SignInForm.handleSubmit response: ', response); // works
if (response.code) {
this.errors.auth.username = this.$store.getters['user/errors'];
} else {
this.$router.push({
name: 'dashboard',
});
}
}).catch(error => {
console.log('big problems: ', error);
});
} catch (error) {
console.log("Error: SignInForm.handleSubmit", error);
}
},
...
Here's my first mistake: I was calling from an async method to another method - but not telling that method to be async so the call(er) method response was executing right away. Here's the updated vuex store:
// user.js (vuex store)
async authenticate({state, commit, dispatch}, credentials) { // now async
dispatch('toggleLoadingStatus', true);
return await UserService.authenticate(credentials)
.then(response => {
console.log('UserService.authenticate response: ', response); // CognitoUser or code
dispatch('toggleLoadingStatus', false);
if (response.code) {
dispatch("setAuthErrors", response.message);
dispatch('toggleAuthenticated', false);
dispatch('setUser', undefined);
} else {
dispatch('toggleAuthenticated', true);
dispatch('setUser', response);
AmplifyEventBus.$emit("authState", "authenticated");
}
return response;
});
},
...
My second error was that I wasn't returning the result of the method at all from the vuex store.
Old way:
UserService.authenticate(credentials)
Better way:
return await UserService.authenticate(credentials)
Hope this saves someone a few hours. ¯_(ツ)_/¯
This works for Vue3:
export default {
name: 'Login',
methods: {
loginUser: function () {
authenticationStore.dispatch("loginUser", {
email: 'peter#example.com',
})
.then(response => {
if (response.status === 200) {
console.log('Do something')
}
});
},
},
}
In the store you can simply pass back the http response which is a promise.
const authenticationStore = createStore({
actions: {
loginUser({commit}, {email}) {
const data = {
email: email
};
return axios.post(`/authentication/login/`, data)
.then(response => {
toastr.success('Success')
return response
})
},
}
})
I am not sure why but my webhook is being fired twice in my cron job. So this cron job is suppose to run once every 15 min which it does, but it is firing off twice. I will post the logs, handler and yml file to help out.
Basically my cron job will make some request to a salsify api to store a url inside a mongodb. Once that file has been completed and built the next time the cron job runs it should trigger the webhook for netlify. Then the process starts all over again.
In my netlify I noticed the build was being ran twice and have pin pointed the source to the serverless cron job.
EDIT: Something I should add in here is that even if my cron job runs twice it still should still only technically call the webhook once if there is a file in the MongoDB. Yet it is still calling it twice somehow which is causing my netlify build to fail because it needs that file in order to build.
function part of serverless.yml:
functions:
salsifyCron:
handler: src/handler.salsifyCron
events:
- schedule:
rate: cron(*/15 * * * ? *)
enabled: true
Logs:
2018-05-17 10:00:41.121 (-05:00) 10d87735-59e3-11e8-be56-69e06899fa1f Trigger Webhook
2018-05-17 10:01:45.941 (-05:00) 10d87735-59e3-11e8-be56-69e06899fa1f Trigger Webhook
handler:
require('envdotjs').load();
import fetch from 'isomorphic-fetch';
import axios from 'axios';
import middy from 'middy';
import { jsonBodyParser, httpErrorHandler, cors } from 'middy/middlewares';
import { connectToDatabase } from '../utils/db';
import Sheet from '../models/Sheet';
import config from '../utils/config';
module.exports.salsifyCron = middy(async (event, context, callback) => {
context.callbackWaitsForEmptyEventLoop = false;
let sheetId;
const options = {
url: `https://app.salsify.com/api/orgs/${
process.env.SALSIFY_ORG_ID
}/export_runs`,
headers: {
Authorization: `Bearer ${process.env.SALSIFY_API_KEY}`,
'Content-Type': 'application/json'
}
};
await connectToDatabase();
const storedData = await Sheet.find({});
if (
storedData.length > 0 &&
storedData[0] &&
storedData[0].status === 'completed' &&
storedData[0].url !== null
) {
console.log('Trigger WebHook');
axios.post('https://api.netlify.com/build_hooks/*****************');
process.exit(0);
return;
}
if (storedData[0]) {
sheetId = storedData[0].sheetId;
}
if (storedData.length === 0) {
const res = await fetch(options.url, {
method: 'POST',
headers: options.headers,
body: JSON.stringify(config)
}).then(res => res.json());
if (res.id && res.status) {
await Sheet.create({
sheetId: res.id,
url: null,
status: res.status
});
sheetId = res.id;
} else {
console.log(res);
process.exit(1);
}
}
const resWithId = await fetch(`${options.url}/${sheetId}`, {
method: 'GET',
headers: options.headers
}).then(res => res.json());
if (resWithId.status === 'running') {
console.log('running cron job');
console.log(resWithId.estimated_time_remaining);
}
if (resWithId.status === 'completed') {
console.log('completed cron job');
await Sheet.findByIdAndUpdate(
storedData[0],
{ status: resWithId.status, url: resWithId.url },
{ new: true }
);
}
})
.use(cors())
.use(jsonBodyParser())
.use(httpErrorHandler());
Lambda timeout. This might not have been the problem in your case, but it is a common problem that causes this result.
Your lambdas are not getting executed simultaneously but with a bit of a delay. This is a clue that it is not just getting a duplicate execution.
I would guess that your lambda is first terminating with an error (for example timing out - the default lambda timeout is quite small) and the lambda is being rerun after it fails.
I have had this problem with timeouts and it is quite confusing if you don't notice that the lambda has timed out.