Serverless Cron Job firing twice - javascript

I am not sure why but my webhook is being fired twice in my cron job. So this cron job is suppose to run once every 15 min which it does, but it is firing off twice. I will post the logs, handler and yml file to help out.
Basically my cron job will make some request to a salsify api to store a url inside a mongodb. Once that file has been completed and built the next time the cron job runs it should trigger the webhook for netlify. Then the process starts all over again.
In my netlify I noticed the build was being ran twice and have pin pointed the source to the serverless cron job.
EDIT: Something I should add in here is that even if my cron job runs twice it still should still only technically call the webhook once if there is a file in the MongoDB. Yet it is still calling it twice somehow which is causing my netlify build to fail because it needs that file in order to build.
function part of serverless.yml:
functions:
salsifyCron:
handler: src/handler.salsifyCron
events:
- schedule:
rate: cron(*/15 * * * ? *)
enabled: true
Logs:
2018-05-17 10:00:41.121 (-05:00) 10d87735-59e3-11e8-be56-69e06899fa1f Trigger Webhook
2018-05-17 10:01:45.941 (-05:00) 10d87735-59e3-11e8-be56-69e06899fa1f Trigger Webhook
handler:
require('envdotjs').load();
import fetch from 'isomorphic-fetch';
import axios from 'axios';
import middy from 'middy';
import { jsonBodyParser, httpErrorHandler, cors } from 'middy/middlewares';
import { connectToDatabase } from '../utils/db';
import Sheet from '../models/Sheet';
import config from '../utils/config';
module.exports.salsifyCron = middy(async (event, context, callback) => {
context.callbackWaitsForEmptyEventLoop = false;
let sheetId;
const options = {
url: `https://app.salsify.com/api/orgs/${
process.env.SALSIFY_ORG_ID
}/export_runs`,
headers: {
Authorization: `Bearer ${process.env.SALSIFY_API_KEY}`,
'Content-Type': 'application/json'
}
};
await connectToDatabase();
const storedData = await Sheet.find({});
if (
storedData.length > 0 &&
storedData[0] &&
storedData[0].status === 'completed' &&
storedData[0].url !== null
) {
console.log('Trigger WebHook');
axios.post('https://api.netlify.com/build_hooks/*****************');
process.exit(0);
return;
}
if (storedData[0]) {
sheetId = storedData[0].sheetId;
}
if (storedData.length === 0) {
const res = await fetch(options.url, {
method: 'POST',
headers: options.headers,
body: JSON.stringify(config)
}).then(res => res.json());
if (res.id && res.status) {
await Sheet.create({
sheetId: res.id,
url: null,
status: res.status
});
sheetId = res.id;
} else {
console.log(res);
process.exit(1);
}
}
const resWithId = await fetch(`${options.url}/${sheetId}`, {
method: 'GET',
headers: options.headers
}).then(res => res.json());
if (resWithId.status === 'running') {
console.log('running cron job');
console.log(resWithId.estimated_time_remaining);
}
if (resWithId.status === 'completed') {
console.log('completed cron job');
await Sheet.findByIdAndUpdate(
storedData[0],
{ status: resWithId.status, url: resWithId.url },
{ new: true }
);
}
})
.use(cors())
.use(jsonBodyParser())
.use(httpErrorHandler());

Lambda timeout. This might not have been the problem in your case, but it is a common problem that causes this result.
Your lambdas are not getting executed simultaneously but with a bit of a delay. This is a clue that it is not just getting a duplicate execution.
I would guess that your lambda is first terminating with an error (for example timing out - the default lambda timeout is quite small) and the lambda is being rerun after it fails.
I have had this problem with timeouts and it is quite confusing if you don't notice that the lambda has timed out.

Related

Dev Server with vite + vue3, route 404 not found

Im in the process of moving an app from Vue 2 -> 3
I decided to take a moment to really upgrade and refactor all my repo and that led to using Vue3 recs on new tech, one being vite
My problem is I don't totally understand how the backend API process works so im struggling to move my api route from vue-cli to vite.
I would like to keep using the logic that call functions from api/users to remain in place but im open to a better option
Ultimately I get 404 - Not Found as my response which means its cant find the route
Heres my api/user.js
import request from '../utils/request'
export function login(data) {
return request({
url: '/user/login',
method: 'post',
data
})
}
export function getInfo(token) {
return request({
url: '/user/info',
method: 'get',
params: { token }
})
}
export function logout() {
return request({
url: '/user/logout',
method: 'post'
})
}
utils/Request.js
import axios from 'axios'
import { ElMessageBox, ElMessage } from 'element-plus'
import { userStore } from '../stores/user'
import { getToken } from '../utils/auth'
// create an axios instance
const service = axios.create({
baseURL: import.meta.env.VUE_APP_BASE_API, // url = base url + request url
// withCredentials: true, // send cookies when cross-domain requests
timeout: 5000 // request timeout
})
// request interceptor
service.interceptors.request.use(
config => {
// do something before request is sent
const useStore = userStore;
console.log("Req", "Req Init");
if (useStore.token) {
// let each request carry token
// ['X-Token'] is a custom headers key
// please modify it according to the actual situation
config.headers['X-Token'] = getToken()
}
return config
},
error => {
// do something with request error
console.log(error) // for debug
console.log("failed") // for debug
return Promise.reject(error)
}
)
// response interceptor
service.interceptors.response.use(
/**
* If you want to get http information such as headers or status
* Please return response => response
*/
/**
* Determine the request status by custom code
* Here is just an example
* You can also judge the status by HTTP Status Code
*/
response => {
const res = response.data
console.log("Res", "Res Init");
// if the custom code is not 20000, it is judged as an error.
if (res.code !== 20000) {
ElMessage({
message: res.message || 'Error',
type: 'error',
duration: 5 * 1000
})
// 50008: Illegal token; 50012: Other clients logged in; 50014: Token expired;
if (res.code === 50008 || res.code === 50012 || res.code === 50014) {
// to re-login
ElMessageBox.confirm('You have been logged out, you can cancel to stay on this page, or log in again', 'Confirm logout', {
confirmButtonText: 'Re-Login',
cancelButtonText: 'Cancel',
type: 'warning'
}).then(() => {
store.dispatch('user/resetToken').then(() => {
location.reload()
})
})
}
return Promise.reject(new Error(res.message || 'Error'))
} else {
return res
}
},
error => {
console.log('err' + error) // for debug
ElMessage({
message: error.message,
type: 'error',
duration: 5 * 1000
})
return Promise.reject(error)
}
)
export default service
And a peek at my store that actually calls the endpoint
import { login, logout, getInfo } from '../api/user'
actions: { // user login
login({ commit }, userInfo) {
const { username, password } = userInfo
// **Call is made here to 'login'**
return new Promise((resolve, reject) => {
login({ username: username.trim(), password: password }).then(response => {
const { data } = response
commit('SET_TOKEN', data.token)
setToken(data.token)
resolve()
}).catch(error => {
reject(error)
})
})
},
Lastly where the call originates from: this.store.login in my login.vue component
this.store.login('user/login', this.loginForm).then(() => {
this.$router.push({ path: this.redirect || '/' })
this.loading = false
}).catch(() => {
this.loading = false
})
This is my first StackOverflow post, so be kind if I need to include something else and thanks to any ideas or tips. Thank you
I've tried to search on different ports but the route still comes back as undefined.
I tried to change the vite config to include the server option but it still doesn't seem to help.
Not sure If I need to modify the config or not but I did have some settings related to the server mock on my old webpack config

Vue send request until get result from another one

I'm from Ruby language so sorry for noob question or if my concept is wrong - please tell me.
In my Vue application user should provide some data and asynchronously get the result below the form. The flow is like:
user provides input data
the app sends POST request (createProductsRequest) to Rails backend app
Vue get response with load_id which represents id of newly created record e.g. 12345 - sample json: { load_id: 12345 }
Vue app use load_id and send GET request (fetchSyncedProductsResultRequest) to Rails backend app endpoint (sample json: {result: [{test: 'test'}]})
check if response.result is nil ({result: nil}), if yes resent request until it will not be nil
display response data
The question is where (and how actually) to put the loop from step 5 that checks if a given response from step 4 does not contain null? Vue should stop sending requests when response is not nil.
Here's what I've produced so far:
import.js
const createProductsRequest = (self, products) => {
const jwtToken = self.$store.state.idToken;
const payload = JSON.stringify({ product_codes: products['product_codes'].split(',') })
return axios
.post(`/api/v1/imports/products_batches`, payload,{
headers: {
Authorization: `Bearer ${jwtToken}`,
'Content-Type': 'application/json',
'Accept': 'application/json'
}
})
.then(response => response.data)
};
const fetchSyncedProductsResultRequest = (token, id) => {
return axios
.get(`/api/v1/imports/products_batches`, {
params: { id: id },
headers: {
Authorization: `Bearer ${token}`,
}
})
.then(response => {
return response.data['result']
})
};
sync_products.vue
<template>
<div class="col-12 col-md-3">
<button
type="button"
class="btn btn-primary"
#click="syncProducts"
>
Sync
</button>
</div>
</template>
<script>
import {
fetchSyncedProductsResultRequest,
createProductsRequest
} from '../../api/imports'
export default {
name: 'SyncProducts',
data() {
return {
fetchedProductSyncStatus: [],
load_id: ''
}
},
async mounted() {
await fetchSyncedProductsResultRequest(this, id)
this.syncedProductsFetched = true
this.pageChanged(this.currentPage)
},
async mounted() {
const jwtToken = this.$store.state.idToken;
fetchSyncedProductsResultRequest(jwtToken).then(data => {
this.fetchedProductSyncStatus = data
})
},
methods: {
async syncProducts() {
let confirmationText = `Do you want to ${this.productsToSyncAmount} sync products?`
if (this.productsToSyncAmount === 0) {
ModalController.showToast('', 'Type product codes for sync first, please!', 'warning')
}
else if (await ModalController.showConfirmation('Confirmation', confirmationText)) {
try {
ModalController.showLoader()
await createProductsRequest(this, this.styleCodes)
const successMessage = `${this.productsToSyncAmount} products have been queued for sync`
await ModalController.showToast('', successMessage)
} catch (data) {
const errorMessage = `Error occurred during queueing products to sync - `
ModalController.showToast('', errorMessage + data?.message, 'error')
} finally {
this.styleCodes = []
ModalController.hideLoader()
}
}
},
}
}
</script>
To spare your backend i'd probably wait x amount of time (enough for the backend to have created the resource) and then send the get request - instead of potentially spamming it.
With that said i think you want to use the setTimeout function to call a function that makes the API call. There you can make the call, check if result is nil and then use setTimeout and call the function again if needed. Like so:
async loadId() {
const data = await makeApiRequest()
if (!data.result) {
setTimeout(this.loadId, waitTimeInMillis);
return;
}
//Do what you want if when result isn't null.
}

How do I listen for new uploads from a specific channel in the YouTube API?

I am making a Discord bot, and I want it to be able to use the YouTube API to fetch new uploads from a specific channel.
I have searched elsewhere, but they all say how to upload videos, not how to track uploads.
Is this possible, and how can I do it?
Edit: Tried PubSubHubbub but it was very confusing and I couldn't get it to work
Here an example built on top of Node.js (v12) and Fastify and published with ngrok:
I wrote some comments explaining what it is happening:
const fastify = require('fastify')({ logger: true })
const xmlParser = require('fast-xml-parser')
const { URLSearchParams } = require('url')
const fetch = require('node-fetch')
// add an xml parser
fastify.addContentTypeParser('application/atom+xml', { parseAs: 'string' }, function (req, xmlString, done) {
try {
const body = xmlParser.parse(xmlString, {
attributeNamePrefix: '',
ignoreAttributes: false
})
done(null, body)
} catch (error) {
done(error)
}
})
// this endpoint needs for authentication
fastify.get('/', (request, reply) => {
reply.send(request.query['hub.challenge'])
})
// this endpoint will get the updates
fastify.post('/', (request, reply) => {
console.log(JSON.stringify(request.body, null, 2))
reply.code(204)
reply.send('ok')
})
fastify.listen(8080)
.then(() => {
// after the server has started, subscribe to the hub
// Parameter list: https://pubsubhubbub.github.io/PubSubHubbub/pubsubhubbub-core-0.4.html#rfc.section.5.1
const params = new URLSearchParams()
params.append('hub.callback', 'https://1f3dd0c63e78.ngrok.io') // you must have a public endpoint. get it with "ngrok http 8080"
params.append('hub.mode', 'subscribe')
params.append('hub.topic', 'https://www.youtube.com/xml/feeds/videos.xml?channel_id=UCfWbGF64qBSVM2Wq9fwrfrg')
params.append('hub.lease_seconds', '')
params.append('hub.secret', '')
params.append('hub.verify', 'sync')
params.append('hub.verify_token', '')
return fetch('https://pubsubhubbub.appspot.com/subscribe', {
headers: { 'content-type': 'application/x-www-form-urlencoded' },
body: params,
method: 'POST'
})
})
.then((res) => {
console.log(`The status must be 204. Received ${res.status}`)
// shows the error if something went wrong
if (res.status !== 204) {
return res.text().then(txt => console.log(txt))
}
})
I used my channel id to do some testing, consider that the notification is not in real-time, the POSTs are triggered after several minutes usually.

How to manage 429 errors in axios with react native?

I have a react native app that uses MongoDB as the database with express and node js I also use Axios to communicate with the client to the server
Now the app constantly sends and receives data from the database rapidly, e.g a user makes as much as 3 to 4 requests to and from the backend per second when the app is in use,
Everything works fine but there are a lot of 429 errors, how to handle this error or prevent it from occurring without compromising the users experiences a lot?
this below is the axios instanace
const instance = axios.create({ baseURL: 'http://9rv324283.ngrok.io' })
this below is fetching the data from the database
<NavigationEvents
onWillFocus={() => {
try {
const response = await instance.get('fetchNewDishes');
this.setState({data: response.data})
} catch(err) {
console.log(err)
}
}}>
this below is send data to the database
<TouchableOpacity onPress={() => instance.patch(`/postNewDish/${this.state.dish}`)}>
<Text style={{ fontSize: 16, color: '#555', padding: 15 }}>Post Dish</Text>
</TouchableOpacity>
I would suggest you to use axios interceptors to actually trace the error handling in axios , see below example :
import ax from 'axios';
import {config} from '../global/constant';
const baseUrl = config.apiUrl;
let axios = ax.create({
baseURL: baseUrl,
withCredentials: true,
headers: {
'Content-Type': 'application/json;charset=UTF-8',
'Access-Control-Allow-Origin': '*',
},
});
axios.interceptors.request.use(req => handleRequest(req));
axios.interceptors.response.use(
res => handleResponse(res),
rej => handleError(rej),// here if its an error , then call handleError and do what you want to do with error.
);
// sending the error as promise.reject
const handleError = error => {
let errorResponse = {...error};
console.log({...error}, 'error');
return Promise.reject({
data: errorResponse.response.data,
code: errorResponse.response.status,
});
};
Hope it helps. feel free for doubts
Are you in control of the backend? It is possible there is a middleware that limits requests such as express-rate-limit
Make sure to either disable these middlewares, or allow many more requests per minute in the middleware configs.
I had a play around with this using https://httpstat.us/429/cors, which always returns error 429 with retry-after set to 5 (seconds), and came up with this using axios-retry:
import axios from "axios";
import axiosRetry from "axios-retry";
let instance = axios.create({ baseURL: "https://httpstat.us" });
axiosRetry(instance, {
retryCondition: (e) => {
return (
axiosRetry.isNetworkOrIdempotentRequestError(e) ||
e.response.status === 429
);
},
retryDelay: (retryCount, error) => {
if (error.response) {
const retry_after = error.response.headers["retry-after"];
if (retry_after) {
return retry_after;
}
}
// Can also just return 0 here for no delay if one isn't specified
return axiosRetry.exponentialDelay(retryCount);
}
});
// Test for error 429
instance({
url: "/429/cors",
method: "get"
})
.then((res) => {
console.log("429 res: ", res);
})
.catch((e) => {
console.log("429 e: ", e);
});
// Test to show that code isn't triggered by working API call
instance({
url: "/200/cors",
method: "get"
})
.then((res) => {
console.log("200 res: ", res);
})
.catch((e) => {
console.log("200 e: ", e);
});
I'm working on adding this to axios-retry properly for https://github.com/softonic/axios-retry/issues/72

node.js timeout restarts api call

Platform:
I have an api in sails.js and a frontend in react. The calls between front and back end are being made with fetch api.
More information:
In the course of some api endpoints I have to execute an external file, at this point I am using the execfile() function of node.js, and I have to wait for it to be executed to respond to the frontend.
What is the problem?
If the file is executed in a short time, for example less than 1 minute everything runs well and the behavior occurs as expected on both sides, but if (in this case) the file takes more than 1 minute to execute, there is something to trigger a second call to api (I do not know where this is being called, but I tested the same endpoint with postman and I did not have this problem so I suspect the react / fetch-api) and the api call with the same data from the first call is redone. This causes the file to run twice.
Something that is even stranger is that if you have the DevTools Network inspector turned on this second call does not appear, but nothing in the documentation of sailjs points to this behavior.
Example of an endpoint in sails.js:
/**
* FooController
*/
const execFile = require("child_process").execFile;
module.exports = {
foo: async (req, res) => {
let result = await module.exports._executeScript(req.body).catch(() => {
res.status(500).json({ error: "something has occurred" });
});
res.json(result);
},
_executeScript: body => {
return new Promise((resolve, reject) => {
let args = [process.cwd() + "/scripts/" + "myExternalFile.js", body];
let elem = await module.exports
._execScript(args)
.catch(err => reject(err));
resolve(elem);
});
},
_execScript: args => {
return new Promise((resolve, reject) => {
try {
execFile("node", args, { timeout: 150000 }, (error, stdout, stderr) => {
if (error || (stderr != null && stderr !== "")) {
console.error(error);
} else {
console.log(stdout);
}
let output = { stdout: stdout, stderr: stderr };
resolve(output);
});
} catch (err) {
reject(err);
}
});
}
};
Example of component react with fetch call:
import React from "react";
import { notification } from "antd";
import "./../App.css";
import Oauth from "./../helper/Oauth";
import Config from "./../config.json";
class App extends React.Component {
constructor(props) {
super(props);
this.state = {
syncInAction: false,
data: null
};
}
componentDidMount() {
this.handleSync();
}
async handleSync() {
let response = await fetch(Config.apiLink + "/foo/foo", {
method: "POST",
mode: "cors",
headers: {
Authorization: Oauth.isLoggedIn(),
Accept: "application/json",
"Content-Type": "application/json"
},
body: JSON.stringify(this.state.myData)
}).catch(err => {
notification.error({
key: "catch-ApiFail",
message: "Erro"
});
});
let json = await response.json();
this.setState({
syncInAction: false,
data: json
});
}
render() {
return <div>{this.state.data}</div>;
}
}
export default App;
What is my expected goal / behavior:
It does not matter if the call takes 1 minute or 10 hours, the file can only be called once and when it finishes, then yes, it can return to the frontend.
Note that the examples do not have the original code and have not been tested. Is a simplified version of the code to explain the behavior
I ended up solving the problem, apparently nodejs has a default timing of 2 minutes on the server, and can be rewritten to miss this timout or increase it.
This was just adding a line of code at the beginning of the foo() endpoint and the problem was solved.
The behavior of redoing the call is that it is not documented, and it is strange not to have this behavior when using the postman, but here is the solution for whoever needs it.
Final result:
foo: async (req, res) => {
req.setTimeout(0);
let result = await module.exports._executeScript(req.body).catch(() => {
res.status(500).json({ error: "something has occurred" });
});
res.json(result);
};

Categories

Resources