I am trying to build a wikipedia web scraper api and this is my code:
const app = express()
const port = 3000
const url = "https://en.wikipedia.org/wiki/Yeti_Airlines_Flight_691"
axios.get(url).then(async (res) => {
try {
if (res.status == 200) {
const result = // Doing cheerio stuff here
app.get('/', (req, res) => {
res.status(200).send(result)
})
app.listen(port, () => {
console.log(`Example app listening on port ${port}`)
})
}
} finally {
//
}
});
How can I send url dynamically to backend using express and do some stuff then send result back to frontend?
Client side:
This is a function you define in your frontend. You set a request link, which must be known here on client side and server side. Thus take something like /request. Then use axios to send a request to the server. You can pass any parameters with „dynamic“ Information as you called it. The server will receive these informations and can handle them.
const getData = () => {
// insert your server url here instead, with the /request at the end
const requestLink = `http://localhost:3001/request`;
axios
.get(requestLink, {
params: { url: "wikipedia.de/test123" },
})
.catch((error) => {
// here you can implement error handling
console.log(error);
})
.then((res) => {
// print result, received from the server
console.log(res);
});
};
Server side:
The backend is going to wait for a request to the defined link /request.
If he receives such a request, he is executing the code. The req variable contains your dynamic data, such as the wiki url.
Use res.json to send data back to your frontend.
app.get(`/request`, (req, res) => {
// do something with the request here
console.log(req.query);
// send result to the frontend
res.json({
status: "This could be an answer to the frontend.",
});
});
Let's say I have a few routes defined in express.
app.get('/all', async (req, res) => {
// foo, bar, baz are database calls
// since they depend on one another, they are run sequentially with await
const fooRes = await foo();
const barRes = await bar(fooRes);
const bazRes = await baz(barRes);
res.send({data: [fooRes,barRes,bazRes]})
});
app.post('/foo', async (req, res) => {
// updates foo
});
app.post('/bar', async (req, res) => {
// updates bar
});
app.post('/baz', async (req, res) => {
// updates baz
});
If a user requests /all, can another user update /foo while the /all handler is still await-ing from the result from baz(bar)? This would result in potentially stale/incorrect data coming back from the /all handler.
Conversely, if the database calls were not dependent on each other, and the 3 db calls were fired at the almost the same instant in the /all handler
app.get('/all', async (req, res) => {
const data = await Promise.all([
foo(),
bar(),
baz()
]);
res.send({ data })
});
// ... /foo, /bar, /baz update routes as above
Let's assume user1's request for /all hits express first, before user2's update request on /baz does.
Is it correct to assume that user1's baz() function call definitely executes before user2's /baz update?
Is it possible that a freak latency spike when querying the db can cause user1's /all request to return the updated baz value? ie. user2's /baz update actually reaches the db before user1's /all read queries.
I have the following snippet:
const express = require('express')
const app = express()
const cronJob = require('cron').CronJob
app.get('/test', (req, res) => {
// Do something here
}
new cronJob('* * * * * *', () => {
// Call localhost:3000/test here
}, null, true, 'Asia/Manila')
app.listen(3000, () => console.log('Successfully listened to app 3000'))
Usually on node, localhost:3000/test runs if this is called on the browser right? I wanted to make the CRON run this without typing it on the browser once the node app starts. If possible also regardless of the hostname, whether it's localhost or not, the CRON should make the request without being typed on the browser. Can this be done?
I read the comments above on the questions itself and decided to add my thoughts even thought it seems like you got a solution.
In my opinion it will be much more clean for you to call the "method" itself instead of hitting "http" for getting the response you need.
You have 2 options:
Hitting the "domain.com/test" endpoint with a request call.
Simply calling the same method the above url is doing.
In this way, you will "save" the overhead of need to "set-up" a new request to the express app with response and request headers. (example below)
let's say this is your code:
const handleTestData = () => return 'something';
app.get('/test', (req, res) => {
const result = handleTestData();
res.send(result);
}
new cronJob('* * * * * *', () => {
// instead of calling http and getting the response
// and adding overhead, just call the function
const result = handleTestData();
// do what you need with the result
}, null, true, 'Asia/Manila')
I'm looking for an easy solution to front-end and back-end communication.
I want to write simple JS front-end client where a user can put a number between 1 an 10 000 to guess the number that server has generated.
So the client job is to send number that user is guessing. The server should test if secretNumber is higher or lower then that provided by the user and it should send back that info.
For now, my server only sends that secret number. I'm getting it inside my client console, so the connection is working.
My question is how should I modify my server code to read the number value from request, test it and then send the right response (example -> your number is higher than the secretNumber)?
This is my server:
const express = require('express');
const app = express();
const cors = require('cors');
app.use(cors());
app.use((request, response, next) => {
console.log(request.headers);
next();
});
app.use((request, response, next) => {
request.secretNumber = Math.floor(Math.random() * 10000) + 1;
next();
});
app.get('/', (request, response) => {
response.json({
secretNumber: request.secretNumber
});
});
app.listen(3001, () => console.log("Listening on 3001"));
Here is my front-end JS code (I'm using axios):
export function guessNumber(guessValue) {
return dispatch => {
dispatch({ type: GUESS_NUMBER });
axios
.post('/guess', {
isNumber: guessValue,
})
.then(response => {
console.log(response);
})
.catch(error => {
console.log(error);
});
};
}
And I was here looking for answer, but maybe I'm to inexperiened and I need some real example...
First you need to persist the secretNumber between requests. At the moment you are generating a new value on each request.
Assuming just one client using the backend concurrently, you can do this by generating the secretNumber when the server starts and keep it in memory (assign it to a variable).
Then you can simply use route params to capture the client's guess:
app.get('/guess/:guess', (request, response) => {
const guess = params.guess;
// compare guess with secretNumber and respond accordingly
});
Alternatively you can use the request's body (https://expressjs.com/en/4x/api.html#req.body) instead of route params.
I'm making a search-page on my server. When the endpoint is reached and the user waits for the search function to return the results and render the page Express falls through to the 404 handler instead, and I get the following error when I suppose the render function is called:
Error: Can't set headers after they are sent.
What am I doing wrong?
router.get("/", async (req, res) => {
try {
const queryString = req.query.q;
const user = helper.checkAndGetUser(req, res);
let s = String(queryString), searchedTags = [""];
if(s.indexOf(",") > -1){
searchedTags = s.replace(" ", "").split(",");
}
const options = {
"query": {tags: {$all: searchedTags}, _forSale: true}
};
const results = await Search.search(options).then(result => result).catch(err => {
throw err;
});
//This res.render -call is called after the 404 splat-route.
return res.render("partial/search.pug", {user: user, search: {
query: queryString,
results: results
}});
//If I'd use res.send for debugging, it is instead called before the splat-route, like the following:
return res.send(results);
} catch(err) {
next(err);
}
});
module.exports = router;
I register the router:
const search = require("./search.js");
app.use("/search", search);
Followed by the 404 splat-route:
app.get("*", async (req, res, next) => {
const user = helper.checkAndGetUser(req, res);
res.status(404);
res.render("partial/404.pug", {user: user});
});
To clarify:
My question is how can I make the res.render function get called just as the res.send function?
UPDATE [2017-10-05]:
I continued with another part of the site, a similar endpoint, and discovered that sending the result provided by the promise worked as expected if using res.send but not res.render. Using res.render the 404-handler kicked in again. Can this be a bug in Express?
This happens if you attempt to write to res after it is sent, so you must be calling additional code after res.render() or you already responded before calling that.
change it to return res.render(...) so it exits the functions, otherwise it will continue through the function and hit other res.render()s etc.
Something is up with that error handler also. I will update my post in a few mins with tips (on phone). It should probably have (req, res, next) and call return next(err) and pass it to your error handling middleware.
Here is the pattern I like to use in async/await Express:
// these routes occur in the order I show them
app.get('/route', async (req, res, next) => {
try {
const data = 'asdf'
const payload = await something(data)
.then((result) => createPayload(result))
// remember, if you throw anywhere in try block, it will send to catch block
// const something = willFail().catch((error) => {
// throw 'Custom error message:' + error.message
// })
// return from the route so nothing else is fired
return res.render('route', { payload })
} catch (e) {
// fire down to error middleware
return next(e)
}
})
// SPLAT
app.get('*', async (req, res, next) => {
// if no matching routes, return 404
return res.status(404).render('error/404')
})
// ERRORS
app.use(async (err, req, res, next) => {
// if err !== null, this middleware fires
// it has a 4th input param "err"
res.status(500).render('error/500')
// and do whatever else after...
throw err
})
Note: next() callback called without param is treated as no error, and proceeds to the next middleware. If anything is passed in, it will fire the error middleware with the param as the value of err in the error handling middleware. You can use this technique in routes and other middlewares, as long as the error middleware comes last. Mind your use of return with res.send/render() to prevent double setting headers.
NEW:
Something looks a little bit off with that .then() having a callback in it. I don't see logically where err would come from since the value of the resolved promise goes into the .then() function as result. At this point, it is suspect and should be removed or refactored if possible. This part here:
try {
let results = [];
await Search.search(options).then(result => {
results = result;
}, err => {
throw err;
});
console.log("res.render");
return res.render("partial/search.pug", {user: user, search: {
query: string,
results: results
}});
} catch(err) {
next(err);
}
First, here is about what I would expect to see with async/await syntax:
router.get("/", async (req, res, next) => {
try {
const queryString = req.query.q;
const user = helper.checkAndGetUser(req, res);
let s = String(queryString), searchedTags = [""];
if (s.indexOf(",") > -1) {
searchedTags = s.replace(" ", "").split(",");
}
const options = {
"query": { tags: { $all: searchedTags }, _forSale: true }
};
// If a promise is ever rejected inside a try block,
// it passes the error to the catch block.
// If you handle it properly there, you avoid unhandled promise rejections.
// Since, we have async in the route function, we can use await
// we assign the value of Search.search(options) to results.
// It will not proceed to the render statement
// until the entire promise chain is resolved.
// hence, then(data => { return data }) energizes `results`
const results = await Search.search(options)
.then(data => data)
// If any promise in this chain is rejected, this will fire
// and it will throw the error to the catch block
// and your catch block should pass it through to your
// error handling middleware
.catch(err => { throw 'Problem occurred in index route:' + err });
return res.render("partial/search.pug", {
user: user, search: {
query: string,
results: results
}
});
} catch (err) {
// look at the top how we added next as the 3rd, callback parameter
return next(err);
}
});
module.exports = router;
Error handler:
// notice how we add `err` as first parameter
app.use((err, req, res, next) => {
const user = helper.checkAndGetUser(req, res);
res.status(404);
res.render("partial/404.pug", {user: user});
});
From the Express docs:
Define error-handling middleware functions in the same way as other middleware functions, except error-handling functions have four arguments instead of three: (err, req, res, next). For example:
app.use(function (err, req, res, next) {
console.error(err.stack)
res.status(500).send('Something broke!')
})
http://expressjs.com/en/guide/error-handling.html
That might be your true issue because the error handler should only fire if next() is called with any input, but yours appears to be firing every time like a normal middleware, so I suspect it's because there is no err parameter on that middleware function, so it is treated as a normal one.
The Default Error Handler
Express comes with a built-in error handler, which takes care of any errors that might be encountered in the app. This default error-handling middleware function is added at the end of the middleware function stack.
If you pass an error to next() and you do not handle it in an error handler, it will be handled by the built-in error handler; the error will be written to the client with the stack trace. The stack trace is not included in the production environment.
If you call next() with an error after you have started writing the response (for example, if you encounter an error while streaming the response to the client) the Express default error handler closes the connection and fails the request.
So when you add a custom error handler, you will want to delegate to the default error handling mechanisms in Express, when the headers have already been sent to the client:
// code example in docs
Note that the default error handler can get triggered if you call next() with an error in your code more than once, even if custom error handling middleware is in place.
I also recommend using that splat route app.get('*', async (req, res, next) => {}) right above the error handler middlware (aka as the last loaded route in your list). This will catch all unmatched routes, such as /sih8df7h6so8d7f and forward the client to your 404. I think the error handler middlware is more suited for error 500 and clean formatted type errors because it gives you a function that can parse the value of next(err) anytime it is called from a route.
I usually do this for authentication failures with JSON web token (as the first line of code inside every auth required route):
if (!req.person) return res.status(403).render('error/403')
I realize some of this may fry your wig wholesale, so try all this stuff out and see each piece working before you determine if you would like to utilize it or not.
After a few days of going through the code again and again I stumbled upon a problem in the checkAndGetUser-function, that when run without the user being signed in, and since it's faster than the async call to the DB, triggered the splat endpoint and thus showed the 404 page.
I believe the reason for not triggering the splat endpoint when the res.render call was replaced by res.send was that the res.send function is that much faster than the render-call, since it doesn't have to parse any HTML.
Thank you #agm1984 for providing very useful information about the Express framework, if anyone else are having the same or a similar problem, make sure to read through his post.