Have an express app that saves a sanitized url to a mongodb database and I want to render the decoded url in a res.json using decodeURI() but it doesn't work as expected and only gives the encoded version back. If I do a res.send(decodeURI(url)) it works. How can I get the res.json to send the decoded url.
// Create a url object with escaped and trimmed data.
var Url = new UrlModel(
{ url: req.body.url }
);
if (!errors.isEmpty()) {
// There are errors. Render the form again with error messages.
res.render('index', { errors: errors.array()});
return;
}
else {
// Data from form is valid.
// Check if Url with same name already exists.
UrlModel.findOne({ 'url': req.body.url })
.exec( function(err, found_url) {
if (err) { return next(err); }
if (found_url) {
// Url exists, redirect to its detail page.
res.json({"original_url": decodeURI(found_url.url) });
//res.send(decodeURI(found_url.url))
}
Update:
Probably wasn't clear in my question. My input is from a mongodb with a sanitized url in the form
https://www.facebook.com
so its html entities that i want to convert and I dont think that decodeUri does that.
My out put from this code
res.json({original_url:found_url.url, decoded: decodeURI(found_url.url) });
is {"original_url":"https://www.facebook.com","decoded":"https://www.facebook.com"}
so the // in the url is not being converted to // . Is there some core javascript function that does this or do I have to use a function with regx and replace?
Updated after question update.
In JavaScript you have some functions to accomplish a similar conversion: encodeURI and encodeURIComponent, and their counterparts decodeURI and decodeURIComponent. encodeURI is used to safely encode a full URL as it won't encode the protocol, hostname or the path; encodeURIComponent will encode everything.
What you are showing in the edited question has nothing (as far as I can tell) to do with JavaScript; you need to get the backend to unsanitize that string before sending it back to you.
If updating the backend is not an option, you could try something like this:
unescape('https://www.facebook.com'.replace(/&#x/g, '%').replace(/;/g, ''))
This will decode those entities into their actual characters, but it should not be a permanent solution as it is marked as deprecated.
Original response.
I am having no issues at all with encodeURI and decodeURI. Are you completely sure it is not being returned as expected? Is there a chance something else in the middle is encoding it again?
I tested this small snippet with Postman.
const express = require('express');
const app = express();
const encoded = encodeURI('http://example.com?query=ÅÍÎÏ˝ÓÔÒÚÆ☃');
const decoded = decodeURI(encoded);
app.get('/json', (req, res) => res.json({ encoded, decoded }));
app.listen(3000, () => console.log('Example app listening on port 3000!'));
Related
I'm doing a link shortener. The link shortener works very well but now I'm trying to do API for it. The problem is that if I pass the URL argument in the get URL it's not working. I tried a lot of things but it's not working. When I do like http://localhost:3500/api/create/google.com it works but when I do http://localhost:3500/api/create/https://google.com it's not working because of the https://. These are the last 3 inputs via my API that failed: http://https:google.com, google.com, http://
I'm using express and mongoose. Here's my code:
app.get('/api/create/:shortUrl(*)', async (req, res, next) => {
if (req.params.shortUrl.includes("https://") || req.params.shortUrl.includes("http://") || req.params.shortUrl.includes("/")) {
req.params.shortUrl = req.params.shortUrl.replace("https://", "").replace("http://", "").replace("/", "")
}
if (req.params.shortUrl == "") {
res.send("invalid URL")
}
await shorturl.create({full: `http://${req.params.shortUrl}`})
const shortUrls = await shorturl.find().sort({_id: -1}).limit(-1)
const latest = shortUrls[0]
res.send("https://p33t.link/" + latest.short)
});
You have to properly encode portions of the URL that contain restricted characters such as : and // that aren't part of the actual protocol to make it a legal URL. So, the idea is that you encode the "parameter" before appending it to the URL. Presumably, you would use encodeURIComponent(), depending upon exactly where you're placing it in the URL.
After parsing the core part of the URL, a web server will decode the remaining components of the URL and give you back the original characters. I would suggest that your particular use would probably work better as a query parameter rather than a part of the path which would give you this when properly encoded:
http://localhost:3500/api/create?u=https%3A%2F%2Fgoogle.com
And, you could then use:
app.get('/api/create', (req, res) => {
console.log(req.query.u);
...
});
I've spent 4 hours trying to parse a simple JSON from Twilio.
The flow is:
Text message containing magnet link
Twilio proxies request to my serverless function on cloud
Parse req. to get the value
Twilio Studio UI
Code
....
var app = express()
app.use(bodyParser.urlencoded({ extended: false }))
app.use(bodyParser.json())
module.exports.magnet = async (event) => {
let requestBody = ''
try {
requestBody = JSON.parse(event.body)
requestBody = requestBody["magnet"]
} catch (err) {
console.error(err)
}
await beginAuth(requestBody
....
I'm just getting malformed JSON. When I play around with stringify and parse together, I just either get malformed error or I get an added \r escape character, which also causes issues.
Not sure if this is just Twilio or me. I just want the magnet link as a string.
I tried
JSON.parse(JSON.stringify(event.body))
but that also didn't help.
Sample Payload
{"magnet": "magnet:?xt=urn:btih:9970E5BF56EDDB06024EF1311109865B893C8EB4&dn=Westworld+-+Season+3+-+Mp4+x264+AC3+1080p&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.zer0day.to%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.pirateparty.gr%3A6969%2Fannounce&tr=udp%3A%2F%2Feddie4.nl%3A6969%2Fannounce&tr=udp%3A%2F%2Fcoppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337&tr=udp%3A%2F%2Ftracker.pirateparty.gr%3A6969&tr=udp%3A%2F%2Feddie4.nl%3A6969&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A80&tr=udp%3A%2F%2Ftracker.zer0day.to%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fcoppersurfer.tk%3A6969%2Fannounce"}
Twilio sends the body usually formatted as a query string, i.e. you have to decode it like this:
const querystring = require('querystring');
requestBody = querystring.parse(event.body);
requestBody = requestBody['magnet']
To verify that it's sent as a query string simply print the event.body after receiving it, in your example it should look similar to this:
magnet=magnet%3A%3Fxt%3Durn%3Abtih%3A9970E5BF56EDDB06024EF1311109865B893C8EB4%26dn%3DWestworld%2B-%2BSeason%2B3%2B-%2BMp4%2Bx264%2BAC3%2B1080p%26tr%3Dudp%253A%252F%252Ftracker.leechers-paradise.org%253A6969%252Fannounce%26tr%3Dudp%253A%252F%252Ftracker.coppersurfer.tk%253A6969%252Fannounce%26tr%3Dudp%253A%252F%252Ftracker.coppersurfer.tk%253A80%252Fannounce%26tr%3Dudp%253A%252F%252Ftracker.opentrackr.org%253A1337%252Fannounce%26tr%3Dudp%253A%252F%252Ftracker.zer0day.to%253A1337%252Fannounce%26tr%3Dudp%253A%252F%252Ftracker.pirateparty.gr%253A6969%252Fannounce%26tr%3Dudp%253A%252F%252Feddie4.nl%253A6969%252Fannounce%26tr%3Dudp%253A%252F%252Fcoppersurfer.tk%253A6969%252Fannounce%26tr%3Dudp%253A%252F%252Ftracker.leechers-paradise.org%253A6969%26tr%3Dudp%253A%252F%252Ftracker.coppersurfer.tk%253A6969%26tr%3Dudp%253A%252F%252Ftracker.opentrackr.org%253A1337%26tr%3Dudp%253A%252F%252Ftracker.pirateparty.gr%253A6969%26tr%3Dudp%253A%252F%252Feddie4.nl%253A6969%26tr%3Dudp%253A%252F%252Ftracker.coppersurfer.tk%253A80%26tr%3Dudp%253A%252F%252Ftracker.zer0day.to%253A1337%252Fannounce%26tr%3Dudp%253A%252F%252Ftracker.leechers-paradise.org%253A6969%252Fannounce%26tr%3Dudp%253A%252F%252Fcoppersurfer.tk%253A6969%252Fannounce
I'm trying to understand how to deal(in a secure way) with Path Traversal.
For example an application receives from a client a file name via REST API in JSON, look for it in the non-accessible(by outside) directory and retrieve a response with the file:
app.get('/', (req, res) => {
const fileName = req.body.fileName;
// some code...
fs.stat(`./nonAccessibleDir/${fileName}`, async function(err, stat) {
// some code...
});
// some code...
}
The problem with the above approach is that a client can send something like "../" in the fileName request and it will "eat" it without an issue. How can one deal with this kind of scenarios, what and how I should fix this vulnerability, please?
Update:
Sorry, but I forgot to mention that I know I should check the input I receive, but what if I need to pass the "/" and "." in the input? Also, if I don't need this characters, is that all I need to check to remove the Path Traversal vulnerability?
An easy way would be to validate the fileName through a regex that detects any ../ segments and returns an error if any are present.
if (fileName.match(/\.\.\//g) !== null) {
// return an api error
}
You could have quite a tight validation rule that prevents any forward slashes in fileName at all, making it only possible to point to a file directly in your desired directory.
I'm making a webpage with Node JS with dustjs and PostgreSQL. How do I make a search query in the html, so I can pass the value to the app.get
Do I need to use JQuery?
app.get('/teachers', function(req, res){
pool.connect(function(err, client, done){
if(err) {
return console.error("error", err);
}
client.query('SELECT * FROM teachers', function(err, result){
if(err){
return console.error('error running query', err)
}
res.render('teacherindex', {teachers: result.rows});
done();
});
});
});
app.get('/teachers/:str', (req,res)=>{
pool.connect((err, client, done) => {
if (err) throw err
client.query('SELECT * FROM teachers WHERE name = $1', [req.query.namesearch], (err, result) => {
done()
if (err) {
console.log(err.stack)
} else {
res.render('teacherindex', {teachers: result.rows});
}
})
})
})
This is my JQuery
$("#myBtn").click(function(){
var str = $("#myInput").val();
var url = '/teachers/'+str;
if(confirm('Search Record?')){
$.ajax({
url: url,
type: 'put',
success: function(result){
console.log('Searching');
window.location.href='/teachers';
},
error: function(err){
console.log(err);
}
});
}
});
My HTML
<input type="text" id="myInput" data-id="namesearch">
<button type="button" id="myBtn">Show Value</button>
Thank you!
FINAL ANSWER:
Ok so it turns out the issue you were having was something completely different. You are trying to use server side rendering for this, and I was showing you how to render the retrieved data on the client side.
I have forked, and updated your repo - which can be found at the link below..
Please review my changes and let me know if you have any questions.
Working repo: https://github.com/oze4/hanstanawi.github.io
Demo Video: https://raw.githubusercontent.com/oze4/hanstanawi.github.io/master/fake_uni_demo.mp4
EDIT:
I went ahead and built a repository to try and help you grasp these concepts. You can find the repo here - I tried to keep things as simple and understandable as possible, but let me know if you have any questions.
I had to make some minor changes to the paths, which I have commented explanations on the code in the repo.
I am using a "mock" database (just a JSON object in a different file) but the logic remains the same.
The index.js is the main entry point and contains all route data.
The index.html file is what gets sent to the user, and is the main HTML file, which contains the jQuery code.
If you download/fork/test out the code in that repo, open up your browsers developer tools, go to the network tab, and check out the differences.
Using req.params
Using req.query
ORIGINAL ANSWER:
So there are a couple of things wrong with your code and why you are unable to see the value of the textbox server side.
You are sending a PUT request but your server is expecting a GET request
You are looking for the value in req.query when you should be looking for it in req.params
You are looking for the incorrect variable name in your route (on top of using query when you should be using params) req.query.namesearch needs to be req.params.str
See here for more on req.query vs req.params
More detailed examples below.
In your route you are specifying app.get - in other words, you are expecting a GET request to be sent to your server.. but your are sending a PUT request..
If you were sending your AJAX to your server by using something like /teachers?str=someName then you would use req.query.str - or if you wanted to use namesearch you would do: /teachers?namesearch=someName and then to get the value: req.query.namesearch
If you send your AJAX to your server by using the something like /teachers/someName then you should be using req.params.str
// ||
// \/ Server is expecting a GET request
app.get('/teachers/:str', (req, res) => {
// GET THE CORRECT VALUE
let namesearch = req.params.str;
pool.connect((err, client, done) => {
// ... other code here
client.query(
'SELECT * FROM teachers WHERE name = $1',
// SPECIFY THE CORRECT VALUE
namesearch,
(err, result) => {
// ... other code here
})
})
});
But in your AJAX request, you are specifying PUT.. (should be GET)
By default, AJAX will send GET requests, so you really don't have to specify any type here, but I personally like to specify GET in type, just for the sake of brevity - just more succinct in my opinion.
Again, specifying GET in type is not needed since AJAX sends GET by default, specifying GET in type is a matter of preference.
$("#myBtn").click(function () {
// ... other code here
let textboxValue = $("#myTextbox").val();
let theURL = "/teachers/" + textboxValue;
// OR if you wanted to use `req.query.str` server side
// let theURL = "/teachers?str=" + textboxValue;
if (confirm('Search Record?')) {
$.ajax({
url: theURL,
// ||
// \/ You are sending a PUT request, not a GET request
type: 'put', // EITHER CHANGE THIS TO GET OR JUST REMOVE type
// ... other code here
});
}
});
It appears you are grabbing the value correctly from the textbox, you just need to make sure your server is accepting the same type that you are sending.
I have the following route in my express application :
app.get('/api/:URI', (req, res) => {
doStuff();
}
The URI parameter passed is an URI encoded on the client side with encodeURIComponent()
It works fine except when the URI contains a hash.
Example: http://foo.bar/foobar/bla#blabla-313fe4ce-4f8d-48b7-b0f3-a59844402ee8
In this case the route is ignored.
On the browser side I receive a code 301, then the result of the next valid route.
If I remove the hash or, weirder, if I disable the cache on the browser side it works perfectly.
Is there any way express can ignore the hash ?
Edit : It's absolutely not a Can I use an at symbol (#) inside URLs? duplicate, the question is more about express routing and/or about browsers cache issues than about allowed characters in an URL.
Is there any way express can ignore the hash ?
I tried using the OR operator. For example,
app.get('/blog' || '/blog#top', (request, response) => {
...
});
So it works even if #top is present or not.