I'm new to Node and server oriented code and am trying to get multiple svg files which are stored in the server.
Here is my code client-side using jQuery:
$.ajax({
url: someURL,
data: someData
})
.done(function(data) {
console.log('data got', data);
callback(null, data);
})
.fail(function() {
callback(new Error('Cannot access files'));
});
And here is my code server side:
// links is an array of links to the different svg files
var svgs = [];
async.eachSeries(links, function(link, next) {
fs.readFile(link, function(err, svg) {
svgs.push(svg);
next(err);
});
}, function(err) {
if (err) {
response.writeHead(500);
response.end(JSON.stringify(err));
return;
}
response.writeHead(200);
response.end(svgs); // Doesn't work
// response.end(svgs[0]); // Works
});
As long as I send only one file to the browser (which seem to be a Buffer instance), everything seems to work fine, but when I try to send multiple ones as an Array the transaction succeed but I got nothing in my returned data. That may be related to the MIME type of what I'm trying to send, but I couldn't find how to handle that.
You'll have to convert svgs into a String or Buffer first. One option is to stringify it as JSON:
response.writeHead(200, {
'Content-Type': 'application/json'
});
response.end(JSON.stringify(svgs));
This is because response.write(), which response.end() is calling to handle the data (svgs), doesn't accept Arrays.
chunk can be a string or a buffer. If chunk is a string, the second parameter specifies how to encode it into a byte stream. By default the encoding is 'utf8'.
Whereas each svg provided by fs.readFile() is a Buffer, so it has no issues writing svgs[0].
Related
I'm trying to upload an arbitrary list of files:
for (let i=0;i<files.length;i++) {
let fileTypeToUse = files[i].type;
fetchWithRetry(url, 1000, 2, {
method: 'POST',
credentials: 'same-origin',
headers: {
'Content-Type': fileTypeToUse
},
body: files[i],
}
}
This works for most file types (including images) by taking the bytes and sending them in the body of the request. But when I try to upload audio of type "audio/mpeg" my server receives a file which is about 60% larger than I expected.
I initially assumed this meant the file was being base64 encoded by the browser, so I tried to decode the file. Unfortunately, this hypothesis seemed to be incorrect. I received a decoding error on the server: base64.StdEncoding.Decode: illegal base64 data at input byte 3
For reference, here is a screenshot of the files object I am trying to upload:
And here is the oversized object being sent by the browser to the server:
Related issue: https://stackoverflow.com/a/40826943/12713117 Unfortunately they are encouraging people to upload the file with form/multipart instead of directly uploading the file.
I have 2 questions. First, why is the uploaded object larger than the file accessible in javascript? Second, how do I know if an arbitrary file will be sent as-is (which is the case with images) or if it will be encoded and need decoding on the server?
Fetch With Retry Code
function fetchWithRetry(url, delay, tries, fetchOptions = {}) {
return fetch(url, fetchOptions).catch((err) => {
let triesLeft = tries - 1;
if (triesLeft == null || triesLeft < 1) {
throw err;
}
return wait(delay).then(() => fetchWithRetry(url, delay * 2, triesLeft, fetchOptions));
});
}
I am trying to build the following and I am learning JS as I go, so I would appreciate some guidance and explanation on how to approach the following:
I have two pages, one with client-side script, that takes an uploaded image, converts it to base64 and fetches it to the second page, where the server-side script uploads it to a location via API.
This is the fetch on my first page:
fetch("xxxxxx", {
method: "POST",
headers: {
"Content-Type": "application/json"
},
body: JSON.stringify({
fileEnc: fileEnc,
fileName: fileName,
fileType: fileType
})
})
.then(function (res) {
console.log("Success!", res);
})
.catch(function (err) {
console.log("Error!", err);
});
};
On my second page, an API call is made and I get the following response:
console.log(uploadImage.StatusCode)
My question is:
How do I pass the response back to my first page and how do I display it?
Thanks
Second page should write the status code in the response rather than in the console. And that text will be available in the "then" part of the fetch function.
assuming that the second page was opened from the first page, write a function in the first page where you want to consume that data which is executed from the second page with the data you need as an argument:
page 1
function logData(data) {
console.log('#data from api: ', data)
}
page2
// some code here
window.opener.logData(dataFromApi)
I have a response from an ebay-api
--MIMEBoundaryurn_uuid_C91296EA5FF69EE9571479882375576565344 Content-Type: application/xop+xml; charset=utf-8; type="text/xml"
Content-Transfer-Encoding: binary Content-ID:
<0.urn:uuid:C91296EA5FF69EE9571479882375576565345>
Success1.1.02016-11-23T06:26:15.576Z514
--MIMEBoundaryurn_uuid_C91296EA5FF69EE9571479882375574545344 Content-Type: application/zip Content-Transfer-Encoding: binary
Content-ID:
PKY'uIi[��#�50014028337_report.xmlUT y�2Xy�2Xux
00�R�j�#��+��[��PlX#�(�x,=l�q]Lfewc��w Ĥ��O��١�HT���t��GGT�
��6�;���'������.$����=d����m;c}Wߦ�RW�A
f�����g�I��4U��x��3��f���ғ{f��xj�,+���ۖI%5��B's��G,#��t,L{�c�����MD笓��)!�9��
�M�o;8_��<�i�y����sz���u���=��Ջ^2�S��%+2�2�`QV�$�����~?�w�ǥ�_Q�퉦�'PKY'uIi[��#���50014028337_report.xmlUTy�2Xux
00PK\�
--MIMEBoundaryurn_uuid_C91296EA5FF69EE9571479882375576565344--
This is of type string. and i extracted the attached zip file data i.e.
PKY'uIi[��#�50014028337_report.xmlUT y�2Xy�2Xux
00�R�j�#��+��[��PlX#�(�x,=l�q]Lfewc��w Ĥ��O��١�HT���t��GGT�
��6�;���'������.$����=d����m;c}Wߦ�RW�A
f�����g�I��4U��x��3��f���ғ{f��xj�,+���ۖI%5��B's��G,#��t,L{�c�����MD笓��)!�9��
�M�o;8_��<�i�y����sz���u���=��Ջ^2�S��%+2�2�`QV�$�����~?�w�ǥ�_Q�퉦�'PKY'uIi[��#���50014028338_report.xmlUTy�2Xux
00PK\�
This shows that it has a report.xml in it. So when i write this data in a zip file, it creates a zip file and upon extract gives error.
fs.writeFile("./static/DownloadFile.zip", fileData, 'binary', function(err){
if (err) throw err;
console.log("success");
});
How can i write this data in a zip file properly. Pls advice. If required any more information.
EDIT:
I tried writing the zip file in PHP and is succssfully writing it with this code:
$zipFilename="DownloadFile.zip";
$data = $fileData;
$handler = fopen($zipFilename, 'wb')
or die("Failed. Cannot Open $zipFilename to Write!</b></p>");
fwrite($handler, $data);
fclose($handler);
Please advice how can i achieve the same thing in nodejs.
Depending on what HTTP Client you are using the implementation might change a little.
With axios I'm doing something like so:
I'm requesting a zip file so I specify the Accept header as application/zip
In order to get a buffer and not Binary, specify the responseType as arrayBuffer
const res = await axios.get('/routToThat/file', {
headers: {
Accept: 'application/zip',
},
responseType: 'arraybuffer',
});
By doing the latter, instead of receiving a Binary from the response:
A#B�ArE⏾�7�ϫ���f�걺N�����Yg���o_M^�D�T�U X_���e?� hi\...
I receive a Buffer:
Buffer(22781691) [80, 75, 3, …]
Once the request is resolved and I have that Buffer, I use that same writeFile function from fs
NOTE: I'm not specifying the Encoding in writeFile
fs.writeFile(name, res.data, (err) => {
if (err) throw err;
console.log("success");
});
As I see in your code example your binary data is already mangled by request module. Just use in request setting
encoding:null
and the zip file is a valid binary in body (now buffer instead of utf-8 string!) you can decompress. As long as you see the questions marks you still have the encoding issue.
I am moving over from jQuery AJAX requests to the new Fetch API (nothing against jQuery, I still have it in my site, but Fetch looks - according to Jake Archibald and David Walsh and also IMHO - to be the new way of sending async requests).
As such, with jQuery, I had the following function (more or less):
function ajaxCall(type, url, data) {
return $.ajax({
type: type,
url: url,
data: data,
})
.fail(function(xhr, status, errorThrown) {
// Do fail stuff
})
.always(function(xhr, status) {
// Do always stuff
});
}
// Later...
var myAjax = ajaxCall(myType, myUrl, myData);
myAjax.done(function(xhr) {
// Do done stuff
});
This way, I could have one function be called to handle any and all ajax requests I could ever need (for the most part at least...). Note that I do not declare a dataType, as I use jQuery's intelligent guess. This way my server can send me whatever response and I could handle it (probably a smarter way to do this would be to pass another parameter with the data type - in the case the "intelligent guess" goes wrong, but this was the way I set it up).
I am now trying to recreate the above with the new Fetch API. What I have so far currently looks like this:
function fetchCall(url, method, body) {
// This if statement is supposed to handle
// query selectors (which in GET requests go in the url)
// on GET requests - as opposed to POST req's which go in the body
if (method === 'GET') {
var data = body;
url = new URL(url, location.protocol + '//' + location.host + '/');
Object.keys(data).forEach(key => url.searchParams.append(key, data[key]));
body = undefined;
}
return fetch(url, {
method: method,
body: body
}).then(function(res) {
if (res.ok) return res;
throw new Error('Server error. Status code: ', res.status);
}).catch(function(err) {
console.log(err);
});
}
// Later...
var myFetch = fetchCall(myUrl, myMethod, myBody);
myFetch.then(function(res) {
console.log(res);
});
The problem I am running into is that if res.ok return res; does not state what type of response it is (i.e. res.json(), res.blob(), res.text(), etc.).
Thus, I am wondering how to set up a dynamic way of setting the type of response body. Is this even possible at the Fetch API's current state of development? Is it just that there is something I am not duplicating in MDN?
After messing around with this, I also realized I could make it always set to return res.text(); and the if the call is supposed to be JSON, use JSON.parse(response);, but I do want it to be dynamic. What if I end up wanting to return a blob()?
So, as far as the conversation has reached, there is a way to understand what type of content has been received, with two remarks:
Typically you have to always know and expect exact content type, and a universal solution is rather odd in case of fetching from a certain remote endpoint, and
The Content-Type header is what will tell you the type of content received, but the server may send a wrong header, which is very unusual to happen and therefore is negligible.
The Response object has header property that is (kind of) a Map, so you can use its get method to get a value by key.
The easiest and cleanest way to check if the returned value is a certain MIME type you expect is by using a regular expression:
// replace url with the actual API endpoint URL
fetch(url).then(response => {
const contentType = response.headers.get('Content-Type'); // -> "text/html; charset=utf-8"
if (/text\/html/i.test(contentType)) {
// do something when the Content-Type is text/html
} else if (/application\/json/.test(contentType)) {
// do something when the Content-Type is application/json
}
// and so on, for every Content-Type you need.
}).catch(error => {
// do something when error happens
});
In my node.js webpage I'm making a page preview similar to the Facebook link preview. I'm making a call to get the html of the page, and use it to create the preview.
$.ajax({
type: 'GET',
data: { "html": url },
url: "/htmlTest",
success: function (data) {
imgArray = [];
$('img', data).each(function () {
imgArray.push(this.src);
});
...
This is the server-side code that handles the request.
app.get('/htmlTest', function (req, res) {
res.writeHead(200, { 'content-type': 'text/html' });
request(req.query.html, function (error, response, body) {
if (error) {
res.write(error.toString());
res.end('\n');
}
else if (response.statusCode == 200) {
res.write(body);
res.end('\n');
}
})
});
Now what I've been noticing, is that it will just insert any css the other page uses into my page, which can really screw everything up. Why is this happening?
Also, while I'm at it, does anyone have any better ideas for a facebook-style page preview?
No. writeHead writes HTTP headers to the underlying TCP stream. It has absolutely nothing to do with HTML.
You're running into an issue because your server returns the wholesale HTML content of the requested URL. You then pass this string into jQuery, which is apparently adding contained CSS styles into your document.
Generally, it is a terrible idea to take random code from a user-supplied URL and run in the context of your page. It opens you to gaping security holes – the CSS artifacts you're seeing are one example.
To be blunt, your code has numerous problems, so bear with me as I point out some issues.
app.get('/htmlTest', function (req, res) {
res.writeHead(200, { 'content-type': 'text/html' });
Here, you respond to the browser with a success status (200) beore your server actually does anything. This is incorrect: you should only respond with either a success or error code after you know if the request succeeded or failed.
request(req.query.html, function (error, response, body) {
if (error) {
res.write(error.toString());
res.end('\n');
}
Here would be a good place to respond with an error code, since we know that the request did actually fail. res.send(500, error) would do the trick.
else if (response.statusCode == 200) {
res.write(body);
res.end('\n');
}
And here's where we could respond with a success code. Rather than use writeHead, use Express's set and send methods – things like Content-Length will be correctly set:
res.set('Content-Type', 'text/html');
res.send(body);
Now what happens if response.statusCode != 200? You don't handle that case. error is only set in the case of network errors (such as inability to connect to the target server). The target server can still respond with a non-200 status, and your node server would never respond to the browser. In fact, the connection would hang open until the user kills it. This could be fixed with a simple else res.end().
Even with these issues resolved, we still haven't addressed the fact that it's not a good idea to try to parse arbitrary HTML in the browser.
If I were you, I'd use something that parses HTML into a DOM on the server, and then I'd return only the necessary information back to the browser as JSON. cheerio is the module you probably want to use – it looks just like jQuery, only it runs on the server.
I'd do this:
var cheerio = require('cheerio'), url = require('url'), request = require('request');
app.get('/htmlTest', function(req, res) {
request(req.query.url, function(err, response, body) {
if (err) res.send(500, err); // network error, send a 500
else if (response.status != 200) res.send(500, { httpStatus: response.status }); // server returned a non-200, send a 500
else {
// WARNING! We should probably check that the response content-type is html
var $ = cheerio.load(body); // load the returned HTML into cheerio
var images = [];
$('img').each(function() {
// Image srcs can be relative.
// You probably need the absolute URL of the image, so we should resolve the src.
images.push(url.resolve(req.query.url, this.src));
});
res.send({ title: $('title').text(), images: images }); // send back JSON with the image URLs
}
});
});
Then from the browser:
$.ajax({
url: '/htmlTest',
data: { url: url },
dataType: 'json',
success: function(data) {
// data.images has your image URLs
},
error: function() {
// something went wrong
}
});