This question already has answers here:
Encoding issue with requesting JSON from StackOverflow API
(2 answers)
Closed 1 year ago.
Problem
Related to Get UTF-8 html content with Node's http.get - but that answer isn't working for me.
I'm trying to call the Stack Overflow API questions endpoint:
https://api.stackexchange.com/2.3/questions?site=stackoverflow&filter=total
Which should return the following JSON response:
{"total":21951385}
Example Code
I'm using the https node module to submit a get request like this:
const getRequest = (url: string) => new Promise((resolve, reject) => {
const options: RequestOptions = {
headers: {
'Accept': 'text/*',
'Accept-Encoding':'identity',
'Accept-Charset' : 'utf8',
}
}
const req = get(url, options, (res) => {
res.setEncoding('utf8');
let responseBody = '';
res.on('data', (chunk) => responseBody += chunk);
res.on('end', () => resolve(responseBody));
});
req.on('error', (err) => reject(err));
req.end();
})
And then invoking it like this:
const questionsUrl = 'https://api.stackexchange.com/2.3/questions?&site=stackoverflow&filter=total'
const resp = await getRequest(questionsUrl)
console.log(resp)
However, I get the response:
▼�
�V*�/I�Q�22�454�0�♣��♥‼���↕
What I've Tried
I've tried doing several variations of the following:
I'm calling setEncoding to utf8 on the stream
I've set the Accept header to text/* - which
Provides a text MIME type, but without a subtype
I've set the Accept-Encoding header to identity - which
Indicates the identity function (that is, without modification or compression)
This code also works just fine with pretty much any other API server, for example using the following url:
https://jsonplaceholder.typicode.com/todos/1
But the StackOverlow API works anywhere else I've tried it, so there must be a way to instruct node how to execute it.
My suggestion is to use an http library that supports both promises and gzip built in. My current favorite is got(). http.get() is like the least featured http request library anywhere. You really don't have to write all this yourself. Here's what your entire code would look like with the got() library:
const got = require('got');
function getRequest(url) {
return got(url).json();
}
This library handles all these things you need for you automatically:
Promises
JSON conversion
Gzip decoding
2xx status detection (other status codes like 404 are turned into a promise rejection which your code does not do).
And, it has many, many other useful features for other general use. The days of coding manually with http.get() should be long over. No need to rewrite code that has already been written and well-tested for you.
FYI, there's a list of very capable http libraries here: https://github.com/request/request/issues/3143. You can pick the one that has the API you like the best.
Response Header - Content-Encoding - Gzip
As jfriend00 pointed out - looks like the server isn't respecting the Accept-Encoding value being passed and returning a gzipped response none-the-less.
Unzipping Response
According to the answer on How do I ungzip (decompress) a NodeJS request's module gzip response body?, you can unzip like this:
import { get } from 'https'
import { createGunzip } from 'zlib'
const getRequest = (url: string) => new Promise((resolve, reject) => {
const req = get(url, (res) => {
const buffer: string[] = [];
if (!res.headers['content-encoding']?.includes('gzip')) {
console.log('utf8')
res.on('data', (chunk) => buffer.push(chunk));
res.on('end', () => resolve(buffer.join("")))
} else {
console.log('gzip')
const gunzip = createGunzip();
res.pipe(gunzip);
gunzip.on('data', (data) => buffer.push(data.toString()))
gunzip.on("end", () => resolve(buffer.join("")))
gunzip.on("error", (e) => reject(e))
}
});
req.on('error', (err) => reject(err));
req.end();
})
Related
How do I download a file into memory via http in nodejs, without the use of third-party libraries?
This answer solves a similar question, but I don't need to write file to disk.
You can use the built-in http.get() and there's an example right in the nodejs http doc.
http.get('http://nodejs.org/dist/index.json', (res) => {
const { statusCode } = res;
const contentType = res.headers['content-type'];
let error;
// Any 2xx status code signals a successful response but
// here we're only checking for 200.
if (statusCode !== 200) {
error = new Error('Request Failed.\n' +
`Status Code: ${statusCode}`);
} else if (!/^application\/json/.test(contentType)) {
error = new Error('Invalid content-type.\n' +
`Expected application/json but received ${contentType}`);
}
if (error) {
console.error(error.message);
// Consume response data to free up memory
res.resume();
return;
}
res.setEncoding('utf8');
let rawData = '';
res.on('data', (chunk) => { rawData += chunk; });
res.on('end', () => {
try {
const parsedData = JSON.parse(rawData);
console.log(parsedData);
} catch (e) {
console.error(e.message);
}
});
}).on('error', (e) => {
console.error(`Got error: ${e.message}`);
});
This example assumes JSON content, but you can change the process in the end event handler to just treat the rawData as text and change the check for json contentType to whatever type you are expecting.
FYI, this is somewhat lower level code and is not something I would normally use. You can encapsulate it in your own function (perhaps with a promise wrapped around it) if you really don't want to use third party libraries, but most people use higher level libraries for this purpose that just make the coding simpler. I generally use got() for requests like this and there is a list of other libraries (all promise-based) here.
I transformed my code so that instead of requiring an extra node_modules, I could just use some HTTPS GET requests, the problem is that when I try to pipe /releases/ which is basically a raw JSON file, my code requires it back and issues occur like SyntaxError: Unexpected end of JSON input, because for some reason, when I console.log() the so called JSON array, the end isn't completed with ] or }. So I try to pipe the response into an array, but now I get an error: dest.on isn't a function,
Code:
https.get({hostname: `api.github.com`, path: `/repos/${username}/${reponame}/releases`, headers: {'User-Agent': 'a user agent'}}, async (response) => {
var file = new Array()
response.pipe(file)
response.on('end', async function() { //issue occurs at response.pipe ???
var releases = JSON.parse(fs.readFileSync('./releases.json', 'utf8'))
console.log(releases)
The JSON file that I access from Github looks like: https://api.github.com/repos/davidmerfield/randomColor/releases (random repository)
But, my file (releases.json) looks like this
Edit: I did extensive testing. I used the same JSON file my pkg outputted, read it with fs and so on, and everything seems fine. So the issue is most likely with https / response
I found out how to pipe the HTTP request into an object, instead of piping it into a file. Thanks to this post. I did that and turned the string into a JSON array.
https.get({hostname: `api.github.com`, path: `/repos/${username}/${reponame}/releases`, headers: {'User-Agent': 'agent'}}, async response => {
var str = ''
response.on('data', (data) => {
str += data
})
response.on('end', async function() {
var releases = JSON.parse(str)
//and so on...
You can require JSON files. So, if you need this file, you can do something like:
const releases = require('./releases.json');
You do not need to read it with fs, unless you really want to.
TypeError: dest.on is not a function
This error will be thrown if you try to pipe to non-Writable Stream object. Check here
Which in this case Array is not a Writable Stream. You can create a writable stream using fs.createWriteStream() and pipe the response to it.
https.get(
{ hostname: `api.github.com`, path: `/repos/${username}/${reponame}/releases`, headers: { "User-Agent": "a user agent" } },
async response => {
const writableStreamFile = fs.createWriteStream("./releases.json");
response.pipe(writableStreamFile);
response.on("end", async function() {
var releases = JSON.parse(fs.readFileSync("./releases.json", "utf8"));
console.log(releases);
});
}
);
I have very little experience working with Node.js and jQuery and have been searching for the last few hours for a solution. I have an API from openweathermap.com () that returns weather information in the JSON format, and I am trying to pull the temperature value.
I am using Node.js to run a program that can be accessed from any device on the network and I have previously used jQuery on the client to read the file using $.getJSON but am in the process transferring most of my code to the server side to prevent needing a browser open at all times in order for the program to run properly. Obviously you can't use jQuery with node.js but i tried server adaptations for node.js including cheerio, jsdom, and a standard jquery add-on but none of them would do the trick. I can't use XMLHttpRequest or http.get because its being run server side and I can't simply use JSON.parse because it is pulling from a website.
How can I pull the data from the website, store it as an object, and then pull data from it while using just pure javascript?
Here is what I originally had running on the client:
var updateWeather = function(){
$.getJSON('http://api.openweathermap.org/data/2.5/weather?id=5802340&units=imperial&appid=80e9f3ae5074805d4788ec25275ff8a0&units=imperial', function(data) {
socket.emit("outsideTemp",data.main.temp);
});
}
updateWeather();
<head>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.4.0/jquery.min.js"></script>
</head>
NodeJS natively supports JSON -- so no "special" work needed. I would recommend using an http client that makes our life easier, like axios, but you can do this natively. I have provided two snippets below for you to get started:
Using popular HTTP Client
const axios = require('axios');
axios.get('http://api.openweathermap.org/data/2.5/weather?id=5802340&units=imperial&appid=80e9f3ae5074805d4788ec25275ff8a0&units=imperial').then((res) => {
console.log(res.data)
})
Plain NodeJS (taken from the NodeJS Docs)
const http = require('http');
http.get('http://api.openweathermap.org/data/2.5/weather?id=5802340&units=imperial&appid=80e9f3ae5074805d4788ec25275ff8a0&units=imperial', (res) => {
const { statusCode } = res;
const contentType = res.headers['content-type'];
let error;
if (statusCode !== 200) {
error = new Error('Request Failed.\n' +
`Status Code: ${statusCode}`);
} else if (!/^application\/json/.test(contentType)) {
error = new Error('Invalid content-type.\n' +
`Expected application/json but received ${contentType}`);
}
if (error) {
console.error(error.message);
// Consume response data to free up memory
res.resume();
return;
}
res.setEncoding('utf8');
let rawData = '';
res.on('data', (chunk) => { rawData += chunk; });
res.on('end', () => {
try {
const parsedData = JSON.parse(rawData);
console.log(parsedData);
} catch (e) {
console.error(e.message);
}
});
}).on('error', (e) => {
console.error(`Got error: ${e.message}`);
});
Many people use request / request promise with node
const req = require('request-promise');
req.get({
uri: 'http://api.openweathermap.org/data/2.5/weather?id=5802340&units=imperial&appid=80e9f3ae5074805d4788ec25275ff8a0&units=imperial',
json: true
}).then(e => {console.log(e.coord)});
I am using Node.js 8.0.0 and wanted to update a file on a platform. For this they have an API with very clear ways to use it.
In this case I have to use a PUT method for this, also the right hostname and path, right auth keys and the file itself that must be of Content-Type: multipart/form-data. So I really wanted to use the node https module and try to not install anything else.
I tried using the request http client (https://github.com/request/request)and worked like a charm but as I previously told, would like to use what we already have in Node without installing anything else. I see some working replies here using Request, but no one using Node https module.
Using the https.request I managed to go to the right URL, pass auth, but it always shows me a correlation error (specifically for this platform not something you can Google I guess).
function update(method, path, params) {
return new Promise((resolve, reject) => {
https.request({
method,
host: HOST,
path: path + (params ? '?' + qs.stringify(params) : ''),
auth: `${USER}:${PASS}`,
formData: document,
}, res => {
let body = '';
res
.on('data', message => body += message)
.on('error', (e) => console.log(e))
.on('end', () => resolve(body));
})
.end();
Where: const document = fs.createReadStream(path.resolve(__dirname, '../../src/', 'myfile.xlf'));
And where I call the update function like this:
await operations.update('PUT', '/right/path/to/update', {
id: `${rightId}`,
});
With this code I don't have any auth problem and I can communicate with the platform, in fact if I use other api method (GET, POST) I can obtain statistics and things like that, but in this case explained before, the response have a 400 Bad Request error, that I am sure is a problem with the way I am "trying to send the file".
Using the request http client, I get no errors and managed to update the document with this code:
function update(path, params) {
const url = 'https://' + HOST + path + (params ? '?' + qs.stringify(params) : '');
return new Promise((resolve, reject) => {
try {
resolve(requests.put({
url,
formData: document,
}).auth(USER, PASS));
} catch (err) {
reject(err);
}
});
}
I am currently in the process of creating a REST API for my personal website. I'd like to include some downloads and I would like to offer the possibility of selecting multiple ones and download those as a zip file.
My first approach was pretty easy: Array with urls, request for each of them, zip it, send to user, delete. However, I think that this approach is too dirty considering there are things like streams around which seems to be quite fitting for this thing.
Now, I tried around and am currently struggling with the basic concept of working with streams and events throughout different scopes.
The following worked:
const r = request(url, options);
r.on('response', function(res) {
res.pipe(fs.createWriteStream('./file.jpg'));
});
From my understanding r is an incoming stream in this scenario and I listen on the response event on it, as soon as it occurs, I pipe it to a stream which I use to write to the file system.
My first step was to refactor this so it fits my case more but I already failed here:
async function downloadFile(url) {
return request({ method: 'GET', uri: url });
}
Now I wanted to use a function which calls "downloadFile()" with different urls and save all those files to the disk using createWriteStream() again:
const urls = ['https://download1', 'https://download2', 'https://download3'];
urls.forEach(element => {
downloadFile(element).then(data => {
data.pipe(fs.createWriteStream('file.jpg'));
});
});
Using the debugger I found out that the "response" event is non existent in the data object -- Maybe that's already the issue? Moreover, I figured that data.body contains the bytes of my downloaded document (a pdf in this case) so I wonder if I could just stream this to some other place?
After reading some stackoveflow threads I found the following module: archiver
Reading this thread: Dynamically create and stream zip to client
#dankohn suggested an approach like that:
archive
.append(fs.createReadStream(file1), { name: 'file1.txt' })
.append(fs.createReadStream(file2), { name: 'file2.txt' });
Making me assume I need to be capable of extracting a stream from my data object to proceed.
Am I on the wrong track here or am I getting something fundamentally wrong?
Edit: lmao thanks for fixing my question I dunno what happened
Using archiver seems to be a valid approach, however it would be advisable to use streams when feeding large data from the web into the zip archive. Otherwise, the whole archive data would need to be held in memory.
archiver does not support adding files from streams, but zip-stream does. For reading a stream from the web, request comes in handy.
Example
// npm install -s express zip-stream request
const request = require('request');
const ZipStream = require('zip-stream');
const express = require('express');
const app = express();
app.get('/archive.zip', (req, res) => {
var zip = new ZipStream()
zip.pipe(res);
var stream = request('https://loremflickr.com/640/480')
zip.entry(stream, { name: 'picture.jpg' }, err => {
if(err)
throw err;
})
zip.finalize()
});
app.listen(3000)
Update: Example for using multiple files
Adding an example which processes the next file in the callback function of zip.entry() recursively.
app.get('/archive.zip', (req, res) => {
var zip = new ZipStream()
zip.pipe(res);
var queue = [
{ name: 'one.jpg', url: 'https://loremflickr.com/640/480' },
{ name: 'two.jpg', url: 'https://loremflickr.com/640/480' },
{ name: 'three.jpg', url: 'https://loremflickr.com/640/480' }
]
function addNextFile() {
var elem = queue.shift()
var stream = request(elem.url)
zip.entry(stream, { name: elem.name }, err => {
if(err)
throw err;
if(queue.length > 0)
addNextFile()
else
zip.finalize()
})
}
addNextFile()
})
Using Async/Await
You can encapsulate it into a promise to use async/await like:
await new Promise((resolve, reject) => {
zip.entry(stream, { name: elem.name }, err => {
if (err) reject(err)
resolve()
})
})
zip.finalize()