How do I download a file into memory via http in nodejs, without the use of third-party libraries?
This answer solves a similar question, but I don't need to write file to disk.
You can use the built-in http.get() and there's an example right in the nodejs http doc.
http.get('http://nodejs.org/dist/index.json', (res) => {
const { statusCode } = res;
const contentType = res.headers['content-type'];
let error;
// Any 2xx status code signals a successful response but
// here we're only checking for 200.
if (statusCode !== 200) {
error = new Error('Request Failed.\n' +
`Status Code: ${statusCode}`);
} else if (!/^application\/json/.test(contentType)) {
error = new Error('Invalid content-type.\n' +
`Expected application/json but received ${contentType}`);
}
if (error) {
console.error(error.message);
// Consume response data to free up memory
res.resume();
return;
}
res.setEncoding('utf8');
let rawData = '';
res.on('data', (chunk) => { rawData += chunk; });
res.on('end', () => {
try {
const parsedData = JSON.parse(rawData);
console.log(parsedData);
} catch (e) {
console.error(e.message);
}
});
}).on('error', (e) => {
console.error(`Got error: ${e.message}`);
});
This example assumes JSON content, but you can change the process in the end event handler to just treat the rawData as text and change the check for json contentType to whatever type you are expecting.
FYI, this is somewhat lower level code and is not something I would normally use. You can encapsulate it in your own function (perhaps with a promise wrapped around it) if you really don't want to use third party libraries, but most people use higher level libraries for this purpose that just make the coding simpler. I generally use got() for requests like this and there is a list of other libraries (all promise-based) here.
Related
I'm trying to get the temperature data from my node.js backend sent to react.js but i kept getting res.send is not a funtion
Sample code here
app.get("/gettemperature", (req, res) => {
const email = req.query.email;
let stmt = `SELECT * FROM users WHERE email=?`;
let todo = [email];
db.query(stmt, todo, (err, results, fields) => {
if (err) {
console.error(err.message);
}
if(results.length > 0 ){
let id = results[0].id;
let getID = `SELECT * FROM controlModules WHERE deviceowner=?`;
let getidData = [id];
db.query(getID, getidData, (err, resulta, fields) => {
if (err) {
console.error(err.message);
}
if(resulta.length > 0){
let lanip = resulta[0].ipaddress;
let url = "http://"+lanip+"/data";
http.get(url,(res) => {
let body = "";
res.on("data", (chunk) => {
body += chunk;
});
res.on("end", () => {
try {
let json = JSON.parse(body);
const temp_actual = json.temperature.value;
console.log(temp_actual);
res.setHeader('Content-Type', 'application/json');
res.end(
JSON.stringify({
value: temp_actual
})
);
} catch (error) {
console.error(error.message);
};
});
}).on("error", (error) => {
console.error(error.message);
});
}
});
}
});
});
i really need to return/send/respond the temperature data to my front end but i'm getting said error, is there a different way to return data?
It looks like you are mixing up an HTTP server you wrote in Node (although you haven't shown any relevant code) and an HTTP client you also wrote in Node.
res is an argument received by the callback you pass to http.get and contains data about the response received by your HTTP client.
Meanwhile, somewhere else (not shown) you have a different variable also called res which is the object your HTTP server uses to send its response to the browser running your React code.
You are calling res.send and wanting res to be the latter but it is really the former.
Since you haven't shown us the HTTP server code, it is hard to say where that res is, but there is a good chance you have shadowed it and can solve your problem by using different names (e.g. client_res and server_res).
That said. I strongly recommend avoiding using the http module directly as the API follows out of date design patterns and isn't very friendly. Consider using fetch or axios for making HTTP requests and Express.js for writing HTTP servers.
This question already has answers here:
Encoding issue with requesting JSON from StackOverflow API
(2 answers)
Closed 1 year ago.
Problem
Related to Get UTF-8 html content with Node's http.get - but that answer isn't working for me.
I'm trying to call the Stack Overflow API questions endpoint:
https://api.stackexchange.com/2.3/questions?site=stackoverflow&filter=total
Which should return the following JSON response:
{"total":21951385}
Example Code
I'm using the https node module to submit a get request like this:
const getRequest = (url: string) => new Promise((resolve, reject) => {
const options: RequestOptions = {
headers: {
'Accept': 'text/*',
'Accept-Encoding':'identity',
'Accept-Charset' : 'utf8',
}
}
const req = get(url, options, (res) => {
res.setEncoding('utf8');
let responseBody = '';
res.on('data', (chunk) => responseBody += chunk);
res.on('end', () => resolve(responseBody));
});
req.on('error', (err) => reject(err));
req.end();
})
And then invoking it like this:
const questionsUrl = 'https://api.stackexchange.com/2.3/questions?&site=stackoverflow&filter=total'
const resp = await getRequest(questionsUrl)
console.log(resp)
However, I get the response:
▼�
�V*�/I�Q�22�454�0�♣��♥‼���↕
What I've Tried
I've tried doing several variations of the following:
I'm calling setEncoding to utf8 on the stream
I've set the Accept header to text/* - which
Provides a text MIME type, but without a subtype
I've set the Accept-Encoding header to identity - which
Indicates the identity function (that is, without modification or compression)
This code also works just fine with pretty much any other API server, for example using the following url:
https://jsonplaceholder.typicode.com/todos/1
But the StackOverlow API works anywhere else I've tried it, so there must be a way to instruct node how to execute it.
My suggestion is to use an http library that supports both promises and gzip built in. My current favorite is got(). http.get() is like the least featured http request library anywhere. You really don't have to write all this yourself. Here's what your entire code would look like with the got() library:
const got = require('got');
function getRequest(url) {
return got(url).json();
}
This library handles all these things you need for you automatically:
Promises
JSON conversion
Gzip decoding
2xx status detection (other status codes like 404 are turned into a promise rejection which your code does not do).
And, it has many, many other useful features for other general use. The days of coding manually with http.get() should be long over. No need to rewrite code that has already been written and well-tested for you.
FYI, there's a list of very capable http libraries here: https://github.com/request/request/issues/3143. You can pick the one that has the API you like the best.
Response Header - Content-Encoding - Gzip
As jfriend00 pointed out - looks like the server isn't respecting the Accept-Encoding value being passed and returning a gzipped response none-the-less.
Unzipping Response
According to the answer on How do I ungzip (decompress) a NodeJS request's module gzip response body?, you can unzip like this:
import { get } from 'https'
import { createGunzip } from 'zlib'
const getRequest = (url: string) => new Promise((resolve, reject) => {
const req = get(url, (res) => {
const buffer: string[] = [];
if (!res.headers['content-encoding']?.includes('gzip')) {
console.log('utf8')
res.on('data', (chunk) => buffer.push(chunk));
res.on('end', () => resolve(buffer.join("")))
} else {
console.log('gzip')
const gunzip = createGunzip();
res.pipe(gunzip);
gunzip.on('data', (data) => buffer.push(data.toString()))
gunzip.on("end", () => resolve(buffer.join("")))
gunzip.on("error", (e) => reject(e))
}
});
req.on('error', (err) => reject(err));
req.end();
})
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 1 year ago.
The community reviewed whether to reopen this question 1 year ago and left it closed:
Original close reason(s) were not resolved
Improve this question
The title really explains it.
I made a discord bot, and I added a ranking system that had its memory in my filesystem. However, if too many people join, my storage would just get filled up. Perhaps there is a way for me to access the node.js server? Maybe localhost? I just want anything that saves data for free, is not managed by anyone other than me, and doesn't take up my storage.
config
For this answer we'll establish a simple config object to store any values -
// config.json
{"counter":0}
server
We will create a simple server using http.createServer. We will use the request method and request URL to look up a handler or respond with 404 when no handler is found -
// server.js
import { createServer } from "http"
import { readFile, writeFile } from "fs/promises"
const server = createServer(async (req, res) => {
const handler = routes?.[req.method.toLowerCase()]?.[req.url]
if (handler == null) {
res.writeHead(404, {'Content-Type': 'text/plain'})
res.end(`No route for ${req.method} ${req.url}`)
}
else {
await handler(req, res)
res.end()
}
})
server.listen(8000)
Next we define the routes to /getConfig and /saveConfig -
// server.js (continued)
const routes = {
get: {
"/getConfig": async (req, res) => {
res.writeHead(200, {'content-type': 'application/json'})
res.write(await readFile("./config.json"))
}
},
post: {
"/saveConfig": async (req, res) => {
await writeFile("./config.json", await readBody(req))
res.writeHead(204)
},
"/reset": async (req, res) => {
await writeFile("./config.json", JSON.stringify({ counter: 0 }))
res.writeHead(204)
}
}
}
This depends on a reusable helper, readBody -
// server.js (continued)
function readBody(req) {
return new Promise((resolve, reject) => {
const body = []
req.on('data', chunk => body.push(Buffer.from(chunk)))
req.on('end', _ => resolve(Buffer.concat(body).toString()))
req.on('error', reject)
})
}
client
In this case your bot is the http client. The node docs for http.get include this long-winded example, but don't let it worry you -
// example from node docs
http.get('http://localhost:8000/', (res) => {
const { statusCode } = res;
const contentType = res.headers['content-type'];
let error;
// Any 2xx status code signals a successful response but
// here we're only checking for 200.
if (statusCode !== 200) {
error = new Error('Request Failed.\n' +
`Status Code: ${statusCode}`);
} else if (!/^application\/json/.test(contentType)) {
error = new Error('Invalid content-type.\n' +
`Expected application/json but received ${contentType}`);
}
if (error) {
console.error(error.message);
// Consume response data to free up memory
res.resume();
return;
}
res.setEncoding('utf8');
let rawData = '';
res.on('data', (chunk) => { rawData += chunk; });
res.on('end', () => {
try {
const parsedData = JSON.parse(rawData);
console.log(parsedData);
} catch (e) {
console.error(e.message);
}
});
}).on('error', (e) => {
console.error(`Got error: ${e.message}`);
});
You're not expected to copy this verbatim. Imagine writing that much code each time you wanted to fetch some JSON. You can think of the http module as a low-level API that enables you to design higher-level functions -
// client.js
import * as http from "http"
function request (href, { body = "", ...options } = {}) {
return new Promise((resolve, reject) =>
http.request(href, options, res => {
const data = []
res.on('data', chunk => data.push(chunk))
res.on('end', _ => resolve({
status: res.statusCode,
headers: res.headers,
data: Buffer.concat(data).toString()
}))
})
.on('error', reject)
.end(body)
)
}
Above our request function resolves a { status, headers, data } object, and we can write specialized forms get and getJson that make it even easier to intereact with -
// client.js (continued)
async function get (href) {
const { status, headers, data } = await request(href)
if (status < 200 || status >= 300)
throw Error(status)
return { status, headers, data }
}
async function getJson (href) {
const { headers, data } = await get(href)
if (!headers['content-type'].startsWith("application/json"))
throw Error(`expected application/json but received ${headers['content-type']}`)
return JSON.parse(data)
}
We can do the same for post -
// client.js (continued)
async function post (href, body = "") {
const { status, headers, data } = await request(href, { body, method: "POST" })
if (status < 200 || status >= 300)
throw Error(status)
return { status, headers, data }
}
Finally here is our bot code. It reads the config via get, updates the config via post, and re-reads it via get to return the confirmed result -
// client.js (continued)
async function bot() {
const config = await getJson("http://localhost:8000/getConfig")
await post("http://localhost:8000/saveConfig", JSON.stringify({counter: config.counter + 1}))
return getJson("http://localhost:8000/getConfig")
}
bot().then(console.log, console.error)
run
Start the server in your terminal -
$ node ./server.js
In a separate terminal, run the client a few times -
$ node ./client.js
{ counter: 1 }
$ node ./client.js
{ counter: 2 }
$ node ./client.js
{ counter: 3 }
node modules
Above we took a sort of DIY approach to the problem. But this kind of problem has been solved many ways before. There are popular libraries like express and koajs that would make much of this a lot easier. Now that you know the purpose they serve, give 'em a try!
Just use a database, mongoDB atlas will work well in your case because it is cloud based and very easy to set up. You can follow this tutorial to connect your discord bot with mongoDB atlas.
I have very little experience working with Node.js and jQuery and have been searching for the last few hours for a solution. I have an API from openweathermap.com () that returns weather information in the JSON format, and I am trying to pull the temperature value.
I am using Node.js to run a program that can be accessed from any device on the network and I have previously used jQuery on the client to read the file using $.getJSON but am in the process transferring most of my code to the server side to prevent needing a browser open at all times in order for the program to run properly. Obviously you can't use jQuery with node.js but i tried server adaptations for node.js including cheerio, jsdom, and a standard jquery add-on but none of them would do the trick. I can't use XMLHttpRequest or http.get because its being run server side and I can't simply use JSON.parse because it is pulling from a website.
How can I pull the data from the website, store it as an object, and then pull data from it while using just pure javascript?
Here is what I originally had running on the client:
var updateWeather = function(){
$.getJSON('http://api.openweathermap.org/data/2.5/weather?id=5802340&units=imperial&appid=80e9f3ae5074805d4788ec25275ff8a0&units=imperial', function(data) {
socket.emit("outsideTemp",data.main.temp);
});
}
updateWeather();
<head>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.4.0/jquery.min.js"></script>
</head>
NodeJS natively supports JSON -- so no "special" work needed. I would recommend using an http client that makes our life easier, like axios, but you can do this natively. I have provided two snippets below for you to get started:
Using popular HTTP Client
const axios = require('axios');
axios.get('http://api.openweathermap.org/data/2.5/weather?id=5802340&units=imperial&appid=80e9f3ae5074805d4788ec25275ff8a0&units=imperial').then((res) => {
console.log(res.data)
})
Plain NodeJS (taken from the NodeJS Docs)
const http = require('http');
http.get('http://api.openweathermap.org/data/2.5/weather?id=5802340&units=imperial&appid=80e9f3ae5074805d4788ec25275ff8a0&units=imperial', (res) => {
const { statusCode } = res;
const contentType = res.headers['content-type'];
let error;
if (statusCode !== 200) {
error = new Error('Request Failed.\n' +
`Status Code: ${statusCode}`);
} else if (!/^application\/json/.test(contentType)) {
error = new Error('Invalid content-type.\n' +
`Expected application/json but received ${contentType}`);
}
if (error) {
console.error(error.message);
// Consume response data to free up memory
res.resume();
return;
}
res.setEncoding('utf8');
let rawData = '';
res.on('data', (chunk) => { rawData += chunk; });
res.on('end', () => {
try {
const parsedData = JSON.parse(rawData);
console.log(parsedData);
} catch (e) {
console.error(e.message);
}
});
}).on('error', (e) => {
console.error(`Got error: ${e.message}`);
});
Many people use request / request promise with node
const req = require('request-promise');
req.get({
uri: 'http://api.openweathermap.org/data/2.5/weather?id=5802340&units=imperial&appid=80e9f3ae5074805d4788ec25275ff8a0&units=imperial',
json: true
}).then(e => {console.log(e.coord)});
I have a problem in nodejs, I make a request to an api using https.request, the response contains an object of 10000 rows.
What happens is that the entire object does not arrive, and parsing gives the error: Unexpected end of JSON input;
Can someone help?
Function to request:
function request({
options,
method,
resource,
queryParams,
bodyParams,
}) {
return new Promise((resolve, reject) => {
const hasBodyParams = !!bodyParams;
const stringifyedQueryParams = strigifyQueryParams(queryParams);
const optionsRequest = {
...options,
method,
path: `${resource}${stringifyedQueryParams}`,
};
const req = https.request(optionsRequest, (res) => {
res.setEncoding(configs.ENCODING);
res.on(events.DATA, data => resolve({
body: JSON.parse(data),
statusCode: res.statusCode,
}));
});
req.on(events.ERROR, error => reject(error) );
hasBodyParams && req.write(bodyParams);
req.end();
});
}
As I suspected in the comments, you're not handling multiple data-events.
When receiving large responses from a request, the data-event is called multiple times, each time with a chunk of data from the response (not the complete response).
When you're parsing a chunk, the complete JSON document hasn't been transmitted yet, so the parsing fails with the "Unexpected end of JSON stream" error
In short, you need to:
Create a variable to collect the complete body
On a data-event, append the new chunk to the complete body
When the end-event is called, parse the full body.
Here is a short example, adopted from the official documentation:
https.request(options, (res) => {
// PARTIAL example
res.setEncoding("utf8"); // makes sure that "chunk" is a string.
let fullBody = "";
res.on("data", data => {
fullBody += data;
});
res.on("end", () => {
const json = JSON.parse(fullBody);
// work with json
});
});