Develop Reference

Save responses to multiple GET requests in a single local json file - node.js

The problem:
I have a function that maps over countries and regions and creates an array of urls, then makes a GET request to each one. I want to save the responses in a single json file, and I want this function to handle that as well.
Expected results:
I expected to be able to run the function as needed (like when source data is updated), and get a new or updated local json file with all the data objects in one array.
Actual results:
A file with only one record, an array with the last response object.
What I've tried:
I tried using fs.writeFile and fs.readFile. I did not get any errors, but the resulting file had only one record, even though console showed all the requests being made. It seemed that each response was being written over the previous.
Minimum reproducable (node.js) example:
const fs = require('fs')
// subset of countries and codes for demo purposes
const countryDirList = [
'antarctica',
'central-asia',
]
const fbCountryCodes = [
{ "region": "antarctica", "codes": ["ay", "bv"] },
{ "region": "central-asia", "codes": ["kg", "kz"] },
]
const callingUrlsInSequence = async () => {
fs.writeFile('./test.json', '[]', function (err) {
if (err) throw err
console.log('File - test.json - was created successfully.')
})
try {
const urlConstructor = countryDirList.map(async (country) => {
console.log('countries mapped', country)
return fbCountryCodes.filter(async (f) => {
if (country === f.region) {
const urls = f.codes.map(async (c) => {
const response = await axios({
method: 'get',
url: `https://raw.githubusercontent.com/factbook/factbook.json/master/${country}/${c}.json`,
responseType: 'json',
headers: {
'Content-Type': 'application/json',
},
})
fs.readFile('./test.json', function (err, data) {
let json = JSON.parse(data)
json.push(response.data)
setTimeout(() => {
fs.writeFile('./test.json', JSON.stringify(json), function (err) {
if (err) throw err
console.log('The "data to append" was appended to file!')
})
}, 1000)
})
return response.data
})
const dataArr = await Promise.all(urls)
dataArr.map((item) =>
console.log(
'dataArr',
item.Government['Country name']['conventional short form']
)
)
}
})
})
} catch (err) {
console.log('axios error: ', err)
}
}
callingUrlsInSequence()
I'm re-writing this question now because it kept getting downvoted, and I could see that it was not very concise.
I can also see now, that obviously, the fs.readFile inside the fs.writeFile is not going to work in the code I provided, but I'm leaving it there in case it might help someone else, combined with the solution I provided in response to my own question.

I ended up learning how to solve this problem with both node-fetch and axios. They are not exactly the same.
For both:
First, check for existence of destination file, and create one if it's not already there.
const createNew = () => {
try {
if (existsSync('./data.json')) {
console.log('file exists')
return
} else {
writeFile('./data.json', '[]', (error, data) => {
if (error) {
console.log('fs.writeFile - create new - error: ', error)
return
}
})
}
} catch (err) {
console.log('fs.existsSync error: ', err)
}
}
createNew()
Then make the array of urls:
const countryDirList = [...countries]
const fbCountryCodes = [...codes]
const urls = []
// maybe a reducer function would be better, but my map + filter game is much stronger X-D
const makeUrls = (countriesArr, codesArr) =>
countriesArr.map((country) => {
return codesArr.filter((f) => {
if (country === f.region) {
return f.codes.map((code) => {
return urls.push(
`https://raw.githubusercontent.com/factbook/factbook.json/master/${country}/${code}.json`
)
})
}
})
})
makeUrls(countryDirList, fbCountryCodes)
Next, make the requests.
Axios:
fs.readFile('./data.json', (error, data) => {
if (error) {
console.log(error)
return
}
Promise.all(
urls.map(async (url) => {
let response
try {
response = await axios.get(url)
} catch (err) {
console.log('axios error: ', err)
return err
}
return response
})
)
.then((res) => {
const responses = res.map((r) => r.data)
fs.writeFile('./data.json', JSON.stringify(responses, null, 2), (err) => {
if (err) {
console.log('Failed to write data')
return
}
console.log('Updated data file successfully')
})
})
.catch((err) => {
console.log('axios error: ', err)
})
})
Node-fetch:
//same basic structure, readFile with fetch and write file inside
fs.readFile('./data2.json', (error, data) => {
if (error) {
console.log(error)
return
}
async function fetchAll() {
const results = await Promise.all(
urls.map((url) => fetch(url).then((r) => r.json()))
)
fs.writeFile('./data2.json', JSON.stringify(results, null, 2), (err) => {
if (err) {
console.log('Failed to write data')
return
}
console.log('Updated data file successfully')
})
}
fetchAll()
})
Both methods produce exactly the same output: a json file containing a single array with however many response objects in it.

Unsubscribe email using Fetch api Javascript

I have a form where i enter an email and it gets ''subscribed'' in a user.json file using a fetch api on node server.My task is to :
upon clicking on the "Unsubscribe" button, implement the functionality for unsubscribing from the community list. For that, make POST Ajax request using http://localhost:3000/unsubscribe endpoint.
I tried to make the function but it wasnt succeseful so i deleted it. Also,i need to do the following :
While the requests to http://localhost:3000/subscribe and
http://localhost:3000/unsubscribe endpoints are in progress, prevent
additional requests upon clicking on "Subscribe" and "Unsubscribe".
Also, disable them (use the disabled attribute) and style them using
opacity: 0.5.
For me ajax requests,fetch and javascript is something new,so i dont know really well how to do this task,if you could help me i'll be happy,thanks in advance.
fetch code for subscribing:
import { validateEmail } from './email-validator.js'
export const sendSubscribe = (emailInput) => {
const isValidEmail = validateEmail(emailInput)
if (isValidEmail === true) {
sendData(emailInput);
}
}
export const sendHttpRequest = (method, url, data) => {
return fetch(url, {
method: method,
body: JSON.stringify(data),
headers: data ? {
'Content-Type': 'application/json'
} : {}
}).then(response => {
if (response.status >= 400) {
return response.json().then(errResData => {
const error = new Error('Something went wrong!');
error.data = errResData;
throw error;
});
}
return response.json();
});
};
const sendData = (emailInput) => {
sendHttpRequest('POST', 'http://localhost:8080/subscribe', {
email: emailInput
}).then(responseData => {
return responseData
}).catch(err => {
console.log(err, err.data);
window.alert(err.data.error)
});
}
index.js from route node server:
const express = require('express');
const router = express.Router();
const FileStorage = require('../services/FileStorage');
/* POST /subscribe */
router.post('/subscribe', async function (req, res) {
try {
if (!req.body || !req.body.email) {
return res.status(400).json({ error: "Wrong payload" });
}
if (req.body.email === 'forbidden#gmail.com') {
return res.status(422).json({ error: "Email is already in use" });
}
const data = {email: req.body.email};
await FileStorage.writeFile('user.json', data);
await res.json({success: true})
} catch (e) {
console.log(e);
res.status(500).send('Internal error');
}
});
/* GET /unsubscribe */
router.post('/unsubscribe ', async function (req, res) {
try {
await FileStorage.deleteFile('user.json');
await FileStorage.writeFile('user-analytics.json', []);
await FileStorage.writeFile('performance-analytics.json', []);
await res.json({success: true})
} catch (e) {
console.log(e);
res.status(500).send('Internal error');
}
});
module.exports = router;
And user.json file looks like this :
{"email":"Email#gmail.com"}
This is my attempt for unsubscribing :
export const unsubscribeUser = () => {
try {
const response = fetch('http://localhost:8080/unsubscribe', {
method: "POST"
});
if (!response.ok) {
const message = 'Error with Status Code: ' + response.status;
throw new Error(message);
}
const data = response.json();
console.log(data);
} catch (error) {
console.log('Error: ' + error);
}
}
It gives the following errors:
Error: Error: Error with Status Code: undefined
main.js:2
main.js:2 POST http://localhost:8080/unsubscribe 404 (Not Found)
FileStorage.js:
const fs = require('fs');
const fsp = fs.promises;
class FileStorage {
static getRealPath(path) {
return `${global.appRoot}/storage/${path}`
}
static async checkFileExist(path, mode = fs.constants.F_OK) {
try {
await fsp.access(FileStorage.getRealPath(path), mode);
return true
} catch (e) {
return false
}
}
static async readFile(path) {
if (await FileStorage.checkFileExist(path)) {
return await fsp.readFile(FileStorage.getRealPath(path), 'utf-8');
} else {
throw new Error('File read error');
}
}
static async readJsonFile(path) {
const rawJson = await FileStorage.readFile(path);
try {
return JSON.parse(rawJson);
} catch (e) {
return {error: 'Non valid JSON in file content'};
}
}
static async writeFile(path, content) {
const preparedContent = typeof content !== 'string' && typeof content === 'object' ? JSON.stringify(content) : content;
return await fsp.writeFile(FileStorage.getRealPath(path), preparedContent);
}
static async deleteFile(path) {
if (!await FileStorage.checkFileExist(path, fs.constants.F_OK | fs.constants.W_OK)) {
return await fsp.unlink(FileStorage.getRealPath(path));
}
return true;
}
}
module.exports = FileStorage;

You should consider using a database for handling CRUD operations on your persisted data. If you must use filestorage, theres a flat file DB library called lowdb that can make working the files easier.
As for preventing duplicate requests, you can track if user has already made a request.
let fetchBtn = document.getElementById('fetch')
let isFetching = false
fetchBtn.addEventListener('click', handleClick)
async function handleClick(){
if (isFetching) return // do nothing if request already made
isFetching = true
disableBtn()
const response = await fetchMock()
isFetching = false
enableBtn()
}
function fetchMock(){
// const response = await fetch("https://example.com");
return new Promise(resolve => setTimeout (() => resolve('hello'), 2000))
}
function disableBtn(){
fetchBtn.setAttribute('disabled', 'disabled');
fetchBtn.style.opacity = "0.5"
}
function enableBtn(){
fetchBtn.removeAttribute('disabled');
fetchBtn.style.opacity = "1"
}
<button type="button" id="fetch">Fetch</button>

Decode a Uint8Array into a JSON

I am fetching data from an API in order to show sales and finance reports, but I receive a type gzip file which I managed to convert into a Uint8Array. I'd like to somehow parse-decode this into a JSON file that I can use to access data and create charts in my frontend with.
I was trying with different libraries (pako and cborg seemed to be the ones with the closest use cases), but I ultimately get an error Error: CBOR decode error: unexpected character at position 0
This is the code as I have it so far:
let req = https.request(options, function (res) {
console.log("Header: " + JSON.stringify(res.headers));
res.setEncoding("utf8");
res.on("data", function (body) {
const deflatedBody = pako.deflate(body);
console.log("DEFLATED DATA -----> ", typeof deflatedBody, deflatedBody);
console.log(decode(deflatedBody));
});
res.on("error", function (error) {
console.log("connection could not be made " + error.message);
});
});
req.end();
};
I hope anyone has stumbled upon this already and has some idea.
Thanks a lot!

Please visit this answer https://stackoverflow.com/a/12776856/16315663 to retrieve GZIP data from the response.
Assuming, You have already retrieved full data as UInt8Array.
You just need the UInt8Array as String
const jsonString = Buffer.from(dataAsU8Array).toString('utf8')
const parsedData = JSON.parse(jsonString)
console.log(parsedData)
Edit
Here is what worked for me
const {request} = require("https")
const zlib = require("zlib")
const parseGzip = (gzipBuffer) => new Promise((resolve, reject) =>{
zlib.gunzip(gzipBuffer, (err, buffer) => {
if (err) {
reject(err)
return
}
resolve(buffer)
})
})
const fetchJson = (url) => new Promise((resolve, reject) => {
const r = request(url)
r.on("response", (response) => {
if (response.statusCode !== 200) {
reject(new Error(`${response.statusCode} ${response.statusMessage}`))
return
}
const responseBufferChunks = []
response.on("data", (data) => {
console.log(data.length);
responseBufferChunks.push(data)
})
response.on("end", async () => {
const responseBuffer = Buffer.concat(responseBufferChunks)
const unzippedBuffer = await parseGzip(responseBuffer)
resolve(JSON.parse(unzippedBuffer.toString()))
})
})
r.end()
})
fetchJson("https://wiki.mozilla.org/images/f/ff/Example.json.gz")
.then((result) => {
console.log(result)
})
.catch((e) => {
console.log(e)
})

Thank you, I actually just tried this approach and I get the following error:
SyntaxError: JSON Parse error: Unexpected identifier "x"
But I managed to print the data in text format using the below function:
getFinancialReports = (options, callback) => {
// buffer to store the streamed decompression
var buffer = [];
https
.get(options, function (res) {
// pipe the response into the gunzip to decompress
var gunzip = zlib.createGunzip();
res.pipe(gunzip);
gunzip
.on("data", function (data) {
// decompression chunk ready, add it to the buffer
buffer.push(data.toString());
})
.on("end", function () {
// response and decompression complete, join the buffer and return
callback(null, buffer.join(""));
})
.on("error", function (e) {
callback(e);
});
})
.on("error", function (e) {
callback(e);
});
};
Now I would need to pass this into a JSON object.

Why is my file from image "request" incomplete when saving to Google Cloud Storage?

I am trying to upload an image from a URL to my Google Cloud Storage (Firebase). The following function shall return the file and a consecutive function will retrieve the actual Signed/Download Url to the new file. After all this I am updating a document in my Firestore Database with the new URL. That part works; the functions wait on uploading the (unfortunately incomplete) image and my document gets updated with the newly created file url. But the actual file/image is incomplete. :-(
async function saveToStorage(fileUrl) {
var storage = admin.storage();
var urlLib = require("url");
var pathLib = require("path");
//Get File Name from provided URL
var parsed = urlLib.parse(fileUrl);
var fileName = pathLib.basename(parsed.pathname);
//Create Storage Reference with new File Name
var bucket = storage.bucket('gs://myprojectname.appspot.com');
//Path Folder
var folderPath = 'data/photos/';
//Path Folder + File
var internalFilePath = folderPath + fileName ;
//Bucket File Ref
var file = bucket.file(internalFilePath);
const request = require('request');
const writeStream = file.createWriteStream({
metadata: {
contentType: 'image/jpg'
}
});
return new Promise((resolve, reject) => {
request.get(fileUrl)
.pipe(writeStream)
.on("error", (err) => {
console.error(`Error occurred`);
reject();
})
.on('finish', () => {
console.info(`Photo saved`);
resolve(file);
});
});
}
The Image that is saved/uploaded/streamed to my Cloud Storage file looks like this:
I have tried using node-fetch and request and rewrote my function in several ways, but always turn out with this result. I'm sure it has something to do with how I use my Promise, because if I omit the Promise the file actually completes but then the main code keeps executing instead of waiting for this Promise.
This has the same result (incomplete file):
return await fetch(fileUrl).then(res => {
const contentType = res.headers.get('content-type');
const writeStream = file.createWriteStream({
metadata: {
contentType
}
});
let p = new Promise((resolve, reject) => {
res.body.pipe(writeStream);
writeStream.on('finish', function() {
console.log("Stream finished")
resolve(file);
});
writeStream.on('error', function() {
reject(new Error("Whoops!"));
});
});
return p.then(
function(file) {
console.log('Photo saved');
return file},
function(error) {
console.error(error);
return;
});
});
And outright returning the stream writes a complete file, but my main code is not waiting for the file (and I need to handle the file)..
return res.body.pipe(writeStream)
.on('finish', () => {
return file;
console.log('Photo')
})
.on('error', err => {
return console.error(err);
});
Thanks for any help on this!

So this is the code that finally worked for me.
return new Promise((resolve, reject) => {
const req = request(fileUrl);
req.pause();
req.on('response', res => {
const writeStream = file.createWriteStream({
metadata: {
contentType: res.headers['content-type']
}
});
req.pipe(writeStream)
.on('finish', () => {
console.log('Photo saved');
resolve(file);
})
.on('error', err => {
writeStream.end();
console.error(err);
reject();
});
req.resume();
});
req.on('error', err => console.error(err));
});

Passing a variable from ReactJS frontend to NodeJS back end using a GET route

I am working on a react app and am trying to find a way to pass a variable I define in my front-end (Question.js) to my back-end (server.js) so that I can issue different queries. I have the code
//Question.js
state = {
data: null
};
componentDidMount() {
this.callBackendAPI()
.then(res => this.setState({ data: res.express }))
.catch(err => console.log(err));
}
callBackendAPI = async () => {
const response = await fetch('/express_backend');
const body = await response.json();
if (response.status !== 200) {
throw Error(body.message)
}
return body;
};
//server.js
con.connect(function (err) {
if (err) throw err;
con.query("SELECT question FROM s1questions WHERE ID = 1", function (err, result, fields) {
if (err) throw err;
app.get('/express_backend', (req, res) => {
var x = JSON.stringify(result[0].question);
res.send({ express: `${x}` });
});
});
});

Your sever should probably split your database connection from your route handler definitions. Also, you could use query parameters to access questions based on their id in the database.
// question.js
callBackendAPI = async () => {
const response = await fetch(`/express_backend?questionId=1`);
const body = await response.json();
if (response.status !== 200) {
throw Error(body.message)
}
return body;
};
// server.js
app.get('/express_backend', (req, res) => {
const { questionId } = req.query;
// query database for question by id
});