I have a big object that I need to send from server (nodejs) to client.
But every time I try send I get "invalid string length" error.
And it's ok, because the object is really big. That's why I'd like to save it to file and then send the file to client.
I don't know the depth of the object. The object itself is and octree.
I don't have any code of saving an object to file, because every time I think about this it leads me to stringify the object, that latter leads to "invalid string length".
Here is a screenshot of the object. Every q(n) key has the same recursive structure as result key.
Thanks!
Firstly try to save in a JSON file and send JSON file directly to client-side
const fs = require('fs');
fs.writeFileSync("file_name.json", data);
res.header("Content-Type",'application/json');
res.sendFile(path.join(__dirname, 'file_name.json'));
A good solution to handling large data transfer between server and client is to stream the data.
Pipe the result to the client like so.
const fs = require('fs');
const fileStream = fs.createReadStream("path_to_your_file.json");
res.writeHead(206, {'Content-Type': 'application/json'})
fileStream.pipe(response)
or follow this blog which stringifies each element at a time and concatenates them.
I will suggest you stream it though.
You could try something like this, I'm unsure if Express would throw the same length error by using res.json.
// Assuming you're using Express and data is valid JSON.
res.json(data);
Related
I'm looking for the best way to send image files to my server using Apollo Express, and Node.
Getting the information there doesn't seem to be an issue, I convert the object into a string but can't find out how to convert it back to a regular file object to store away.
What I have so far;
JS - let buffer = await toBase64(file);
Through Apollo server..
Node - let buffer = Buffer.from(args.image, 'base64');
This gives me a Buffer. I'm unsure how to proceed with NodeJS to convert this back to a file object.
Thanks
I hope this will be helpfull for you
const file = new File([
new Blob(["decoded_base64_String"])
], "output_file_name");
You can use one of the various write or writeFile methods which accept a Buffer.
const fs = require("fs");
let buffer = Buffer.from(
"iVBORw0KGgoAAAANSUhEUgAAAAgAAAAGCAIAAABxZ0isAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAAQSURBVBhXY/iPAwygxP//AAjcj3EdtT3BAAAAAElFTkSuQmCC",
"base64"
);
fs.writeFile("pic.png", buffer, (err) => {
if (err) throw err;
console.log("The file has been saved!");
});
This script is run in server using NodeJS,
I want to use pdf-table-extractor with remote file input directly from axios, is that can be done ?
here is what i have tried
const axios = require('axios')
const pdf_table_extractor = require("pdf-table-extractor")
const getPDF = await axios.get(`domain/a.pdf`,{responseType: 'arraybuffer'})
pdf_table_extractor(new Uint8Array(getPDF.data))
the error say The argument 'path' must be a string or Uint8Array without null bytes. Received Uint8Array(118456)
pdf-table-extractor is expecting a filepath and you are passing a typed array. It can't work this way.
There are many options, one of them is to save the data from getPDF.data to disk using writeFile, and then provide the path of the saved file to pdf_table_extractor.
I need to get number of items of specific directory in Node.js
If I get the items like
var dirItems = fs.readdirSync(__dirname+'/my_dir');
and then get specific item like
dirItems[1]
everything is ok
But if I try to get their number like
dirItems.length
or
Object.keys(dirItems).length
the page doesn't work in the browser
How to get the number of directory items?
UPDATED
My full code:
var http = require('http'),
fs = require('fs');
http.createServer(function(req, res) {
var dirItems = fs.readdirSync(__dirname+'/my_dir');
res.writeHead(200, {'Content-Type': 'text/html'});
res.end(dirItems.length);
}).listen(80, 'localhost');
I was able to reproduce the error nyou get.
res.end() for the basic http server class is very picky about what you send it. You must give it a string (the error you got should have been a big clue here).
So, change this:
res.end(dirItems.length);
to this:
res.end(dirItems.length.toString());
And, it works for me. I was able to reproduce your original error and then make it work by making this simple change.
Logically, you can only send string data as an http response so apparently res.end() isn't smart enough to attempt a string conversion on its own. You have to do it yourself.
FYI, if you use a higher level framework like Express, it is more tolerant of what you send it (it will attempt a string conversion in a situation like this).
Here is how I would do it:
const fs = require('fs');
const dir = './somedir';
fs.readdir(dir, (err, files) => {
console.log(files.length);
});
I've distilled my issue to some really basic functionality here. Basically, we're sending a request to a server (you can go ahead and c/p the URL and see the json document we get in response).
We get the response, we pipe it into a write stream and save it as a .json file - but the problem is that the file keeps being cut off. Is the .json file too large? Or am I missing something? Node.js newbie - massively appreciate any help I can get.
var fs = require('fs');
var url = 'https://crest-tq.eveonline.com/market/10000002/history/?type=https://crest-tq.eveonline.com/inventory/types/34/'
var request = require('request');
request(url).pipe(fs.createWriteStream('34_sell.json'));
var fs = require('fs');
var request = require('request');
var url = 'https://crest-tq.eveonline.com/market/10000002/history/?type=https://crest-tq.eveonline.com/inventory/types/34/';
request(url).pipe(fs.createWriteStream('34_sell.json'));
Not an answer but this is the code I used. I'm using request version 2.74.0. And node version v5.4.1.
Try writing a get request to the url and send the json as response and write an error handling statement like if err then throw err..console log and see the result..hope it works
I would like to be able to post user-generated JSON objects to a NodeJS server, and store them in an array so that refreshing the page does not affect the data. I would then like to be able to read the info back into a table, so that it constantly reflects the contents of the array.
Is this possible in NodeJS? If so, could someone please give me a pointer in the right direction? I'm able to pass JSON objects (as req.body), but I don't know where to go from here.
Thanks!
As Sri suggested, you should definitively store your data array into a data base.
Use either Redis, Mongo, MySQL... It entierly depends on your data shape and the ways you will use it.
If you just neet to store temporary an array (by temporary, I mean "as long as the browsing session" or "as long as the server is up"), you can just store you data in memory.
For exemple with ExpressJs:
var express = require('express');
// in memory storage
var data = [];
var app = express()
.use(express.bodyParser())
.use(express.cookieParser())
.use(express.cookieSession({secret:'sample'}))
.post('/data', function(req, res){
// do not need session, just memmory ? use this line
// data.push(req.body);
// creates the session if needed, and store data
if (!req.session.data) {
req.session.data = [];
}
req.session.data.push(req.body);
res.send(204);
})
.get('/data', function(req, res){
res.setHeader('Content-Type', 'application/json');
// stored in session ?
res.send(req.session.data || []);
// stored in memory ?
// res.send(data);
})
.listen(80);
----- EDIT -----
To add new data and return it in a single request, just send your array in the POST response:
.post('/data', function(req, res){
// store in memory
data.push(req.body);
// send it back, updated, in json
res.setHeader('Content-Type', 'application/json');
res.send(data);
})
There is an assumption: you should call your post with an ajax call, and not a regular HTML form. Please refer to jquery.Ajax doc to get the parsed server result.