Develop Reference

How to write file using fs.createWriteStream

am trying to build a web scraper that downloads all the pdfs in a website. i've written all the logic necessary to do this but for some reason it downloads an empty pdf file which is not suppose to be so, the problem seems to be coming from the downloadFile function when i try to pipe the data which for some reason seems not to be working because i get an empty pdf file after the function is ran. i'll would appreciate it if someone can help me out with this problem, thanks.
here's a sample of my code:
app.js
const fs = require("fs");
const path = require("path");
const cheerio = require("cheerio");
const axiosInstance = require("./getAxios");
const axios = axiosInstance();
const Surl = "https://www.health.gov.ng/";
// linkList sample: "https://www.health.gov.ng/index.php?option=com_content&view=article&id=143&Itemid=512";
let = connectionFailCount = 0;
let linkList = [];
let dlinkList = [];
const getWebsiteLinks = async (Surl) => {
try {
console.log(`Crawling all links from: ${Surl}`);
const response = await axios.get(Surl);
const $ = cheerio.load(response.data);
const ranges = $("a").each(function (idx, el) {
if ($(el).attr("href")) {
return $(el).attr("href");
}
});
for (let index = 0; index < ranges.length; index++) {
let raw_links = $("a")[index].attribs.href;
if (raw_links.startsWith("/")) {
linkList.push(Surl + raw_links);
}
}
if (linkList.length > 0) {
console.log(`Finished crawling links: Found ${linkList.length} links`);
console.log(
"--------------------------------------------------------\n\n"
);
}
return;
} catch (error) {
if (connectionFailCount === 0) {
connectionFailCount += 1;
getWebsiteLinks(Surl);
console.log(`Connection error. \n
Reconnecting to server....`);
} else if (connectionFailCount === 5) {
console.error(`Can not connect to server. Try again later.`);
}
}
};
const downloadLinks = async (linkList) => {
try {
console.log("Crawling links to find pdf links. this may take a while...");
for (const link of linkList) {
const response = await axios.get(link);
// Skip where there's delayed server response
if (response.code === "ECONNRESET") continue;
const $ = cheerio.load(response.data);
$("a").each(function (idx, el) {
if ($(el)?.attr("href")?.endsWith(".pdf")) {
let addr = $(el).attr("href");
let dlink = Surl + addr;
dlinkList.push({
pathName: addr,
url: dlink,
});
}
});
}
console.log(dlinkList);
if (dlinkList.length > 0) {
console.log(`Crawling Finish: Found ${dlinkList.length} pdf links`);
console.log(
"--------------------------------------------------------\n\n"
);
}
} catch (error) {
if (connectionFailCount === 0) {
connectionFailCount += 1;
console.log(`Connection error. \n
Reconnecting to server: ${connectionFailCount} count`);
downloadLinks(linkList);
}
if (connectionFailCount === 3) {
console.error(`Can not connect to server. Try again later.`);
return;
}
// console.error("downloadLinksError: ", error);
}
};
const downloadFiles = async (dlinkList) => {
console.log("Creating directory to save PDF files");
const appRoot = path.dirname(path.resolve(__dirname));
// Had to change and restructure code due to error
const folderName = `PDF/${Surl.split("/").pop()}`;
const subFolderName = Surl.split("/").pop();
try {
if (!fs.existsSync(path.join(appRoot, folderName))) {
fs.mkdirSync(path.join(appRoot, "PDF"));
fs.mkdirSync(path.join(`${appRoot}/PDF`, subFolderName));
}
dlinkList.forEach(async (link) => {
let name = link.pathName;
let url = link.url;
let file = fs
.createWriteStream(
`${appRoot}/${folderName}/${name.split("/").pop()}`,
"utf-8"
)
.on("error", (err) => {
console.error("createWriteStreamError: ", err);
});
try {
console.log("Downloading PDF file...");
const { data } = await axios({
url,
method: "GET",
responseType: "stream",
});
if (data) {
console.log("PDF file Downloaded");
data.pipe(file);
}
} catch (error) {
console.error(error);
}
});
return;
} catch (error) {
console.error("downloadFilesError: ", error);
}
};
(async () => {
await getWebsiteLinks(Surl);
await downloadLinks(linkList);
await downloadFiles(dlinkList);
})();
getAxios.js
const axios = require("axios");
const https = require("https");
module.exports = function () {
const domain = "https://www.health.gov.ng/";
let instance;
if (!instance) {
//create axios instance
instance = axios.create({
baseURL: domain,
timeout: 60000, // Increase time out incase of network delay or delayed server response
maxContentLength: 500 * 1000 * 1000, // Increase maximum response ata length
httpsAgent: new https.Agent({ keepAlive: true }),
headers: { "Content-Type": "application/xml" },
});
}
return instance;
};

Broadcasting to all clients with Deno websocket

I want to add notifications to an application I've developed.
Unfortunately, Deno has removed the ws package.(https://deno.land/std#0.110.0/ws/mod.ts)
That's why I'm using the websocket inside the denon itself. Since it doesn't have many functions, I have to add some things myself.
For example, sending all messages to open clients.
What I want to do is when the pdf is created, a (data, message) comes from the socket and update the notifications on the page according to the incoming data.
I keep all open clients in a Map. and when the pdf is created, I return this Map and send it to all sockets (data, message).
However, this works for one time.
server conf...
import {
path,
paths,
ctid,
} from "../deps.ts";
const users = new Map();
const sockets = new Map()
const userArr = [];
export const startNotif = (socket,req) => {
const claims = req.get("claims");
const org = req.get("org");
claims.org = org;
console.log("connected")
users.set(claims.sub, {"username":claims.sub,"socket":socket})
users.forEach((user)=>{
if(userArr.length === 0){
userArr.push(user)
}
else if(userArr.every((w)=> w.username !== user.username) )
userArr.push(user)
})
sockets.set(org, userArr)
function broadcastMessage(message) {
sockets.get(org).map((u)=>{
console.log(u.socket.readyState)
u.socket.send(message)
})
}
if (socket.readyState === 3) {
sockets.delete(uid)
return
}
const init = (msg) => {
socket.send(
JSON.stringify({
status: "creating",
})
);
};
const ondata = async (msg) => {
const upfilepath = path.join(paths.work, `CT_${msg.sid}_report.pdf`);
try {
const s=await Deno.readTextFile(upfilepath);
if(s){
socket.send(
JSON.stringify({
status: "end",
})
);
} else {
socket.send(
JSON.stringify({
status: "creating",
})
);
}
} catch(e) {
if(e instanceof Deno.errors.NotFound)
console.error('file does not exists');
}
};
const end = () => {
try {
const endTime = Date.now()
const msg = "Your PDF has been created"
const id = ctid(12) // random id create
broadcastMessage(
JSON.stringify({
id: id,
date: endTime,
status: "done",
message: msg,
read: 'negative',
action: 'pdf'
})
);
} catch (e) {
console.log(400, "Cannot send.", e);
}
}
socket.onmessage = async (e) => {
const cmd = JSON.parse(e.data);
if(cmd.bid === 'start'){
await init(cmd)
}
if(!cmd.bid && cmd.sid){
await ondata(cmd)
}
if(cmd.bid === 'end'){
await end();
}
}
socket.onerror = (e) => {
console.log(e);
};
}
client conf...
export const webSocketHandler = (request) =>
new Promise((res, rej) => {
let url;
if (!process.env.NODE_ENV || process.env.NODE_ENV === 'development') {
url = `http://localhost:8080/api/notifications/ws`.replace('http', 'ws');
} else {
url = `${window.location.origin}/api/notifications/ws`.replace('http', 'ws');
}
const token = JSON.parse(sessionStorage.getItem('token'));
const orgname = localStorage.getItem('orgname');
const protocol = `${token}_org_${orgname}`;
const socket = new WebSocket(url, protocol);
const response = Object.create({});
socket.onopen = function () {
socket.send(
JSON.stringify({
bid: 'start',
})
);
};
socket.onmessage = function (event) {
response.data = JSON.parse(event.data);
if (response.data.status === 'creating') {
socket.send(
JSON.stringify({
sid: request.sid,
})
);
} else if (response.data.status === 'end') {
socket.send(
JSON.stringify({
bid: 'end',
})
);
} else if (response.data.status === 'done') {
try {
res(response);
} catch (err) {
rej(err);
}
}
};
socket.onclose = function (event) {
response.state = event.returnValue;
};
socket.onerror = function (error) {
rej(error);
};
});
onclick function of button I use in component...
const donwloadReport = async (type) => {
const query = `?sid=${sid}&reportType=${type}`;
const fileName = `CT_${sid}_report.${type}`;
try {
type === 'pdf' && setLoading(true);
const response = await getScanReportAction(query);
const request = {
sid,
};
webSocketHandler(request)
.then((data) => {
console.log(data);
dispatch({
type: 'update',
data: {
id: data.data.id,
date: data.data.date,
message: data.data.message,
action: data.data.action,
read: data.data.read,
},
});
})
.catch((err) => {
console.log(err);
});
if (type === 'html') {
downloadText(response.data, fileName);
} else {
const blobUrl = await readStream(response.data);
setLoading(false);
downloadURL(blobUrl, fileName);
}
} catch (err) {
displayMessage(err.message);
}
};
Everything works perfectly the first time. When I press the download button for the pdf, the socket works, then a data is returned and I update the notification count with the context I applied according to this data.
Later I realized that this works in a single tab. When I open a new client in the side tab, my notification count does not increase. For this, I wanted to keep all sockets in Map and return them all and send a message to each socket separately. But in this case, when I press the download button for the second time, no data comes from the socket.
Actually, I think that I should do the socket initialization process on the client in the context. When you do this, it starts the socket 2 times in a meaningless way.
In summary, consider an application with organizations and users belonging to those organizations. If the clients of A, B, C users belonging to X organization are open at the same time and user A pressed a pdf download button, I want A, B, C users to be notified when the pdf is downloaded.
I would be very grateful if someone could show me a way around this issue.

Have you looked at the BroadcastChannel API? Maybe that could solve your issue. See for example:
Deno specific: https://medium.com/deno-the-complete-reference/broadcast-channel-in-deno-f76a0b8893f5
Web/Browser API: https://developer.mozilla.org/en-US/docs/Web/API/Broadcast_Channel_API

Uploading image to firebase using expo react-native

I am working on an app and I am using expo, I want to make sure each user can upload an image to firebase, and later publish this image on the profile page.
Using expo this is how I upload images:
const pickImage = async () => {
let pickerResult = await ImagePicker.launchImageLibraryAsync({
mediaTypes: ImagePicker.MediaTypeOptions.All,
allowsEditing: true,
aspect: [4, 3],
quality: 1,
});
console.log(pickerResult);
handleImagePicked(pickerResult);
};
the result in the console is:
Object {
"cancelled": false,
"height": 312,
"type": "image",
"uri": "file:///data/user/0/host.exp.exponent/cache/ExperienceData/%2540anonymous%252Fallergyn-app-77bfd368-65fd-43f9-8c34-9c35cef42c25/ImagePicker/daaa229c-c352-4994-ae18-ca2dbb3534ce.jpg",
"width": 416,
}
and this is how I upload to the firebase:
const handleImagePicked = async (pickerResult) => {
try {
if (!pickerResult.cancelled) {
setImage(pickerResult.uri);
await uploadImageAsync(pickerResult.uri);
console.log("done");
}
} catch (e) {
console.log(e);
alert("Upload failed, sorry :(");
} finally {
}
};
async function uploadImageAsync(uri) {
const blob = await new Promise((resolve, reject) => {
const xhr = new XMLHttpRequest();
xhr.onload = function () {
resolve(xhr.response);
};
xhr.onerror = function (e) {
console.log(e);
reject(new TypeError("Network request failed"));
};
xhr.responseType = "blob";
xhr.open("GET", uri, true);
xhr.send(null);
});
const ref = firebase
.storage()
.ref()
.child("images" + Math.random());
const snapshot = await ref.put(blob);
// We're done with the blob, close and release it
blob.close();
return await snapshot.ref.getDownloadURL();
}
this code works it saves the path of the image this: "file:///data/user/0/host.exp.exponent/cache/ExperienceData/%2540anonymous%252Fallergyn-app-77bfd368-65fd-43f9-8c34-9c35cef42c25/ImagePicker/daaa229c-c352-4994-ae18-ca2dbb3534ce.jpg" in the firebase under user collection using the uid of the user.
I am not sure if this is good, because I want to make sure the image itself is uploaded to firebase, I saw some threads in StackOverflow regarding this issue either too old or no answers, so I am hoping to get some sort of solution to what I need to do.
if I use
const ref = firebase
.storage()
.ref()
.child("images" + Math.random());
.putFile(uri);
this tells me that putFile is not a function. the same with put(uri)

Try this one. This function returns the path of the saved image from firebase which you will store in the user's document instead.
const handleImagePicked = async (pickerResult) => {
if (!pickerResult.cancelled) {
setImage(pickerResult.uri);
const result = await uploadImageAsync(pickerResult.uri);
if(result) {
console.log('success');
//save the result path to firestore user document
return;
}
alert("Upload failed, sorry :(");
}
};
export const uploadImageAsync = async (uri: string) => {
let filename = uri;
if (Platform.OS === 'ios') {
filename = uri.replace('file:', '');
}
const ext = filename.split('.').pop();
const path = `images/${id}.${ext}`;
const ref = firebase.storage().ref(path);
try {
const response = await fetch(filename);
const blob = await response.blob();
await ref.put(blob);
return path;
} catch {
return null;
}
};

This worked for me , using rn-fetch-blob
import launchImageLibrary from 'react-native-image-picker';
import RNFetchBlob from 'rn-fetch-blob';
import storage from '#react-native-firebase/storage';
const pickImage = () => {
let options = {
mediaType: 'photo',
quality: 0.5,
};
launchImageLibrary(options, (response) => {
console.log('Response = ', response);
uploadImagePicked(response);
});
};
const uploadImagePicked = (response) => {
if (response.fileName) {
const fileName = response.fileName;
var storageRef = storage().ref(`receiptImages/${fileName}`);
RNFetchBlob.fs.readFile(response.uri , 'base64')
.then(data => {
storageRef.putString(data, 'base64', {contentType:"image/jpg"})
.on(
storage.TaskEvent.STATE_CHANGED,
snapshot => {
console.log("snapshot: " + snapshot.state);
console.log("progress: " + (snapshot.bytesTransferred / snapshot.totalBytes) * 100);
if (snapshot.state === storage.TaskState.SUCCESS) {
console.log("Success");
}
},
error => {
console.log("image upload error: " + error.toString());
},
() => {
storageRef.getDownloadURL()
.then((downloadUrl) => {
console.log("File available at: " + downloadUrl);
})
})
})
.catch(error => {
console.log(error);
})
}
else {
console.log("Skipping image upload");
}
}

multi file upload with skipper-better-s3 and sailjs returns the same key

As seen in the title, I am currently using sailjs + skipper-better-s3 for s3 upload. Started with uploading one file which works great, then because change request the need of multi-file upload at once so I added a for loop but by doing this, all keys will be the same and ended up the only one file is uploaded which is the last uploaded file but with the first upload filename.
I did read some articles and people are saying something like The problem is because for loop is synchronous and file upload is asynchronous and people saying the result of this is using a recursion which I tried too but no luck though, the same thing happens.
My recursive code is below...
s3_upload_multi: async (req, res) => {
const generatePath = (rootPath, fieldName) => {
let path;
// this is just a switch statement here to check which fieldName is provided then value of path will depend on it
// as for the other two variable is just checking if upload content type is correct
return { path };
};
const processUpload = async ({
fieldName,
awsOp,
fileExtension,
rootPath,
fileName,
}) => {
return new Promise(function (resolve, reject) {
req.file(fieldName).upload(awsOp, async (err, filesUploaded) => {
if (err) reject(err);
const filesUploadedF = filesUploaded[0]; // F = first file
const response = {
status: true,
errCode: 200,
msg: 'OK',
response: {
url: filesUploadedF.extra.Location,
size: filesUploadedF.size,
type: fileExtension,
filename: filesUploadedF.filename,
key: filesUploadedF.extra.Key,
field: fieldName,
}
};
resolve(response);
});
});
}
const process_recur = async (files, fieldName) => {
if (files.length <= 0) return;
const fileUpload = files[0].stream;
const rootPath = `${sails.config.aws.upload.path.root}`;
const fileCType = fileUpload.headers['content-type'];
// console.log(fileCType, 'fileCType');
const { path } = generatePath(rootPath, fieldName);
const fileName = fileUpload.filename;
const fileExtension = fileUpload.filename.split('.').pop();
const genRan = await UtilsService.genRan(8);
const fullPath = `${path}${genRan}-${fileName}`;
const awsOp = {
adapter: require('skipper-better-s3'),
key: sails.config.aws.access_key,
secret: sails.config.aws.secret_key,
saveAs: fullPath,
bucket: sails.config.aws.bucket,
s3params: {
ACL: 'public-read'
},
};
const config = {
fieldName,
awsOp,
fileExtension,
rootPath,
fileName,
}
const procceed = await processUpload(config);
files.shift();
await process_recur(files, fieldName);
};
try {
const fieldName = req._fileparser.upstreams[0].fieldName;
const files = req.file(fieldName)._files;
await process_recur(files, fieldName);
} catch (e) {
console.log(e, 'inside UploadService');
return false;
}
}
below is the code for me using for loop which is quite similiar from above though
s3_upload_multi: async (req, res) => {
const generatePath = (rootPath, fieldName) => {
let path;
// this is just a switch statement here to check which fieldName is provided then value of path will depend on it
// as for the other two variable is just checking if upload content type is correct
return { path };
};
const processUpload = async ({
fieldName,
awsOp,
fileExtension,
rootPath,
fileName,
}) => {
return new Promise(function (resolve, reject) {
req.file(fieldName).upload(awsOp, async (err, filesUploaded) => {
if (err) reject(err);
const filesUploadedF = filesUploaded[0]; // F = first file
const response = {
status: true,
errCode: 200,
msg: 'OK',
response: {
url: filesUploadedF.extra.Location,
size: filesUploadedF.size,
type: fileExtension,
filename: filesUploadedF.filename,
key: filesUploadedF.extra.Key,
field: fieldName,
}
};
resolve(response);
});
});
}
try {
const fieldName = req._fileparser.upstreams[0].fieldName;
const files = req.file(fieldName)._files;
for (const file of files) {
const fileUpload = file.stream;
const rootPath = `${sails.config.aws.upload.path.root}`;
const fileCType = fileUpload.headers['content-type'];
// console.log(fileCType, 'fileCType');
const fileName = fileUpload.filename;
const { path } = generatePath(rootPath, fieldName);
const fileExtension = fileUpload.filename.split('.').pop();
// using a variable here because if this is an image, a thumbnail will be created with the same name as the original one
const genRan = await UtilsService.genRan(8);
const fullPath = await `${path}${genRan}-${fileName}`;
const awsOp = {
adapter: require('skipper-better-s3'),
key: sails.config.aws.access_key,
secret: sails.config.aws.secret_key,
saveAs: fullPath,
bucket: sails.config.aws.bucket,
s3params: {
ACL: 'public-read'
},
};
const config = {
fieldName,
awsOp,
fileExtension,
rootPath,
fileName,
}
const procceed = await processUpload(config);
console.log(procceed, 'procceed');
}
} catch (e) {
console.log(e, 'inside UploadService');
return false;
}
}
Which part am I making mistake that's causing such behavior? I checked my path it's totally correct with correct filename too when I console.log
Thanks in advance for any suggestions and help.

Took me quite a lot of time to figure this out ages ago.
Especially you are using skipper-better-s3 which did not conclude as much detailed documentation as skipper, going back to look into skipper documentation actually the saveAs field doesn't only take string but also a function which you can then use that to get each file's filename and return it as needed so actually you do not even need to use neither resursive or for loop at all.
for example with some of your codes
const awsOp = {
adapter: require('skipper-better-s3'),
key: sails.config.aws.access_key,
secret: sails.config.aws.secret_key,
saveAs: (__newFileStream, next) => {
// generatePath is what you wrote
// __newFileStream.filename would the filename of each each before uploading
// the path is pretty much the s3 key which includes your filename too
const { path } = generatePath(rootPath, __newFileStream.filename, fieldName);
return next(undefined, path);
},
bucket: sails.config.aws.bucket,
s3params: {
ACL: 'public-read'
},
};
skipper documentation https://www.npmjs.com/package/skipper#customizing-at-rest-filenames-for-uploads

Angular 2 Synchronous File Upload

I am trying to upload a file to web api which takes the file as byte array using angular 2 application.
I am not able to pass the byte array from angular 2 page to web api. It looks like the File Reader read method is asynchronous. How do I make this as synchronous call or wait for the file content to be loaded before executing the next line of code?
Below is my code
//attachment on browse - when the browse button is clicked
//It only assign the file to a local variable (attachment)
fileChange = (event) => {
var files = event.target.files;
if (files.length > 0) {
this.attachment = files[0];
}
}
//when the submit button is clicked
onSubmit = () => {
//Read the content of the file and store it in local variable (fileData)
let fr = new FileReader();
let data = new Blob([this.attachment]);
fr.readAsArrayBuffer(data);
fr.onloadend = () => {
this.fileData = fr.result; //Note : This always "undefined"
};
//build the attachment object which will be sent to Web API
let attachment: Attachment = {
AttachmentId: '0',
FileName: this.form.controls["attachmentName"].value,
FileData: this.fileData
}
//build the purchase order object
let order: UpdatePurchaseOrder = {
SendEmail: true,
PurchaseOrderNumber: this.form.controls["purchaseOrderNumber"].value,
Attachment: attachment
}
//call the web api and pass the purchaseorder object
this.updatePoService
.updatePurchaseOrder(this.form.controls["purchaseOrderRequestId"].value, order)
.subscribe(data => {
if (data) {
this.saveSuccess = true;
}
else {
this.saveSuccess = false;
}
},
error => this.errors = error,
() => this.res = 'Completed'
);
}
Any hint would be useful.
regards,
-Alan-

You cannot make this async call synchronous. But you can take advantage of the observables to wait for the files to be read:
//when the submit button is clicked
onSubmit = () => {
let file = Observable.create((observer) => {
let fr = new FileReader();
let data = new Blob([this.attachment]);
fr.readAsArrayBuffer(data);
fr.onloadend = () => {
observer.next(fr.result);
observer.complete()
};
fr.onerror = (err) => {
observer.error(err)
}
fr.onabort = () => {
observer.error("aborted")
}
});
file.map((fileData) => {
//build the attachment object which will be sent to Web API
let attachment: Attachment = {
AttachmentId: '0',
FileName: this.form.controls["attachmentName"].value,
FileData: fileData
}
//build the purchase order object
let order: UpdatePurchaseOrder = {
SendEmail: true,
PurchaseOrderNumber: this.form.controls["purchaseOrderNumber"].value,
Attachment: attachment
}
return order;
})
.switchMap(order => this.updatePoService.updatePurchaseOrder(this.form.controls["purchaseOrderRequestId"].value, order))
.subscribe(data => {
if (data) {
this.saveSuccess = true;
} else {
this.saveSuccess = false;
}
},
error => this.errors = error,
() => this.res = 'Completed'
);
}

I arrived here looking for a solution for a similar issue. I'm performing requests to an endpoint which can response a binary blob if anything goes well or a JSON file in event of error.
this.httpClient.post(urlService, bodyRequest,
{responseType: 'blob', headers: headers})
.pipe(map((response: Response) => response),
catchError((err: Error | HttpErrorResponse) => {
if (err instanceof HttpErrorResponse) {
// here, err.error is a BLOB containing a JSON String with the error message
} else {
return throwError(ErrorDataService.overLoadError(err, message));
}
}));
As FileReaderSync apparently doesn't work in Angular6 I took n00dl3's solution (above) to throw the error after parsing the Blob content:
return this.httpClient.post(urlService, bodyRequest,
{responseType: 'blob', headers: headers})
.pipe(map((response: Response) => response),
catchError((err: Error | HttpErrorResponse) => {
const message = `In TtsService.getTts(${locale},${outputFormat}). ${err.message}`;
if (err instanceof HttpErrorResponse) {
const $errBlobReader: Observable<HttpErrorResponse> = Observable.create((observer) => {
const fr = new FileReader();
const errorBlob = err.error;
fr.readAsText(errorBlob, 'utf8');
fr.onloadend = () => {
const errMsg = JSON.parse(fr.result).message;
const msg = `In TtsService.getTts(${locale},${outputFormat}). ${errMsg}`;
observer.error(ErrorDataService.overLoadError(err, msg));
};
fr.onerror = (blobReadError) => {
observer.error(blobReadError);
};
fr.onabort = () => {
observer.error('aborted');
};
});
return $errBlobReader;
} else {
return throwError(ErrorDataService.overLoadError(err, message));
}
}));
Thanks! You really saved my day!