Download CSV from URL in Node - javascript

I created a web scraper using Puppeteer to extract data from https://www.jobnimbus.com/ and would like to download a CSV file generated by JobNimbus. When I send a GET request to the CSV download URL, I receive .aspx file type instead of .csv.
JobNimbus requires a login so I'm only showing a sample of the code I'm using:
require("dotenv").config();
const download = require("download");
(async () => {
let url =
`https://app.jobnimbus.com/ReportDownload.aspx?type=csv&id=${process.env.USER_ID}&rid=null`;
await download(url, "output");
})();
It downloads ReportDownload.aspx which appears to be HTML. How do I download this file as CSV? Thank you in advance for your help!
https://i.imgur.com/lX7eIdA.png

Related

Trying to get a stream link of a video file from GDRIVE API

I'm trying to get a stream link for my video files to stream it on my web app from google drive api, but its not working properly. I have double checked the docs for any errors in syntax and i can't seem to find any.
for context here is my code:
`
drive.files.get({fileId: myfileId,alt: 'media'},{responseType: 'stream'}, (err, res) => {
if (err) return console.log(`The API returned an error: ${err}`);
console.log(res)
});
`
I'm getting a passthrough object in res.data field and its giving an error of "Unknown output format: 'media' ". The file i'm trying to stream is a .mp4 file.
I have also double checked my authentication and its working fine because i was able to retrieve my folder id and file id using the api.
Am i doing anything wrong here? Any help would be appreciated.
THANKS.
Once you have authenticated the client library, you can use the following code to get a stream link for a video file stored in Google Drive
// Replace fileId with the ID of the video file you want to stream
const fileId = '1234567890';
// Get the file from Google Drive
const file = await drive.files.get({ fileId, alt: 'media' });
// Get the stream link for the file
const streamLink = file.data;

Download File in Specified folder [Javascript]

I've been creating a small project with the aim of downloading youtube videos on my server. in theory it already works, but I want that, when I press download, it downloads the file in a specific folder inside the server.
this is my code for the function.
Used library: ytdl-core
app.get("/download", async(req, res) => {
const info = await req.query.url;
const v_id = info.split('v=')[1];
res.header("Content-Disposition", `attachment; filename="${v_id}.mp3`);
ytdl(info, {format: 'mp3'}).pipe(res);
});

how to download file using api angular

I have an API that downloads a file, I have a button on the button I have a click that sends a request to the API for download a file, but it doesn't work request sending successfully but the file is not downloaded, but when I'm adding the URL into the browser the file is successfully downloaded
HTML
<button (click)="exportFile()">Download</button>
TS
exportFile(): void{
this.companiesService.export().subscribe((res) => {
console.log(res);
});
}
Service
export(){
const headers = this.httpOptions.headers.set('Authorization', `Bearer ${this.cookieService.get('access-token')}`);
return this.http.get(`${this.API_URL}/company/export/`,{headers});
}
You need to process the returned blob and save it as a file. Just returning it is not enough. Perhaps this demo can give you more insight how to improve your service. https://stackblitz.com/edit/angular-file-download-progress-qsqsnf?file=src%2Fapp%2Fdownload.ts

Reading Excel file in client side JavaScript

I am working on some react web app in which I am trying to read excel file on client side as below.
import XLSX from "xlsx";
const targetCsvPath = window.location.origin + "/XRayConfig.xlsx";
const workbook = XLSX.readFile(targetCsvPath)
const json = XLSX.utils.sheet_to_json(workbook.Sheets.FOV);
But this gives error TypeError: _fs.readFileSync is not a function. When I run this code snippet using node, it runs flawlessly. I think client side JavaScript does not run on Node, so is the error.
window.location.origin points to public folder of react app and the excel file is in that folder.
This link almost answers this question, but excel file is uploaded from client side using input tag and then it is processed. But My excel file is on server side. How can I solve this?
I am answering my own question. Using file system APIs does not work on client side JavaScript as it does not run on Node. So first the excel content should be fetched in the form of blob and use that blob.
Following solution works for me.
import XLSX from "xlsx";
const targetCsvPath = window.location.origin + "/XRayConfig.xlsx";
const reader = new FileReader();
reader.onload = function (e) {
const workbook = XLSX.read(e.target.result, { type: "binary" });
// your operations on workbook comes here
}
fetch(targetCsvPath)
.then((response) => response.blob())
.then((data) => {
reader.readAsBinaryString(data);
})
.catch((err) => console.log(err);

AWS S3 File Download from the client-side

I am currently trying to download the file from the s3 bucket using a button from the front-end. How is it possible to do this? I don't have any idea on how to start this thing. I have tried researching and researching, but no luck -- all I have searched are about UPLOADING files to the s3 bucket but not DOWNLOADING files. Thanks in advance.
NOTE: I am applying it to ReactJS (Frontend) and NodeJS (Backend) and also, the file is uploaded using Webmerge
UPDATE: I am trying to generate a download link with this (Tried node even if I'm not a backend dev) (lol)
see images below
what I have tried so far
onClick function
If the file you are trying to download is not public then you have to create a signed url to get that file.
The solution is here Javascript to download a file from amazon s3 bucket?
for getting non public files, which revolves around creating a lambda function that will generate a signed url for you then use that url to download the file on button click
BUT if the file you are trying to download you is public then you don't need a signed url, you just need to know the path to the file, the urls are structured like: https://s3.amazonaws.com/ [file path]/[filename]
They is also aws amplify its created and maintain by AWS team.
Just follow Get started and downloading the file from your react app is simply as:
Storage.get('hello.png', {expires: 60})
.then(result => console.log(result))
.catch(err => console.log(err));
Here is my solution:
let downloadImage = url => {
let urlArray = url.split("/")
let bucket = urlArray[3]
let key = `${urlArray[4]}/${urlArray[5]}`
let s3 = new AWS.S3({ params: { Bucket: bucket }})
let params = {Bucket: bucket, Key: key}
s3.getObject(params, (err, data) => {
let blob=new Blob([data.Body], {type: data.ContentType});
let link=document.createElement('a');
link.href=window.URL.createObjectURL(blob);
link.download=url;
link.click();
})
}
The url in the argument refers to the url of the S3 file.
Just put this in the onClick method of your button. You will also need the AWS SDK

Categories

Resources