I have to choose one of these languages:
Python (with Selenium or any suggestion)
Javascript (with node with any module)
Bash (with curl for example)
To do the following:
Make a request to an API (Scrapy cloud) and get some value, in my case I just need the id of the response:
{"count": 1, "id": "195457/7/19", "width": 32, "height": 27, "started_time": "2017-06-22T08:20:26", "total": 1, "status": "ok"}
And then make another request with the id to download that provides a download to a file with a CSV/JSON format.
What I tried:
Python:
With Selenium (Firefox driver) open and get the id, it works fine but when I try to download the file with the next API request it asks me for what I want to do with the file (download or open with...). So, as I have to interact with the dialog it is not viable.
Javascript:
I found a module to download files but it is just to download files as images from image web URLs and not for download a file (like the Linux wget command).
Bash:
With curl it works but I can just get the whole response and then I cannot get the id value so I cant continue with what I want. Also I tried to download de file of the second step and this works fine with a simple curl -o myfile.csv URL
Any help would be appreciated. Thanks for reading!
Here is a node version. Its quite broad but the 2 main functions are the callApi and downloadFile.
I dont know the structure of your API url so for me now I have mocked some simple ones - change to what you need.
You will need to npm install request and update the variables to match your API.
index.js
const request = require('request');
const http = require('http');
//const https = require('https'); maybe required
const fs = require('fs');
const apiEndPoint = 'http://scrapycloud?someparam=';
const fileName = 'data.csv';
const assetEndPoint = 'http://assetUrl?id=';
// This will call your api and get the asset id then calls the downloadFile function.
function callApi(assetId, callback) {
request(apiEndPoint + assetId, function (error, response, body) {
if (error) {
return callback(error);
}
const info = JSON.parse(body);
const assetId = info.id;
downloadFile(assetId, callback);
});
}
// This function creates a writeSteam to save a file to your local machine, performs a http request to the assets and pipes it back into the write stream
function downloadFile(assetId, callback) {
var file = fs.createWriteStream(fileName);
//use the following line if your requests needs to be https
//var request = https.get(assetEndPoint + assetId, function (response) {
var request = http.get(assetEndPoint + assetId, function (response) {
response.pipe(file);
file.on('finish', function () {
file.close(callback);
});
}).on('error', function (err) {
fs.unlink(dest);
if (callback) callback(err.message);
});
}
// Called when everything is finished or an error
function complete(err) {
if (err) {
return console.log(err);
}
console.log('file downloaded');
}
// Starts the process, pass it an id and a callback
callApi('123131', complete);
Related
How to save and write data acquired by request or fetch as executable?
For example I use
require('request').get('https://github.com/harujisaku/mini-player/raw/master/mini-player.exe', async (e, r, b) => {
require('fs').writeFileSync(path+'test.exe', b);
});
But instead of working .exe file I get a broken file. How to save and write file from server data (from github as example) correctly, so .exe file would not break?
This might help you, credits to 'Michelle Tilley' from : https://stackoverflow.com/a/11944984/5203821
const http = require('http');
const fs = require('fs');
const file = fs.createWriteStream("mini-player.exe");
const request = http.get("https://github.com/harujisaku/mini-player/raw/master/mini-player.exe", function(response) {
response.pipe(file);
});
I'm trying to execute this function but in the terminal with Node.JS
var WebTorrent = require('webtorrent')
var client = new WebTorrent()
var magnetURI = 'magnet: ...'
client.add(magnetURI, { path: '/path/to/folder' }, function (torrent) {
torrent.on('done', function () {
console.log('torrent download finished')
})
})
I mean, for example, create an <button> tag, and when is clicked,
that the previous function be executed in the nodejs console, not in the browser console.
EXTRA:
I'm executing this two files:
app.js
let http = require('http');
let fs = require('fs');
let handleRequest = (request, response) => {
response.writeHead(200, {
'Content-Type': 'text/html'
});
fs.readFile('./index.html', null, function (error, data) {
if (error) {
response.writeHead(404);
respone.write('Whoops! File not found!');
} else {
response.write(data);
}
response.end();
});
};
http.createServer(handleRequest).listen(8000);
And
index.html that contains the <button> tag but does nothing.
Client(browser) and server are two different entities, when client is browser the only way to communicate is through HTTP protocol, in simple terms use internet.
Now browser understand only it's own kind of javascript, more precisely ECMA but not nodejs. So the following code could not be executed in browser
var WebTorrent = require('webtorrent')
var client = new WebTorrent()
Hence I would assume it is running on server which your machine and hence console.log will print to your terminal.
To run it on browser, I assume you will have to code it differently, either you will have to use browserify and analyze the client side script OR code only in client side with below libaray :
<script src="webtorrent.min.js"></script>
For more details refer, complete web page example at https://github.com/webtorrent/webtorrent/blob/master/docs/get-started.md
I would like my bot to save on my computer every image, video or even a gif that's sent in a specific channel. Is that possible to do?
I know that this kind of stuff can be made by fs directory but I'm not sure how would that code look like could you guys help me?
To get all the images, gifs and videos from a message you can use the .attachments property of a message. This will give you all the files from that message (if it contains files).
With this property you can create a loop where on every message send, you itterate through the message.attachments collection. Then, as stated by the documentation, you can call the .url property on every attachment to get the link from which to download the file.
Once you have the link, you can follow this answer from a different question to download the file. Here's the code copied from the answer:
var http = require('http');
var fs = require('fs');
var download = function(url, dest, cb) {
var file = fs.createWriteStream(dest);
var request = http.get(url, function(response) {
response.pipe(file);
file.on('finish', function() {
file.close(cb); // close() is async, call cb after close completes.
});
}).on('error', function(err) { // Handle errors
fs.unlink(dest); // Delete the file async. (But we don't check the result)
if (cb) cb(err.message);
});
};
This means you will have to use the Node.js modules fs and http. To use the code, the download method can be called like
download(<The file url to download>, <The file name you want to save it as>, <a callback function which can be called when an error occurs>);
This code must help you,
import shutil
from discord.ext import commands
import requests
TOKEN = ""
prefix = "?"
bot = commands.Bot(command_prefix=prefix)
bot.run(TOKEN)
#bot.event
async def on_message(message):
print("The message's content was", message.content)
url = message.attachments[0]['url']
if url[0:26] == "https://cnd.discordapp.com":
r = requests.get(url, stream=True)
with open(String.join(uuid.uuid4(),".png"), 'wb') as out_file:
shutil.copyfileobj(r.raw, out_file)
use this and create a bot and install package requests, discord.py to run the bot.
Finally, add this bot to your channel and give it a bot role.
I want to download page (https://www.csfd.cz/tvurce/65871) in NodeJS, but I get just random data.
�}Ms�F������+i"��)�Jْ;�e���7�KM0��LƩ��]��Yg��b��
Ow7U��J�#�K�9��L
I thought it is just wrong encoding, but even size is wrong (downloaded page have 44K, whereas this file have only 19K. What's more surprising is that simple downloading it by python works good.
Python code:
import requests
url = "https://www.csfd.cz/tvurce/65871"
r = requests.get(url)
with open('pyth.txt','wb') as handle:
handle.write(r.content)
JavaScript code:
const request = require('request-promise')
const fs = require('fs')
request('https://www.csfd.cz/tvurce/65871').then((html) => {
fs.writeFileSync('output.html', html)
})
I tried also additional methods like request.get with parameters and so on, but still the same result. Can you please tell me what I am doing wrong?
Use compressed option in request module, see example with request module (https://github.com/request/request).
You need also followRedirect and followAllRedirect parameters to automatically follow 301 and 302 redirection cuz your request is returning 302 :
curl -X GET https://www.csfd.cz/tvurce/65871 --compressed -v -i
Response : 302
<h1>Redirect</h1>
<p><a href="https://www.csfd.cz/tvurce/65871-kit-harington/">Please
click here to continue</a>.</p>
In addition replace your writeFileSync with standard writeFile function
const request = require('request')
const fs = require('fs')
request.get({
url:'https://www.csfd.cz/tvurce/65871',
gzip: true,
followRedirect: true,
followAllRedirect: true
}, function(err, response, body){
if(err || !response || response.statusCode != 200)
{
// error case, do stg
}
else
{
fs.writeFile('output.html', body, "utf8", function(err){
if(err)
{
// error do stg
}
else
{
// success
}
});
}
})
I tried different things, different options and encodings, some parsers, and I didn't get it to work with request and request-promise. From the docs, I would say you aren't doing anything wrong.
I tried then a different module, unirest (npm install unirest --save), and it worked out of the box.
const unirest = require('unirest');
const fs = require('fs');
var Request = unirest.get('https://www.csfd.cz/tvurce/65871')
.end(function(res) {
console.log(res.body);
fs.writeFileSync('output.html', res.body)
});
Hope this is of help.
Read the Content-Encoding header. It's most likely compressed, which would explain the size difference.
I have a SailsJS Backend where i generate a zip File, which was requested by my Frontend, a React App with Redux. I'm using Sagas for the Async Calls and fetch for the request. In the backend, it tried stuff like:
//zipFilename is the absolute path
res.attachment(zipFilename).send();
or
res.sendfile(zipFilename).send();
or
res.download(zipFilename)send();
or pipe the stream with:
const filestream = fs.createReadStream(zipFilename);
filestream.pipe(res);
on my Frontend i try to parse it with:
parseJSON(response) => {
return response.clone().json().catch(() => response.text());
}
everything i tried ends up with an empty zip file. Any suggestions?
There are various issues with the options that you tried out:
res.attachment will just set the Content-Type and Content-Disposition headers, but it will not actually send anything.
You can use this to set the headers properly, but you need to pipe the ZIP file into the response as well.
res.sendfile: You should not call .send() after this. From the official docs' examples:
app.get('/file/:name', function (req, res, next) {
var options = { ... };
res.sendFile(req.params.name, options, function (err) {
if (err) {
next(err);
} else {
console.log('Sent:', fileName);
}
});
});
If the ZIP is properly built, this should work fine and set the proper Content-Type header as long as the file has the proper extension.
res.download: Same thing, you should not call .send() after this. From the official docs' examples:
res.download('/report-12345.pdf', 'report.pdf', function(err) { ... });
res.download will use res.sendfile to send the file as an attachment, thus setting both Content-Type and Content-Disposition headers.
However, you mention that the ZIP file is being sent but it is empty, so you should probably check if you are creating the ZIP file properly. As long as they are built properly and the extension is .zip, res.download should work fine.
If you are building them on the fly, check this out:
This middleware will create a ZIP file with multiples files on the fly and send it as an attachment. It uses lazystream and archiver
const lazystream = require('lazystream');
const archiver = require('archiver');
function middleware(req, res) {
// Set the response's headers:
// You can also use res.attachment(...) here.
res.writeHead(200, {
'Content-Type': 'application/zip',
'Content-Disposition': 'attachment; filename=DOWNLOAD_NAME.zip',
});
// Files to add in the ZIP:
const filesToZip = [
'assets/file1',
'assets/file2',
];
// Create a new ZIP file:
const zip = archiver('zip');
// Set up some callbacks:
zip.on('error', errorHandler);
zip.on('finish', function() {
res.end(); // Send the response once ZIP is finished.
});
// Pipe the ZIP output to res:
zip.pipe(res);
// Add files to ZIP:
filesToZip.map((filename) => {
zip.append(new lazystream.Readable(() => fs
.createReadStream(filename), {
name: filename,
});
});
// Finalize the ZIP. Compression will start and output will
// be piped to res. Once ZIP is finished, res.end() will be
// called.
zip.finalize();
}
You can build around this to cache the built ZIPs instead of building them on the fly every time, which is time and resource consuming and totally unadvisable for most uses cases.