I want to download file using absolute FTP URL, like ftp://host:port/dir/file.extension
I've tried node-libcurl, wget, wget-improved, request. All failed saying that the protocol must be either HTTP or HTTPS.
There are FTP clients available for Node (available on npmjs). But, as per their documentation, they require creating a connection to FTP Server, change directory and then download it.
Is there any simple solution?
I will outline a simple approach here (and no complete solution with code!). FTP is based upon TCP with a simple human readable protocol. In order to fetch a file from an FTP server you need to do the following:
Create a TCP socket using net.Socket
Use socket.connect to connect to your FTP server on port 21
Communicate with the server using socket.write to send data and socket.on('data') to read data
An example of FTPs protocol for a simple file retrieval is provided in this blog post and can be summarized as follows:
Connect to server using net.Socket.connect
Set user with USER command
Authenticate with PASS
Go to desired directory using CWD
Change to passive mode using PASV
Read server reply to find out IP and port to connect to in order to fetch the file
Open another socket on IP and port of previous step
Voilà!
Anyone wanting to have simple single line solution that in addition to FTP would also work with FTPS and SFTP, you can try ftp-any-get
import { getFile } from "#tpisto/ftp-any-get"
async function main() {
// Fetch from FTP server
let ftpFile = await getFile("ftp://demo:password#my-ftp-server.net/my-file.txt");
// Fetch from FTP server using TLS
let ftpsFile = await getFile("ftps://demo:password#my-ftp-server.net/my-file.txt");
// Fetch file using SFTP. SFTP runs over the SSH protocol.
let sftpFile = await getFile("sftp://demo:password#my-ftp-server.net/my-file.txt");
}
main();
You can use node-libcurl, I don't know exactly how you did it, but here is some working code.
var Curl = require( 'node-libcurl' ).Curl,
Easy = require( 'node-libcurl' ).Easy,
path = require( 'path' ),
fs = require( 'fs' );
var handle = new Easy(),
url = 'ftp://speedtest.tele2.net/1MB.zip',
// Download file to the path given as first argument
// or to a file named 1MB.zip on current dir
fileOutPath = process.argv[2] || path.join( process.cwd(), '1MB.zip' ),
fileOut = fs.openSync( fileOutPath, 'w+' );
handle.setOpt( Curl.option.URL, url );
handle.setOpt( Curl.option.WRITEFUNCTION, function( buff, nmemb, size ) {
var written = 0;
if ( fileOut ) {
written = fs.writeSync( fileOut, buff, 0, nmemb * size );
}
return written;
});
handle.perform();
fs.closeSync( fileOut );
The repository currently has one example showing how to download a file using wildcard matching, I just changed the URL to point directly at the file, and removed the WILDCARDMATCH and CHUNK_*_FUNCTION options.
Related
I've started to build a typescript library (intended to be used on the server side) and right now I'm trying to use the node repl to play around with my code and see what happens in certain situations... I've built and required the file, but now I'm having a problem: I have a function that takes a http Request (type Request from express.js), and I'd like to try and run it in the repl providing it with a copy of a request that I previously made from my browser. Is this feasible?
I thought maybe I could do it by either:
doing regex magic on the request exported as cURL or
sending the request to node, but then how am I going to receive it while in the repl?
I'm not sure I understand your use-case, but you can try something like this:
In some temp folder type:
npm install "request-promise"
Then from the same temp folder, enter the REPL and type:
(async () => {const response = await require("request-promise").get("https://cnn.com"); console.log(response)})()
This example is for get, but it can be easily changed to other HTTP methods.
I've found a fairly simple way to do what I want... It involved quickly setting up a basic express server (set up following this tutorial):
mkdir scratch && cd scratch && npm init
(select defaults except entrypoint app.js)
npm i express
Create an app.js (vi app.js) with the following contents:
var express = require('express');
var app = express();
var circ = {};
circ.circ = circ;
var cache = [];
app.get('/', function (req, res) {
res.send(JSON.stringify(req, (key, value) => {
if (typeof value === 'object' && value !== null) {
// Duplicate reference found, discard key
if (cache.includes(value)) return;
// Store value in our collection
cache.push(value);
}
return value;
}));
});
app.listen(3000, function () {
console.log('Example app listening on port 3000!');
});
(See this answer for JSON.stringify custom replacer resp. second argument to JSON.stringify). You can optionally use flatted instead, which I discovered later and is surely better.
Now do the following:
Run the app with node app.js
In your browser, navigate to the website where your desired request is posted to.
Open your browsers development tools (Ctrl+shift+c works for me).
Go to the network tab.
Find the request that interests you and right click on it.
Click on copy > copy as curl (or similar, depending on which browser you're using).
Run that curl request, but change the url it's posted to to 127.0.0.1:3000 (e.g. change curl 'example.com' \...etc to curl '127.0.0.1:3000' \...etc
You should now get that request on standard output as a JSON object and it's in the format that express usually deals with. Yay! Now, pipe it into your clipboard (likely xclip -selection c on linux) or probably even better, redirect it to a file.
...
Step 2 - ?
Step 3 - Profit :)
I was trying to find an online tool to generate a new link which randomly redirects to user defined array of links.
Please check out http://mathstatic.co.nz/auto to see what I am trying to do. Problem with that site is that it is wordpress and you can only enter 2 links. I want to do something similar but with user defined number of links.
Please take a look at the code I have for what I put together today: http://rpantaev.com/RedirectLinkTool/
After researching, it seems that I need to use PHP and perhaps a database like mysql to store users' data, but I am not familiar with PHP so any pointers will be greatly appreciated.
So far php is installed on my server and I added .htaccess so html can read php. I put together a webpage with html/css/js that takes in user defined number of links, checks for http://, stores in array and redirects to randomly chosen url (array is not persistent and no new url is generated).
How can I recreate what the first link does and incorporate that into my website?
Thank you!
So the language you use will depend on what languages your server supports or whether you can install your own. If your looking to run a server on your computer to just test this out, you just need to install the language you want to use then find a way to run as server on your computer.
You can use any database (postgres, mongodb), to store your urls. If it's ok to use memory for this project then you can use a dictionary to store the random urls.
Here's a way to do this using Node.js. In this solution, I create a server on port 9898 and create an in memory database using a dictionary. When there are entries, the database will look like this:
{
"uuid1": { id: "uuid1", urls: ["https://www.google.com", "https://youtube.com"] },
"uuid2": { id: "uuid2", urls: ["https://www.google.com", "https://youtube.com"] }
}
uuid1 and uuid2 would be replaced with actual unique IDs.
In this solution, I'm expecting that the client will send a json object.
The solution depends on expressjs and body-parser packages. Here's actual code that I test on my computer.
const express = require('express')
const bodyParser = require('body-parser');
const app = express()
app.use(bodyParser.json()); // for parsing application/json
app.use(bodyParser.urlencoded({ extended: true })); // for parsing application/x-www-form-urlencoded
const port = 9898
const host = `http://localhost:${port}`
// you would replace this with an actual database
const db = {}
// to generate a uuid
// from: https://stackoverflow.com/a/2117523
function uuidv4() {
return 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx'.replace(/[xy]/g, function(c) {
const r = Math.random() * 16 | 0, v = c == 'x' ? r : (r & 0x3 | 0x8);
return v.toString(16);
});
}
function success(message) {
return JSON.stringify({
success: message
})
}
function error(message) {
return JSON.stringify({
error: message
})
}
// assuming body looks like this { urls: ['www.google.com', 'www.gmail.com'] }
app.post('/gen', (req, res) => {
const gen = req.body
// check if the json in the request has the list of urls
if(!(gen && gen.urls && gen.urls.length > 0)) {
// if not then log the body and return a json object with the error object
console.log('body', req.body)
return res.send(error('I cannot find the list of urls in json body.'))
}
// generate a uuid
const id = uuidv4()
// save the urls with the new id
db[id] = {
id: id,
urls: gen.urls
}
// respond with the url that the user should use
res.send(success(host + '/' + id))
})
app.get('/:id', (req, res) => {
// get the id paramater from the request
const id = req.params.id
// check if that id was saved, if not send an error message
if(!db[id])
return res.send(error('that url does not exist'))
// grab a random element from the list of urls
const urls = db[id].urls
const url = urls[Math.floor(Math.random() * urls.length)]
// redirect the user to the random url
res.redirect(url)
})
// start the server
app.listen(port, () => {
console.log('listening on port: '+port);
})
/*
example request:
curl -X POST -H "Content-Type: application/json" -d '{"urls": ["https://www.google.com", "https://www.yahoo.com", "https://www.youtube.com"]}' localhost:9898/gen
after doing this you grab the url generated and paste into your browser.
for example when I ran this it generated: http://localhost:9898/aa582177-4ab5-4ede-99c5-a06b43976c12
*/
If you've never setup a nodejs project before this page can get you up to speed: https://docs.npmjs.com/getting-started/installing-node
After install nodejs and npm. create a folder for the project. In your command line, go to this directory. To setup the project and install the two packages, write:
npm init -y
npm install express body-parser
After doing that create a file named index.js in that folder. Then paste the code in. Then to run the server use this command:
node index.js
If your on MacOS or Linux, or have a way to use curl on Windows you can use this example request:
curl -X POST -H "Content-Type: application/json" -d '{"urls": ["https://www.google.com", "https://www.yahoo.com", "https://www.youtube.com"]}' localhost:9898/gen
Then that should return a url for you to use. For example for me it generated this:
http://localhost:9898/aa582177-4ab5-4ede-99c5-a06b43976c12
You would paste the url generated into your browser and it should reroute you to one of the urls in the list.
I want to send asynchronous data to the node on configuration. I want to
perform a SQL request to list some data in a .
On node creation, a server side function is performed
When it's done, a callback send data to the node configuration
On node configuration, when data is received, the list is created
Alternatively, the binary can request database each x minutes and create a
cache that each node will use on creation, this will remove the asynchronous
part of code, even if it's no longer "live updated".
In fact, i'm stuck because i created the query and added it as below :
module.exports = function(RED) {
"use strict";
var db = require("../bin/database")(RED);
function testNode(n) {
// Create a RED node
RED.nodes.createNode(this,n);
// Store local copies of the node configuration (as defined in the
.html
var node = this;
var context = this.context();
this.on('input', function (msg) {
node.send({payload: true});
});
}
RED.nodes.registerType("SQLTEST",testNode);
}
But I don't know how to pass data to the configuration node. I thought of
Socket.IO to do it, but, is this a good idea and is it available? Do you know any solution ?
The standard model used in Node-RED is for the node to register its own admin http endpoint that can be used to query the information it needs. You can see this in action with the Serial node.
The Serial node edit dialog lists the currently connected serial devices for you to pick from.
The node registers the admin endpoint here: https://github.com/node-red/node-red-nodes/blob/83ea35d0ddd70803d97ccf488d675d6837beeceb/io/serialport/25-serial.js#L283
RED.httpAdmin.get("/serialports", RED.auth.needsPermission('serial.read'), function(req,res) {
serialp.list(function (err, ports) {
res.json(ports);
});
});
Key points:
pick a url that is namespaced to your node type - this avoids clashes
the needsPermission middleware is there to ensure only authenticated users can access the endpoint. The permission should be of the form <node-type>.read.
Its edit dialog then queries that endpoint from here: https://github.com/node-red/node-red-nodes/blob/83ea35d0ddd70803d97ccf488d675d6837beeceb/io/serialport/25-serial.html#L240
$.getJSON('serialports',function(data) {
//... does stuff with data
});
Key points:
here the url must not begin with a /. That ensures the request is made relative to wherever the editor is being served from - you cannot assume it is being served from /.
I'm trying to query a VoltDB using its api:
const url = 'http://server:8080/api/1.0/'
const queryParam = encodeURIComponent('select * from table')
const queryURL = url + `?Procedure=#AdHoc&Parameters=['${queryParam}']&jsonp=console.log`
fetch(queryURL).then( response => {
response.text().then( text => console.log(text) )
})
With that code throws an "No Access-Control-Allow-Origin" error.
If I change the fetch call to this:
fetch(queryURL, { mode: 'no-cors').then( response => {
response.text().then( text => console.log(text) )
})
It does nothing
This is a browser security feature. If you are serving a web page from one url and within the page you have embedded url calls to another host or port, then the browser won't allow this.
One way to get around this is to add a proxy to your web server, so it can make the calls to port 8080, and pass the responses back to the web page from the same origin.
You may see some answers on Stack Overflow about using CORS to get around this error, but that requires changing the headers that VoltDB uses on port 8080, so that's not something you can do yourself, and we have no plans to do that.
Another solution is to use the voltdb.js file provided in some of our demos, such as the NBBO demo dashboard: https://github.com/VoltDB/voltdb/tree/master/examples/nbbo/web
I think this uses low-level javascript to open a socket to make the HTTP call without using XMLHttpRequest, so it avoids the No Access-Control-Allow-Origin error.
In the example, the code that is specific to the NBBO example is in demo.js, voltdb-dashboard.js contains code that is common to various example dashboards, and voltdb.js is the base library that provides access to call procedures asynchronously.
You should encode all the URI parameters not just the procedure parameters
$ curl --data 'Procedure=#AdHoc&Parameters=["select count(*) from store;"]' http://127.0.0.1:8080/api/1.0/
{"status":1,"appstatus":-128,"statusstring":null,"appstatusstring":null,"results":[{"status":-128,"schema":[{"name":"C1","type":6}],"data":[[100000]]}]}
or
$ curl --data 'Procedure=%40AdHoc&Parameters=%5B%22select+count(*)+from+store%3B%22%5D' http://127.0.0.1:8080/api/1.0/; echo
{"status":1,"appstatus":-128,"statusstring":null,"appstatusstring":null,"results":[{"status":-128,"schema":[{"name":"C1","type":6}],"data":[[100000]]}]}
(This title may not be clear but whatever you read here is a refinement trying to get past the "quality standards" script, which seem to be a lot stricter based on the tags chosen)
On the client side is there any way in jquery to set the name of the file downloaded?
lets say on my server the file is stored as QXYZO123 , but the file is really information.xls , a spreadsheet. Even the hyperlink says information.xls as I have information linking the random file name to the original file name.
When the user clicks on the link, by default it will try to download the file name as stored on the server, QXYZO123, but I want it instead to say information.xls , how can I set this to suggest the file name to save as?
I call it a suggestion because based on the user's browser settings, it will either auto download with my filename suggestion or ask the user where to download and save it, with the filename suggestion
There is no setting file properties from JQuery directly. JQuery means some Javascript codes and Javascript is a client side script language. This means you can do something, only on client side, not on server side.
But there is some indirect methods to do this. First of all, only way to do this is setting the file name on server side. It depends on what language you are using on server side. Depens on server side language, You can write your own download web api. Pass the file name as a request parameter to your api and let it give you the file with your customized file name. I don't know what language you are using on server side but I prepared a serverside code with node.js that accepts customized file name in 'GET' request as parameter.
var sys = require ('sys'),
url = require('url'),
http = require('http'),
qs = require('querystring');
var path = require('path');
var mime = require('mime');
var fs = require('fs');
var server=http.createServer(
function (request, response) {
if(request.method=='GET') {
var filePath='path_to_your_file_in_your_server_file_system';
var filestream = fs.createReadStream(filePath);
var url_parts = url.parse(request.url,true);
console.log(url_parts);
var fileName= url_parts.query.fileName; //Taking your customized file name from GET request parameters.
if(!fileName)
{
fileName=path.basename(filePath);
}
var mimetype = mime.lookup(filePath);
response.setHeader('Content-disposition', 'attachment; filename=' + fileName);
response.setHeader('Content-type', mimetype);
filestream.pipe(response);
response.writeHead( 200 );
response.end();
}
}
);
server.listen( 9080 );
When you run this server code with node.js you can get file with your customized file name from the url which localhost:9080?fileName=yourCustomizedFileName. As you see we are giving the fileName 'GET' request parameter in url. You can use javascript windows.location or something else to get the file with your customized file name on client side Javascript code.
Don't let the node.js code make you confused. The point of the solution is writing a downloading api which takes fileName parameter from 'GET' request, to send the file with this name to clients.
You can apply this node.js code for your server side language or research how to apply it. Probably, It won't be hard to find and apply it.