Hi I'm trying to interact with an HTTP api and I am required to send an HTTP post request some form data. And then in the HTTP-BODY element I am required to supply some more data (can be either csv or xml). So my challenge is in understanding how to do this via the Needle API as its unclear via just the documentation.
I have been able to successfully do :
needle.post( host, formdata, function( err, resp, body){ });
The above works for the situations where I don't require posting the HTTP-BODY content.
I have tried the following but receive an error :
options = {
headers : { 'Content-Type': 'text/tab-separated-values; charset=iso-8859-1'},
body : text
}
needle.post( host, formdata, options, function( err, resp, body){ });
Could any one guide me to the correct way to do this somewhat basic task.
Thanks.
I used the Restler package and ditched Needle. Needle looks great and I'm sure there is a way to do this, but I couldn't figure it out in time.
https://github.com/danwrong/restler
Related
I am currently trying to set up ParseHub's API in order to feed data from the web scraping software into a MongoDB database. I am currently attempting to take the last ready scraping run and write it to a JSON file. However, I receive the following error. view error
Here is the code I am currently attempting. I'm not very familiar with request and it seems to be depreciated (ignore the "insert key here" sections).
request({
uri: 'https://www.parsehub.com/api/v2/runs/KEYGOESHERE/data',
method: 'GET',
gzip: true,
qs: {
api_key: "INSERTED KEY",
format: "json"
}
}, function(err, resp, body) {
console.log(body);
fs.writeFile('APIsample.json', body);
});
Any help would be much appreciated, or even a way to run this without request. Thanks!
I'm trying to use NodeJS to scrape a website that requires a login by POST.
Then once I'm logged in I can access a separate webpage by GET.
The first problem right now is logging in. I've tried to use request to POST the login information, but the response I get does not appear to be logged in.
exports.getstats = function (req, res) {
request.post({url : requesturl, form: lform}, function(err, response, body) {
res.writeHeader(200, {"Content-Type": "text/html"});
res.write(body);
res.end();
});
};
Here I'm just forwarding the page I get back, but the page I get back still shows the login form, and if I try to access another page it says I'm not logged in.
I think I need to maintain the client side session and cookie data, but I can find no resources to help me understand how to do that.
As a followup I ended up using zombiejs to get the functionality I needed
You need to make a cookie jar and use the same jar for all related requests.
var cookieJar = request.jar();
request.post({url : requesturl, jar: cookieJar, form: lform}, ...
That should in theory allow you to scrape pages with GET as a logged-in user, but only once you get the actual login code working. Based on your description of the response to your login POST, that may not be actually working correctly yet, so the cookie jar won't help until you fix the problems in your login code first.
The request.jar(); didn't work for me. So I am using the headers response to make another request like this:
request.post({
url: 'https://exampleurl.com/login',
form: {"login":"xxxx", "password":"xxxx"}
}, function(error, response, body){
request.get({
url:"https://exampleurl.com/logged",
header: response.headers
},function(error, response, body){
// The full html of the authenticated page
console.log(body);
});
});
Actualy this way is working fine. =D
Request manages cookies between requests if you enable it:
Cookies are disabled by default (else, they would be used in
subsequent requests). To enable cookies, set jar to true (either in
defaults or options).
const request = request.defaults({jar: true})
request('http://www.google.com', function () {
request('http://images.google.com')
});
I want to send some post data to an api
10.11.12.13/new/request/create
this is an API to create a new request in portal. now I am making one application in NodeJs and want to create request from node js application.
now I have to send in this format
{"user":"demo", "last_name":"test","contact":"989898989"}
so how can I send data on above url to create a new request.
I am a beginner in NodeJs and don't have much idea.
any help will be appreciated.
Thanks in advance
I would recommend to use axios or any other request lib :
const axios = require('axios');
axios.post('10.11.12.13/new/request/create', {
user: 'demo',
last_name: 'test',
contact: '989898989',
});
here is an example using request module
var headers = {
'Content-Type': 'application/json'
}
var options = {
url: "10.11.12.13/new/request/create" ,
method: 'POST',
headers: headers,
json: true,
body: {user:"demo", last_name:"test",contact:"989898989"}
}
request(options, function (error, response, body) {
if (error) {
//do something
}
console.log(body)//do something with response
})
You can use postman REST client for GET method using your URL and Body (which you want to post) and click on * Code * and select NodeJS and their you will find code generated for you to work with. Here is the link https://www.getpostman.com/docs/postman/sending_api_requests/generate_code_snippets
With my experience, it is good to start with Request package for node js. Here is the link for your reference: https://www.npmjs.com/package/request
I have a simple API route set up in Express (consumed mainly via my Angular frontend):
app.post('/api/sendEmail', emailApi.sendEmail);
I've made a module that sits in my backend and also needs to call this service. I figured the easiest way was to do a POST request:
request({
url: '/api/sendEmail',
method: 'POST',
json: {
template: template.toLowerCase(),
obj: mailObj
}
}, function(error, response, body){
console.log('error', error);
console.log('response', response);
console.log('body', body);
});
However, I get this error:
Error: Invalid URI "/api/sendEmail"
What am I doing wrong here?
Change Url to'http://127.0.0.1:3000/api/sendEmail', because you're
calling an internal api with in express or you can also use localhost
in place of 127.0.0.1.
request({
url: 'http://127.0.0.1:3000/api/sendEmail', //on 3000 put your port no.
method: 'POST',
json: {
template: template.toLowerCase(),
obj: mailObj
}
}, function (error, response, body) {
console.log({error: error, response: response, body: body});
});
emailApi.sendEmail is just a function. You are much better off calling it directly. Using the network in this manner would be a serious waste of resources.
On a practical note, there are some complex issues about how to address yourself on the network. Usually you can accomplish this via localhost, but there's no guarantee that the server is listening at that address. So you'll have to take that into account.
You need to use an absolute URI (including the protocol, domain, and port if it's not listening on the default port).
For example, if you know that the server will be listening at localhost:3000, you would want to replace your url value with 'http://localhost:3000/api/sendEmail'.
Assuming you are not using a web server like nginx and are developing on localhost. The express app does not know from where the request has originated. Try setting your Url as http://localhost:300/api/sendEmail.
I am using Angularjs and nodejs in the project and working on file uploads. I want to send the post request to the url endpoint in a secure way as I need to attach accesstoken with the request. So the way I did this was, I added the directive to choose the file from UI and once it gets the file, I append it using FormData() like this in the controller
var fd = new FormData();
fd.append('file',myFile);
and sending this formdata object to the nodejs server like mentioned here http://uncorkedstudios.com/blog/multipartformdata-file-upload-with-angularjs
expect this request will be going to my nodejs server url from there I will be making another post request to external web service
$http.post('api/collections/upload',fd, {
transformRequest: angular.identity,
headers: {
'Content-type': undefined
}
});
So it will attach the right content-type and boundaries in the request. I am getting the file on server side nodejs when I do
function(req,res){
console.log(req.files); //I am able to see the file content
}
It is uploaded on the nodejs server side.
Now I want to make a post request using the req.files to a different endpoint along with proper accessToken and headers. Tried many things but not able to make the request go thru. Not sure how can I attach the imagedata/ req.files along with the request. I tried these two things mentioned in request npm module https://www.npmjs.org/package/request
1)
request.post({
url: 'https://www.example.com/uploadImage',
headers: {
'Authorization': <accessToken>,
'Content-type': 'multipart/form-data'
},
body: req.files
});
Don't know how can I attach and binary data with this request and how can I put boundary. Is boundary needed when you want to send the uploaded image with this request?
2)
fs.createReadStream(req.files.file.path, {encoding: base64}).pipe(request.post({
url: 'https://www.example.com/uploadImage',
headers: {
'Content-type': 'multipart/form-data'
}
}));
Can someone please suggest some ideas on how can I send this post request using request npm module? Thanks.
Documentation here has lots of examples of doing exactly what you describe:
https://github.com/mikeal/request#streaming
As can be seen in that link, the request library adds a .pipe() method to your http's req object, which you should be able to use like the examples in the link:
function(req, res) {
req.pipe(request.post('https://www.example.com/uploadImage');
}
Or something similar.
You were nearly there with your #2 try, but that would only work if you have previously written the file out to disk and were reading it in with fs.createReadStream()
your suggestion helped me to atleast know what I was trying was right. Another article that solved my problem was this http://aguacatelang.wordpress.com/2013/01/05/post-photo-from-node-js-to-facebook/ .Basically, here is what I did and it worked. Thanks for your suggestion.
var form = new FormData();
form.append('file', fs.createReadStream(__dirname + '/image.jpg'));
var options = {
url: 'https://www.example.com/uploadImage?access_token='+ <accesstoken>,
headers: form.getHeaders()
};
form.pipe(request.post(options,function(err,res){
if(err){
log.debug(err);
}
else {
log.debug(res);
}
}));