I am currently trying to set up ParseHub's API in order to feed data from the web scraping software into a MongoDB database. I am currently attempting to take the last ready scraping run and write it to a JSON file. However, I receive the following error. view error
Here is the code I am currently attempting. I'm not very familiar with request and it seems to be depreciated (ignore the "insert key here" sections).
request({
uri: 'https://www.parsehub.com/api/v2/runs/KEYGOESHERE/data',
method: 'GET',
gzip: true,
qs: {
api_key: "INSERTED KEY",
format: "json"
}
}, function(err, resp, body) {
console.log(body);
fs.writeFile('APIsample.json', body);
});
Any help would be much appreciated, or even a way to run this without request. Thanks!
Related
I've been stuck on this issue for some time now, I am trying to subscribe to Linkedin's webhook using ngrok for testing on localhost, and have been trying for some time now, i have tried using encode uri's as well but still running into error, I have verified that the APP_ID, profileId and organizationId i'm using are correct, but still i get the same error. I have also tried using the Restli protocol that linkedin suggests in their documentation but to no avail.
let url = `https://api.linkedin.com/v2/eventSubscriptions/(developerApplication:urn:li:developerApplication:${config.IN.APP_ID},user:urn:li:person:${profileId},entity:urn:li:organization:${organizationId},eventType:ORGANIZATION_SOCIAL_ACTION_NOTIFICATIONS)`;
return new Promise((resolve, reject) => {
request(
{
url,
method: 'PUT',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${accessToken}`,
// 'X-Restli-Protocol-Version': '2.0.0',
},
json: {
webhook: "https://url.ngrok.io/api/v1/webhook/linkedin/callback"
},
},
(err, response, body) => {
if (err) {
reject(err);
} else {
resolve(body);
}
},
);
});
I have been receiving this error constantly no matter what I try, I have tried sending the url like this:
https://api.linkedin.com/v2/eventSubscriptions/(developerApplication:urn:li:developerApplication:{${config.IN.APP_ID}},user:urn:li:person:{${profileId}},entity:urn:li:organization:{${organizationId}},eventType:ORGANIZATION_SOCIAL_ACTION_NOTIFICATIONS)
https://api.linkedin.com/v2/eventSubscriptions/${encodeURIComponent((developerApplication:urn:li:developerApplication:${config.IN.APP_ID},user:urn:li:person:${profileId},entity:urn:li:organization:${organizationId},eventType:ORGANIZATION_SOCIAL_ACTION_NOTIFICATIONS))
All I receive is this error
'{"serviceErrorCode":100,"message":"Unpermitted fields present in RESOURCE_KEY: Data Processing Exception while processing fields [/key]","status":403}'
Any help would be appreciated, I have been stuck on this for a while now.
The request seems to be OK , but the method should be GET (not PUT)
One thing is to check which ID are you using for application_id. The application ID is the one in the url - https://www.linkedin.com/developers/apps/<id from here>/settings
. You need to use and uncomment the header for Restli.
I'd say that your url needs to look like this, as this is from their original POSTMAN collection.
https://api.linkedin.com/v2/eventSubscriptions/(developerApplication:urn%3Ali%3AdeveloperApplication%3A{{application_id}},user:urn%3Ali%3Aperson%3A{{person_id}},entity:urn%3Ali%3Aorganization%3A{{organization_id}},eventType:ORGANIZATION_SOCIAL_ACTION_NOTIFICATIONS)
You can validate here, their full collection - https://www.postman.com/linkedin-developer-apis/workspace/e781b3ac-4101-4d60-8981-afcb4812623d/request/16069442-9d0bf046-ea81-4af0-9515-d07246a0ab39
LinkedIn webhooks does not support ngrok URIs
I am really struggling to get a successful response when doing a post request to the google recaptcha api. I am receiving the following response:
{
"success": false,
"error-codes": [
"invalid-input-response",
"invalid-input-secret"
]
}
I had a look at reCAPTCHA - error-codes: 'missing-input-response', 'missing-input-secret' when verifying user's response (missing details on POST) and followed the answer as closely as possible but with no success.
Here is my file below:
var request = require('request');
module.exports = {
verifyCaptcha: function(req, res) {
var secret = 'SECRET_KEY';
var response = JSON.stringify(req.body.response);
request({
url: 'https://www.google.com/recaptcha/api/siteverify',
method: 'POST',
headers: { "Content-Type": "application/x-www-form-urlencoded" },
body: `secret=${secret}&response=${response}`,
}, function (err, response, body) {
if (err) {
res.status(500).send({
error: "Could not verify captcha"
});
} else {
res.status(200).send({
message: body
});
}
});
},
}
If anyone has a solution to this problem please let me know!
Due to the docs: https://developers.google.com/recaptcha/docs/verify
invalid-input-secret: The secret parameter is invalid or malformed.
Maybe you have mixed the site_key and the secret_key.
You need to add the user remote IP address.
var user_ip = req.headers['x-forwarded-for'] || req.connection.remoteAddress;
request({
url: 'https://www.google.com/recaptcha/api/siteverify',
method: 'POST',
headers: { "Content-Type": "application/x-www-form-urlencoded" },
body: `secret=${secret}&response=${response}&remoteip=${user_ip}`}...
Another thing I see that you are not using template literal, you should change the quotes to ` instead of '.
OR, You should use a ready-made module for reCaptcha, like this one:
https://www.npmjs.com/package/recaptcha
For reCAPTCHA Enterprise, check the official docs: https://cloud.google.com/recaptcha-enterprise/docs/create-assessment.
In short, you need to use the library that Google provides:
const { RecaptchaEnterpriseServiceClient } =
require('#google-cloud/recaptcha-enterprise');
const client = new RecaptchaEnterpriseServiceClient();
const [ response ] = await client.createAssessment({...});
RecaptchaEnterpriseServiceClient requires a service account to be created beforehand as described here. The key for that account with the right roles set can then be read by the app. Check the arguments of the constructor to see the available options to pass the data if the file cannot be retrieved automatically.
var response = JSON.stringify(req.body.response);
The stringifying here is probably the cause of the invalid-input-response error.
If your body is something like {"g-recaptcha-response": "..."}, you need to pull out the response value and pass that directly in your post.
Regarding invalid-input-secret, if you have set up your key and secret through the classic interface at https://www.google.com/u/1/recaptcha/admin/create, then you shouldn't have a problem.However if you set up a key with recaptcha Enterprise on Google Cloud, then it requires that you do Oauth authentication to the Google Cloud API and then use the create.assessment endpoint to get back information on the validity of the user. As Yuuhn implied, the Google provided library makes interaction with recaptcha Enterprise easier, without a lot of documentation digging to find where your REST API calls need to go.
I want to send some post data to an api
10.11.12.13/new/request/create
this is an API to create a new request in portal. now I am making one application in NodeJs and want to create request from node js application.
now I have to send in this format
{"user":"demo", "last_name":"test","contact":"989898989"}
so how can I send data on above url to create a new request.
I am a beginner in NodeJs and don't have much idea.
any help will be appreciated.
Thanks in advance
I would recommend to use axios or any other request lib :
const axios = require('axios');
axios.post('10.11.12.13/new/request/create', {
user: 'demo',
last_name: 'test',
contact: '989898989',
});
here is an example using request module
var headers = {
'Content-Type': 'application/json'
}
var options = {
url: "10.11.12.13/new/request/create" ,
method: 'POST',
headers: headers,
json: true,
body: {user:"demo", last_name:"test",contact:"989898989"}
}
request(options, function (error, response, body) {
if (error) {
//do something
}
console.log(body)//do something with response
})
You can use postman REST client for GET method using your URL and Body (which you want to post) and click on * Code * and select NodeJS and their you will find code generated for you to work with. Here is the link https://www.getpostman.com/docs/postman/sending_api_requests/generate_code_snippets
With my experience, it is good to start with Request package for node js. Here is the link for your reference: https://www.npmjs.com/package/request
I have a simple API route set up in Express (consumed mainly via my Angular frontend):
app.post('/api/sendEmail', emailApi.sendEmail);
I've made a module that sits in my backend and also needs to call this service. I figured the easiest way was to do a POST request:
request({
url: '/api/sendEmail',
method: 'POST',
json: {
template: template.toLowerCase(),
obj: mailObj
}
}, function(error, response, body){
console.log('error', error);
console.log('response', response);
console.log('body', body);
});
However, I get this error:
Error: Invalid URI "/api/sendEmail"
What am I doing wrong here?
Change Url to'http://127.0.0.1:3000/api/sendEmail', because you're
calling an internal api with in express or you can also use localhost
in place of 127.0.0.1.
request({
url: 'http://127.0.0.1:3000/api/sendEmail', //on 3000 put your port no.
method: 'POST',
json: {
template: template.toLowerCase(),
obj: mailObj
}
}, function (error, response, body) {
console.log({error: error, response: response, body: body});
});
emailApi.sendEmail is just a function. You are much better off calling it directly. Using the network in this manner would be a serious waste of resources.
On a practical note, there are some complex issues about how to address yourself on the network. Usually you can accomplish this via localhost, but there's no guarantee that the server is listening at that address. So you'll have to take that into account.
You need to use an absolute URI (including the protocol, domain, and port if it's not listening on the default port).
For example, if you know that the server will be listening at localhost:3000, you would want to replace your url value with 'http://localhost:3000/api/sendEmail'.
Assuming you are not using a web server like nginx and are developing on localhost. The express app does not know from where the request has originated. Try setting your Url as http://localhost:300/api/sendEmail.
Hi I'm trying to interact with an HTTP api and I am required to send an HTTP post request some form data. And then in the HTTP-BODY element I am required to supply some more data (can be either csv or xml). So my challenge is in understanding how to do this via the Needle API as its unclear via just the documentation.
I have been able to successfully do :
needle.post( host, formdata, function( err, resp, body){ });
The above works for the situations where I don't require posting the HTTP-BODY content.
I have tried the following but receive an error :
options = {
headers : { 'Content-Type': 'text/tab-separated-values; charset=iso-8859-1'},
body : text
}
needle.post( host, formdata, options, function( err, resp, body){ });
Could any one guide me to the correct way to do this somewhat basic task.
Thanks.
I used the Restler package and ditched Needle. Needle looks great and I'm sure there is a way to do this, but I couldn't figure it out in time.
https://github.com/danwrong/restler