I have constructed an API with AWS API Gateway that is used to call an AWS Lambda function to do some serverless computing.
The payload size that I am passing through is too great (over 10 mb) so I have looked into compressing the payload clientside through the Javascript library pako making use of its gzip method. This outputs a compressed Unit8Array that I then pass through the body and make an API request call with fetch().
Now for some reason, AWS API Gateway does not acknowledge this and simply fails. There is no log output in Cloudwatch for the POST method and there is no response returned that might indicate what went wrong.
I have worked through https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-gzip-compression-decompression.html , https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-enable-compression.html , https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-make-request-with-compressed-payload.html but neither of these tutorials have helped sadly.
I have even tried passing through the exact same example shown in the 3rd link (by generating a gzip of the JSON Object they construct) and I have been attempting my own tests and this still has not succeeded in being recognised. I do believe this means that I must have missed something in the AWS API Gateway but I am very unsure of what. If anyone has any thoughts, thanks in advance.
I have attempted (and others):
Introducing Binary-Media Types in the settings of the API Gateway
Using AWS Lambda Proxy Integration so the request is sent through. The issue here is that when trying to convert I am getting Type Errors in Python stating it expected a Binary Type not Str despite using the correct methods to convert.
var myHeaders = new Headers();
myHeaders.append("Content-Encoding", "gzip");
myHeaders.append("Content-Type", "application/json");
json = JSON.stringify({"Hello": "World"});
var zlibOpts = {
level: 6,
to: 'string'
};
data = pako.gzip(json);
var requestOptions = {
method: 'POST',
headers: myHeaders,
body: data
};
fetch("https://<endpoint>", requestOptions)
.then(response => response.text())
.then(result => {
console.log(result);
})
.catch(error => console.log('error', error));
Related
I am trying to accept the URL Encoded format in postman to post some data to the Vue JS app, I am using the below-encoded format, how can I achieve that which npm package should I use?
you can use axios
const axios = require('axios')
const params = new URLSearchParams()
params.append('name', 'Akexorcist')
params.append('age', '28')
params.append('position', 'Android Developer')
params.append('description', 'birthdate=25-12-1989&favourite=coding%20coding%20and%20coding&company=Nextzy%20Technologies&website=http://www.akexorcist.com/')
params.append('awesome', true)
const config = {
headers: {
'Content-Type': 'application/x-www-form-urlencoded'
}
}
axios.post(url, params, config)
.then((result) => {
// Do somthing
})
.catch((err) => {
// Do somthing
})
x-www-form-urlencoded data is sent via HTTP headers
Most HTTP headers are not visible to your front-end JavaScript application. They are only visible to the server responding to the request. You cannot read them directly from JavaScript running in a web browser.
However, there are options...
Change the source; have the POST request changed to a GET and encode the parameters in the URL
A reverse proxy for your application could convert from POST parameters to GET parameters with some additional coding or configuration
Receive the request on your server and feed them into your Vue.js application; use something like php/asp/etc to serve your html instead of static HTML files and embed the posted parameters in the generated HTML page
There may be other options if you are creative, but the simplest is the first - just change the source so it no longer posts data.
I resolved it by adding middleware(server-side code such as .net web API) and then redirected it using query string)
I'm simply trying to send some urlencoded parameters via a GET request using fetch. I'm just trying to print the parameters using Express at the moment, like so:
app.get('/api', function (req, res) {
console.log(req.body);
res.sendStatus(200);
return;
});
This works just fine in Postman using a GET request and x-www-form-urlencoded key-value pairs. The webserver will print all the key-value pairs just fine.
But when I try and use fetch to do the exact same thing I get nothing but problems. I've tried two different methods:
fetch(`http://localhost:3000/api?user=test&password=123`, {
headers: {
'Content-Type': 'application/x-www-form-urlencoded'
}
});
The request does go through using this method, but the webserver only prints {} - an empty object.
var myHeaders = new Headers();
myHeaders.append("Content-Type", "application/x-www-form-urlencoded");
var urlencoded = new URLSearchParams();
urlencoded.append("user", "test");
urlencoded.append("password", "123");
var requestOptions = {
method: 'GET',
headers: myHeaders,
body: urlencoded,
};
fetch("localhost:3000/api", requestOptions)
.then(response => response.text())
.then(result => console.log(result))
.catch(error => console.log('error', error));
The request does not go through using this method, and the browser gives me the error TypeError: Window.fetch: HEAD or GET Request cannot have a body.
This code was generated using the request that works in Postman using the generate code snippets option.
What am I doing wrong?
The parameters in this URL:
http://localhost:3000/api?user=test&password=123
are in the query string, not in the body and thus the content-type does not apply to them - they are properly encoded to be in a URL. In Express, you would access these with req.query. You should see a value for req.query.user and req.query.password in your Exprss request handler.
Note, it is not recommended that you send user credentials in a URL like this because URLs are often present in log files at your ISP, at the recipient server, in proxies, in your browser history, etc... User credentials like this should be sent in POST request over https where the credentials would go encoded in the body (where it won't be logged or saved by intermediaries).
The fetch error is accurate. GET requests do not have a body sent with them. That would be for POST or PUT requests. A GET request is a "get" request for a resource that you specify only with a URL.
You're confusing request body with a query string.
Your second request (you don't need a Content-Type for it)
fetch("http://localhost:3000/api?user=test&password=123");
would be handled by the following Express function:
app.get('/api', function (req, res) {
console.log(req.query); // Note that query, not body is used.
res.sendStatus(200);
return;
});
You can access fields from the query object as req.query.user && req.query.password.
As for having a request body in a GET request: while RFC doesn't explicitly fordbid it, it requires server to not change response based on the contents of the body, i.e. the body in GET has no meaning in the standard, so JS HTTP APIs (both fetch & XmlHttpRequest) deny it.
firstly if you are trying to get some data from your API or others API you should do GET request in order to get your desired data from server for example, if you want to get a specific things like a user or something else you can pass your data in GET request URL using query string or route params.
secondly, if you want to authenticate and send your credentials to the server its not recommended to use GET request as i said earlier GET request simply is for fetching some data from server, so if you want to send your credential or anything else you are better off using POST request to send data to the server and you can't do POST request in the browser, so you have to use something like postman or insomnia in order to send your POST request to the server. i hope it could help you to solve your issue.
I am trying to make a site where I fetch my favorite podcasts from Podbean API.
I have worked with fetch before, but those API's were much easier to setup and there was no auth part. So that's what I am struggling with.
So this is basically what I have used before :
fetch('https://jsonplaceholder.typicode.com/todos/1')
.then(response => response.json())
.then(json => console.log(json))
From what I understand looking through various other threads :
I need to send get request with my client_id and client_secret to the get the access_token
From there on with access_token I get access and therefore can fetch date from the podcasts object.
I would appreciate any kind of guidance how to handle the auth part and obviously let me know if some of my thought processes are completely wrong.
Thanks in advance!
Ok, so I know the question is over 2 years old but I will still share my solution as I have struggled A LOT to understand how to use this API. This solution is only applicable if you are the owner of the podcast (or at least have access to the dev account).
The thing with the API is if you do not want to use auth2 (which I am still not sure how it works exactly with podbean), you have to fecth the data with a POST method not a GET and provide parameters (body and headers) and use HTTP basic authentication scheme. Their documentation is only in php but with some research you get what they are doing, the section applicable to this solution can be found here.
Here is the code:
const fetch = require("node-fetch");
const btoa = require('btoa');
const client_id = 'Enter your client id';
const client_secret = 'Enter your client secret';
const uri = 'https://api.podbean.com/v1/oauth/token';
// Base 64 encode client_id and client_secret to use basic authentication scheme
const auth = "Basic " + btoa(client_id + ':' + client_secret);
// Set POST request params
const options = {
method: 'POST',
body: 'grant_type=client_credentials',
headers : {
"Content-Type": "application/x-www-form-urlencoded",
"Authorization": auth
}
}
// Fetch
fetch(uri, options)
.then(res => res.json())
.then(data => console.log(data))
I hope this helps anyone who would try to use this API with javascript in the future.
Good Day All,
I'm trying to do a POST request using the puppeteer headless chrome library. I can't seem to get the below code to work.
// Get csrf token
let token = await page.evaluate(() => document.querySelector('[name="CSRFToken"]').value);
let postResponse = await page.evaluate(async(token, cookies) => {
let response = fetch("/loyalty/points", {
method : 'POST',
cookie : cookies,
postData : 'CSRFToken=' + token,
}).then(response => response.text()).catch(error => console.log(error));
return response;
});
console.log('Final response');
console.log(postResponse);
I keep on getting an error that the CSRF token has not been set or that the cookie is invalid.
My question is, am I using the correct method in puppeteer to do a POST? If so, is there any way for me to do some debugging that I can see the actual POST request that was sent?
I appreciate any advice or help. Thanks
You are not creating a request body: hence the error. The postData attribute you set on the request object is not any known attribute, so it won't be set on the request either, meaning that the server will never see your CSRF token. You should look into the MDN docs on fetch().
I believe you should be all good by simply replacing postData with body, but it's hard to know without access to your endpoint. For all we know it might require special headers.
Given that you only post normal form data (which is implied by your key=value code), I would also start using the FormData objects provided by your browser to avoid manual coding of implementation details.
const formData = new FormData();
formData.append("CSRFToken", token);
const response = fetch("/loyalty/points", {
method : 'POST',
cookie : cookies,
body : formData,
headers : {
'cookie' : cookies,
/* other headers you need, possibly content-type (see below) */
},
}).then(response => response.text()).catch(error => console.log(error));
return response;
});
Caveat: using the FormData API will always set the content-type of the data to multipart/form-data. If your server for some reason doesn't support that encoding, and you need to use application/x-www-form-urlencoded (see here for difference),
you can't blindly change the Content-Type: you also need to url encode the content.
For debugging I would simply use a normal Chrome instance to see this. You should be able to run the code there and see the network requests in DevTools (where it would be immediately noticeable that you POST an empty request).
I'm trying to get an OAuth token for the Reddit API following the Application Only OAuth instructions. My reddit app is an installed app, so for my grant_type I'm using https://oauth.reddit.com/grants/installed_client.
Currently I'm running a very short JS script to query the API and get a token:
const APP_ID = 'MY_APP_ID'
const DEVICE_ID = 'TRACKING_ID_20_TO_30_CHARS'
let form = new FormData()
form.append('grant_type', 'https://oauth.reddit.com/grants/installed_client')
form.append('device_id', DEVICE_ID)
fetch('https://www.reddit.com/api/v1/access_token', {
method: 'POST',
headers: new Headers({
'Content-Type': 'application/x-www-form-urlencoded',
'Authorization': `Basic ${btoa(`${APP_ID}:`)}`,
}),
body: form })
.then(handleResponse)
.then(function(data) {
console.log(data)
})
.catch(error => console.error(error))
function handleResponse(response) {
return response.json()
}
(Note: running the snippet as-is will give you a NetworkError because the APP_ID isn't a real one and I don't want to give mine out.)
The response I get is:
{
"error": "unsupported_grant_type"
}
When I try the same API request using a REST client I get back the expected response, so this makes me think that the problem is JavaScript-related. Since the grant_type matches what the instructions say I'm not really sure what to do with the error. I'm hoping someone else more experienced with OAuth will know what is going on here.
The problem was the use of the FormData object. In earlier stages of troubleshooting I found this answer on Reddit and decided to use it, but that didn't work for me.
It was submitting the data as multipart/form-data rather than application/x-www-form-urlencoded, which Reddit's OAuth server did not like. I wrote a helper function based on this answer which did the trick:
function urlEncode(data) {
let out = [];
for (let key in data) {
out.push(`${key}=${encodeURIComponent(data[key])}`);
}
return out.join('&')
}