Using Puppeteer, I am able to intercept HTTPResponses and their HTTPRequests:
page.on("response", async response => {
let request = response.request(); // Getting the response request
let responseHeaders = response.headers(); // Check response headers
response.buffer().then(buffer => {
// Play with response content
});
})
Depending on the response content, I need to send the request again like a fresh one and get its response buffer. Instantiating an identical and new request is a valid option.
I know I could use node-fetch as a last resort, but Puppeteer seems to have everything embedded to do it without adding packages.
Do you know how to achieve this?
Puppeteer HTTPRequest doc
Puppeteer HTTPRequest class
You can use page.evaluate to send a post request using fetch API
await page.evaluate(() => {
return fetch('url', {method: 'POST', body: 'test' }).then(res => res.json())
})
So then you can make a request after the requestfinished event fired.
page.setRequestInterception(true)
page.on('requestfinished', async (request: Request) => {
let response = request.response() // Getting the response request
let responseHeaders = response.headers() // Check response headers
let responseBuffer = await response.buffer() // Get the buffer required
let responseJSON = await response.json() // Get parsed JSON body
await page.evaluate(([headers, buffer, json]) => { // Replay request with buffer received
let someData1 = buffer.toString() // Change buffer to string type
let someData2 = headers['Content-Type'] // or maybe use some headers data
let someData3 = json.properties.value // or use response data object properties
// This fetch API below will do the rest
return fetch('url', { method: 'POST', body: 'test' }).then(res => res.json())
}, [responseHeaders, responseBuffer, responseJSON])
})
Related
How can i send a form parameter in a post (JavaScript fetch()) request?
e.g.
curl --form "avatar=#me.jpg" "https://example.com/api/v4/endpoint"
I tried the folloing code:
const form = new FormData()
form.append("foo":"bar")
fetch( "https://someapi.org/api", {
method: 'POST',
headers:form
} )
.then( response => response.json() )
.then( response => {
console.log(response)
} );
}
which doesn't work for me.
The FormData should be supplied as the body property of the RequestInit, like this:
Make sure you use the correct Content-Type header. You can read about Using FormData Objects on MDN.
TS Playground
so-70865955.ts:
async function example () {
const form = new FormData();
form.append("foo", "bar");
const apiAddress = "https://someapi.org/api";
const init: RequestInit = {
method: 'POST',
headers: new Headers([['content-type', 'application/x-www-form-urlencoded']]),
body: form,
};
const response = await fetch(apiAddress, init);
const data = await response.json();
console.log(data);
}
// Invoke it
example();
In your console:
deno run --allow-net so-70865955.ts
I am wondering if someone might be able to help figure out how to pass a post body to another endpoint with cloudflare workers?
I am trying to get the incoming request post to post to url.
const url = 'https://webhook.site/#!/b2f75ce2-7b9e-479a-b6f0-8934a89a3f3d'
const body = {
results: ['default data to send'],
errors: null,
msg: 'I sent this to the fetch',
}
/**
* gatherResponse awaits and returns a response body as a string.
* Use await gatherResponse(..) in an async function to get the response body
* #param {Response} response
*/
async function gatherResponse(response) {
const { headers } = response
const contentType = headers.get('content-type') || ''
if (contentType.includes('application/json')) {
return JSON.stringify(await response.json())
} else if (contentType.includes('application/text')) {
return response.text()
} else if (contentType.includes('text/html')) {
return response.text()
} else {
return response.text()
}
}
async function handleRequest() {
const init = {
body: JSON.stringify(body),
method: 'POST',
headers: {
'content-type': 'application/json;charset=UTF-8',
},
}
const response = await fetch(url, init)
const results = await gatherResponse(response)
return new Response(results, init)
}
addEventListener('fetch', (event) => {
return event.respondWith(handleRequest())
})
I created a worker at https://tight-art-0743.ctohm.workers.dev/, which basically forwards your POST request's body to a public requestbin. You can check what is it receiving at: https://requestbin.com/r/en5k768mcp4x9/24tqhPJw86mt2WjKRMbmt75FMH9
addEventListener("fetch", (event) => {
event.respondWith(
handleRequest(event.request).catch(
(err) => new Response(err.stack, { status: 500 })
)
);
});
async function handleRequest(request) {
let {method,headers}=request,
url=new URL(request.url)
// methods other than POST will return early
if(method!=='POST') return new Response(`Your request method was ${method}`);
const forwardRequest=new Request("https://en5k768mcp4x9.x.pipedream.net/", request)
forwardRequest.headers.set('X-Custom-Header','hey!')
return fetch(forwardRequest)
}
You can see it working with a simple CURL request
curl --location --request POST 'https://tight-art-0743.ctohm.workers.dev/' \
--header 'Content-Type: application/json' \
--data-raw '{"environment": {"name": "Sample Environment Name (required)"}}'
Two things worth noting, in the worker's code:
I'm passing the original request as the init parameter, through which original headers and body are transparently forwarded to the requestbin, also allowing for some extra header manipulation if neeeded.
In this example I'm not actually doing anything with the request body. Therefore there's no need to await it. You just connect incoming and outgoing streams and let them deal with each other.
Another example: let's add a /csv route. Requests starting with /csv will not forward your POST body. Instead they will download a remote CSV attachment and POST it to the requestbin. Again, we aren't awaiting for the actual CSV contents. We pass a handle to the response body to the forwarding request
async function handleRequest(request) {
let {method,headers}=request,
url=new URL(request.url)
if(method!=='POST') return new Response(`Your request method was ${method}`);
const forwardRequest=new Request("https://en5k768mcp4x9.x.pipedream.net/",request)
if(url.pathname.includes('/csv')) {
const remoteSource=`https://cdn.wsform.com/wp-content/uploads/2018/09/country_full.csv`,
remoteResponse=await fetch(remoteSource)
return fetch(forwardRequest,{body:remoteResponse.body})
}
forwardRequest.headers.set('X-Custom-Header','hey!')
return fetch(forwardRequest)
}
While your code should theoretically work, the fact that you're unwrapping the response means your worker could be aborted due to hitting time limits, or CPU, or memory. On the contrary, when using the streams based approach,
your worker's execution finishes as soon as it returns the forwarding fetch. Even if the outgoing POST is still running, this isn't subject to CPU or time limits.
I'm trying to access the returned content-type from my GET request so I can decide the kind of preview I want to like for html maybe pass through an iframe and for a PDF maybe some viewer. The problem is when I do console.log(response.headers) the object returned doesn't have content-type in it but when I check the networks tab the response headers has content-type:html/text. How can I get the content-type from the response headers?
this is how my GET request looks like
const getFile = async () => {
var requestOptions = {
method: "GET",
headers: context.client_header,
redirect: "follow",
};
let statusID = context.currentStatus.ApplicationID;
var response = await fetch(
process.env.REACT_APP_API_ENDPOINT +
"/services/getStatus?ApplicationID=" +
statusID,
requestOptions
);
console.log(response.headers);
if (response.ok) {
let fileHtml = await response.text();
setfileURL(fileHtml);
} else {
alert.show("Someting went wrong");
}
};
The Headers object isn't a great candidate for console.log() since it is not easily serialisable.
If you want to see everything in it, try breaking it down to its entries via spread syntax
console.log(...response.headers)
You'll probably find that you can in fact access what you want via
response.headers.get("content-type")
See Headers.get()
I'm trying to make a fetch request with custom herokuapp proxy to an API, but when I do that it gives an error. Error says "There is no Target-Endpoint header in the request". Here is my code.
var userTargetUrl = `http://api.steampowered.com/ISteamUser/ResolveVanityURL/v0001/?key=${steamApiKey}&vanityurl=${url}`
const response = await fetch(proxyUrl + userTargetUrl, {
headers: {
'Target-Endpoint': 'http://api.steampowered.com/ISteamUser/ResolveVanityURL/v0001/?'
}
})
const data = await response.json()
url = data['response']['steamid']
I'm following their instructions, but I couldn't figure it how to do it.
I won't pretend to be entirely sure, but maybe you can try
const response = await fetch(proxyUrl, {
headers: {
"Target-Endpoint": userTargetUrl
}
});
I'm trying to consume the /api/stream endpoint of a Drone 1.0 server. This endpoint keeps the HTTP connection opened and just streams new upcoming events to notify the consumer of events.
I tried with that piece of code using the javascript Fetch API
await fetch("https://drone.company.com/api/stream/", {
headers
})
.then(function(response) {
return response.text();
})
.then(function(data) {
console.log(data);
});
This code works well, but the then callback is always called after the request has ended.
Is there a way I can make fetch a stream of the body received while the request is processing ?
you can read from response.getReader() like
let response = await fetch("https://drone.company.com/api/stream/", {
headers
});
const reader = response.body.getReader();
while(true) {
const {done, value} = await reader.read();
if (done) {
break;
}
const text = new TextDecoder("utf-8").decode(value);
console.log(`Received ${text}`)
}
we need to use TextDecoder since value contains Uint8Array data not text. Is not supported by IE. fetch is not supported either btw.
based on https://javascript.info/fetch-progress