Develop Reference

Cloudflare Worker TypeError: One-time-use body

I'm trying to use a Cloudflare Worker to proxy a POST request to another server.
It is throwing a JS exception – by wrapping in a try/catch blog I've established that the error is:
TypeError: A request with a one-time-use body (it was initialized from a stream, not a buffer) encountered a redirect requiring the body to be retransmitted. To avoid this error in the future, construct this request from a buffer-like body initializer.
I would have thought this could be solved by simply copying the Response so that it's unused, like so:
return new Response(response.body, { headers: response.headers })
That's not working. What am I missing about streaming vs buffering here?
addEventListener('fetch', event => {
var url = new URL(event.request.url);
if (url.pathname.startsWith('/blog') || url.pathname === '/blog') {
if (reqType === 'POST') {
event.respondWith(handleBlogPost(event, url));
} else {
handleBlog(event, url);
}
} else {
event.respondWith(fetch(event.request));
}
})
async function handleBlog(event, url) {
var newBlog = "https://foo.com";
var originUrl = url.toString().replace(
'https://www.bar.com/blog', newBlog);
event.respondWith(fetch(originUrl));
}
async function handleBlogPost(event, url) {
try {
var newBlog = "https://foo.com";
var srcUrl = "https://www.bar.com/blog";
const init = {
method: 'POST',
headers: event.request.headers,
body: event.request.body
};
var originUrl = url.toString().replace( srcUrl, newBlog );
const response = await fetch(originUrl, init)
return new Response(response.body, { headers: response.headers })
} catch (err) {
// Display the error stack.
return new Response(err.stack || err)
}
}

A few issues here.
First, the error message is about the request body, not the response body.
By default, Request and Response objects received from the network have streaming bodies -- request.body and response.body both have type ReadableStream. When you forward them on, the body streams through -- chunks are received from the sender and forwarded to the eventual recipient without keeping a copy locally. Because no copies are kept, the stream can only be sent once.
The problem in your case, though, is that after streaming the request body to the origin server, the origin responded with a 301, 302, 307, or 308 redirect. These redirects require that the client re-transmit the exact same request to the new URL (unlike a 303 redirect, which directs the client to send a GET request to the new URL). But, Cloudflare Workers didn't keep a copy of the request body, so it can't send it again!
You'll notice this problem doesn't happen when you do fetch(event.request), even if the request is a POST. The reason is that event.request's redirect property is set to "manual", meaning that fetch() will not attempt to follow redirects automatically. Instead, fetch() in this case returns the 3xx redirect response itself and lets the application deal with it. If you return that response on to the client browser, the browser will take care of actually following the redirect.
However, in your worker, it appears fetch() is trying to follow the redirect automatically, and producing an error. The reason is that you didn't set the redirect property when you constructed your Request object:
const init = {
method: 'POST',
headers: event.request.headers,
body: event.request.body
};
// ...
await fetch(originUrl, init)
Since init.redirect wasn't set, fetch() uses the default behavior, which is the same as redirect = "automatic", i.e. fetch() tries to follow redirects. If you want fetch() to use manual redirect behavior, you could add redirect: "manual" to init. However, it looks like what you're really trying to do here is copy the whole request. In that case, you should just pass event.request in place of the init structure:
// Copy all properties from event.request *except* URL.
await fetch(originUrl, event.request);
This works because a Request has all of the fields that fetch()'s second parameter wants.
What if you want automatic redirects?
If you really do want fetch() to follow the redirect automatically, then you need to make sure that the request body is buffered rather than streamed, so that it can be sent twice. To do this, you will need to read the whole body into a string or ArrayBuffer, then use that, like:
const init = {
method: 'POST',
headers: event.request.headers,
// Buffer whole body so that it can be redirected later.
body: await event.request.arrayBuffer()
};
// ...
await fetch(originUrl, init)
A note on responses
I would have thought this could be solved by simply copying the Response so that it's unused, like so:
return new Response(response.body, { headers: response.headers })
As described above, the error you're seeing is not related to this code, but I wanted to comment on two issues here anyway to help out.
First, this line of code does not copy all properties of the response. For example, you're missing status and statusText. There are also some more-obscure properties that show up in certain situations (e.g. webSocket, a Cloudflare-specific extension to the spec).
Rather than try to list every property, I again recommend simply passing the old Response object itself as the options structure:
new Response(response.body, response)
The second issue is with your comment about copying. This code copies the Response's metadata, but does not copy the body. That is because response.body is a ReadableStream. This code initializes the new Respnose object to contain a reference to the same ReadableStream. Once anything reads from that stream, the stream is consumed for both Response objects.
Usually, this is fine, because usually, you only need one copy of the response. Typically you are just going to send it to the client. However, there are a few unusual cases where you might want to send the response to two different places. One example is when using the Cache API to cache a copy of the response. You could accomplish this by reading the whole Response into memory, like we did with requests above. However, for responses of non-trivial size, that could waste memory and add latency (you would have to wait for the entire response before any of it gets sent to the client).
Instead, what you really want to do in these unusual cases is "tee" the stream so that each chunk that comes in from the network is actually written to two different outputs (like the Unix tee command, which comes from the idea of a T junction in a pipe).
// ONLY use this when there are TWO destinations for the
// response body!
new Response(response.body.tee(), response)
Or, as a shortcut (when you don't need to modify any headers), you can write:
// ONLY use this when there are TWO destinations for the
// response body!
response.clone()
Confusingly, response.clone() does something completely different from new Response(response.body, response). response.clone() tees the response body, but keeps the Headers immutable (if they were immutable on the original). new Response(response.body, response) shares a reference to the same body stream, but clones the headers and makes them mutable. I personally find this pretty confusing, but it's what the Fetch API standard specifies.

xhr caching values from getResponseHeader?

I'm running up against a very frustrating bug. I'm not exactly sure what is happening, but I think xhr is doing some kind of cache on the response headers.
My app is using devise_token_auth for the backend authentication service. We're using it with rotating access-tokens, and so I have written a function that runs after every request.
function storeAndGetResponseHeaders(xhr) {
const headersObj = {};
headerKeys.filter((key) => xhr.getResponseHeader(key))
.forEach((key) => {
headersObj[key] = xhr.getResponseHeader(key);
window.sessionStorage.setItem(key, xhr.getResponseHeader(key));
});
return headersObj;
}
where headerKeys is ['access-token', 'client', 'expiry', 'uid', 'token-type']. So any response that has these headers it should save them into sessionStorage and then return them in an object which gets stored within my AJAX service that I wrote and added to every request. We're using rxjs, and this service is just a thin wrapper around it. This is what RxAjax.ajax looks like.
ajax(urlOrRequest) {
const request = typeof urlOrRequest === 'string' ? { url: urlOrRequest } : urlOrRequest;
request.headers = Object.assign({}, this.headers, urlOrRequest.headers);
request.url = `${this.baseUrl}${request.url}`;
return Observable.ajax(request).map(this.afterRequest, this);
}
where this.headers is the stored headers from last request (or the loaded headers from sessionStorage). this.afterRequest is what sets the headers from the response xhr.
My problem is that I'm getting bad values into my headers object (specifically old access tokens). What I've noticed is that when I add a logging statement of headersObj after assignment, sometimes it will have old response headers from a past request. However when I look at the request itself in the dev console Network tab, it doesn't show any of the auth headers in the response headers ('access-token', 'client', etc...). This gets fixed for a little while if I do a hard refresh on the browser, but comes back seemingly inexplicably.
Note we're using rxjs to make our requests, which might be relevant (but I don't think it is the cause of this problem, as I'm trying to read the headers from the original xmlhttprequest object). Thanks!

As Barmar suggested in the comments, it was a caching issue. There may be a bug in the chrome console, where it wasn't showing the cached headers that were on the cached request. Hence even though it looked like there were no auth headers there really were.
It looks like if you're using jQuery you can add the option cache: false to the request in order to prevent caching. Because I'm not, the first thing I did was try adding ?cache=${new Date().toJSON} to each request, which successfully busted the cache and fixed my problem (that is what cache: false in jQuery does).
Our backend is in rails, and so I ended up adding
before_action :set_cache_headers
...
private
def set_cache_headers
response.headers["Cache-Control"] = "no-cache, no-store, must-revalidate"
end
to my application controller. Now no requests are cached by the browser. Not sure if this will be our long term solution or not

Change query string parametesr of JavaScript requests

Is there a way to change the query string of JavaScript-induced requests? I want to add "&myParam=myValue" to any request sent by my HTML/JS application.

I don't think there's anything built in that lets you do that.
In my apps, I always have a central function XHR goes through so I have a single point to do things like this. If you don't have that or need to intercept calls from 3rd party libs:
You could wrap XMLHttpRequest.open to handle the XHR ones:
var originalOpen = XMLHttpRequest.open;
XMLHttpRequest.open = function() {
var args = Array.prototype.slice.call(arguments);
args[0] += (args[0].indexOf("?") == -1 ? "?" : "&") + "myParam=" + encodeURIComponent("myValue");
return originalOpen.apply(this, args);
};
...and then similar for fetch. But it seems brittle.
Alternately, you might look at using a cookie for the parameter, as the browser will add the cookie to the requests. (That assumes the requests are going to an origina you can add cookies for in your code.)

You could use partial application to lock in defaults when you declare your fetch function and essentially decorate the standard call that will merge your defaults and the passed params.
const fetchFactory = defaults => (url, data) => {
// make a copy of the defaults
const params = Object.assign({}, defaults)
// assign the passed in data with the defaults
params.body = JSON.stringify(Object.assign(params.body, data))
// call fetch with the params
return fetch(url, params)
}
// create a default for POST using the factory
const postFetch = fetchFactory({
method: 'post',
headers: {
'x-requested-with': 'fetch',
'Authorization': 'basic:' + btoa('a secret')
},
body: {
myParam: 'value'
}
})
// now you can call your function
postFetch('http://somewhere.com', {
one: 1,
two: 2
})
.then(respone => response.json())

It seems to me that you are asking how to set/edit URL parameters in http requests. Something quite similar has been asked here: here
If you are using XMLHttpRequest then the accepted answer in the link should work perfectly. The two key things the note are
the url parameters are simply a javascript object that you convert
into a JSON string. This happens through JSON.stringify({ myParam:
'hi'});
the question/answer linked are making post requests but you
may not want to make that request as well, I suggest doing some
research about which HTTP request method you want -
https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html

JS - Getting either text or JSON with Fetch API

I am moving over from jQuery AJAX requests to the new Fetch API (nothing against jQuery, I still have it in my site, but Fetch looks - according to Jake Archibald and David Walsh and also IMHO - to be the new way of sending async requests).
As such, with jQuery, I had the following function (more or less):
function ajaxCall(type, url, data) {
return $.ajax({
type: type,
url: url,
data: data,
})
.fail(function(xhr, status, errorThrown) {
// Do fail stuff
})
.always(function(xhr, status) {
// Do always stuff
});
}
// Later...
var myAjax = ajaxCall(myType, myUrl, myData);
myAjax.done(function(xhr) {
// Do done stuff
});
This way, I could have one function be called to handle any and all ajax requests I could ever need (for the most part at least...). Note that I do not declare a dataType, as I use jQuery's intelligent guess. This way my server can send me whatever response and I could handle it (probably a smarter way to do this would be to pass another parameter with the data type - in the case the "intelligent guess" goes wrong, but this was the way I set it up).
I am now trying to recreate the above with the new Fetch API. What I have so far currently looks like this:
function fetchCall(url, method, body) {
// This if statement is supposed to handle
// query selectors (which in GET requests go in the url)
// on GET requests - as opposed to POST req's which go in the body
if (method === 'GET') {
var data = body;
url = new URL(url, location.protocol + '//' + location.host + '/');
Object.keys(data).forEach(key => url.searchParams.append(key, data[key]));
body = undefined;
}
return fetch(url, {
method: method,
body: body
}).then(function(res) {
if (res.ok) return res;
throw new Error('Server error. Status code: ', res.status);
}).catch(function(err) {
console.log(err);
});
}
// Later...
var myFetch = fetchCall(myUrl, myMethod, myBody);
myFetch.then(function(res) {
console.log(res);
});
The problem I am running into is that if res.ok return res; does not state what type of response it is (i.e. res.json(), res.blob(), res.text(), etc.).
Thus, I am wondering how to set up a dynamic way of setting the type of response body. Is this even possible at the Fetch API's current state of development? Is it just that there is something I am not duplicating in MDN?
After messing around with this, I also realized I could make it always set to return res.text(); and the if the call is supposed to be JSON, use JSON.parse(response);, but I do want it to be dynamic. What if I end up wanting to return a blob()?

So, as far as the conversation has reached, there is a way to understand what type of content has been received, with two remarks:
Typically you have to always know and expect exact content type, and a universal solution is rather odd in case of fetching from a certain remote endpoint, and
The Content-Type header is what will tell you the type of content received, but the server may send a wrong header, which is very unusual to happen and therefore is negligible.
The Response object has header property that is (kind of) a Map, so you can use its get method to get a value by key.
The easiest and cleanest way to check if the returned value is a certain MIME type you expect is by using a regular expression:
// replace url with the actual API endpoint URL
fetch(url).then(response => {
const contentType = response.headers.get('Content-Type'); // -> "text/html; charset=utf-8"
if (/text\/html/i.test(contentType)) {
// do something when the Content-Type is text/html
} else if (/application\/json/.test(contentType)) {
// do something when the Content-Type is application/json
}
// and so on, for every Content-Type you need.
}).catch(error => {
// do something when error happens
});

Generating a unique value for a header on each $http call

I have a fairly large AngularJS application and for logging purposes, I am being tasked with adding a custom header to all of our HTTP requests from the app that contain a unique ID for each request. This really is more valuable to our API calls, but as of now I'm just aiming for all requests (getting templates, styles, etc.)
I am currently using a provider decorator to patch each of the methods exposed by $HttpProvider (implementation based on this post), to attempt to call the ID method each time one of those $http methods runs, and add the appropriate header:
module.config([
'$provide',
function ($provide) {
$provide.decorator('$http', [
'$delegate',
function addUniqueIdHeader($http) {
var httpMethods = ['get', 'post', 'put', 'patch', 'delete'];
/**
* Patched HTTP factory function that adds a request ID each time it is called.
* #param {string} method - A valid HTTP method.
* #return {function} A function that sets various request properties.
*/
function httpWithHeader(method) {
return function(url, data, config) {
config = config || {};
config.headers = config.headers || {};
// the magic
config.headers['My-Custom-Header'] = aUniqueId();
data = data || {};
config.method = method.toUpperCase();
// return `$http` with a modified config, adding the URL and data passed in
// `_.extend()` is lodash, not underscore.
return $http(_.extend(config, {
url: url,
data: data
}));
}
};
// back up the orginal methods and patch
_.each(httpMethods, function (httpMethod) {
var backupMethod = '_' + httpMethod;
$http[backupMethod] = $http[httpMethod];
$http[httpMethod] = httpWithHeader(httpMethod);
});
return $http;
}
]);
}
]);
What I have so far works some of the time, but doesn't seem to work consistently (some API requests have it, some don't). I should note that we are using a quite old version of AngularJS (1.0.6) and no, I cannot upgrade (as much as I would love to) so the use of request interceptors is not possible. Additionally, we use Restangular for the majority of our API interactions.
My question is, is using a provider decorator the right way to go? If so, is there a cleaner way to add the header without having to override/patch each individual HTTP method that I'm overlooking?
Thanks in advance.

I ended up solving my issue by utilizing Restangular's request interceptors, since the version of Angular we use doesn't have them baked in.