Develop Reference

How do you implement digest authentication with SHA-256 when modern web browsers disable SubtleCrypto when you're not in a secure context (https)?

I've got a very basic web server running on an ESP8266 microcontroller, so system resources are very limited, but I figure it can probably deal with SHA-256 (I'll guess we'll see, but that's a separate issue).
I've got the barebones digest authentication implementation working well enough that I can access the site via curl.
This microcontroller is not meant to be exposed to the internet, it's only something you'd access internally via your lan, so SSL isn't an option (not sure how well the microcontroller would hold up if it tried to support HTTPS).
So, here's my scenario:
User tries to access the site on the microcontroller from their favourite web browser. They are given a 401 code and redirected to a login page (along with the WWW-Authenticate header).
This login page needs to take the information from the WWW-Authenticate header, as well as the username and password input by the user and generate the hash it needs to send in an Authorization header.
Unfortunately, the built in functionality for generating SHA-256 hashes is disabled if you're not connected via HTTPS (according to https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto) - So does this mean that digest authentication is not supported by web browsers natively (unless they choose to use MD5 instead of something more secure)? If you're implementing digest authentication, you need to supply your own hashing functions?
I also don't quite understand why disabling the ability to do hashes helps user security (in a non-https context).
An assumption:
After a cursory google search, it seems that digest authentication only makes sense for http, as if you're using https then you've already got better encryption protecting your credentials.

So to answer my own question (partly):
It turns out that the user side is handled entirely by the browser (i.e. Generic login modal pop-up). Java script doesn't even have access to the response headers anyway.
I guess to prevent poorly implemented non-secure logins?
The examples I've seen work around it by providing functionality entirely via xhttprequests.
I still don't know what benefit denying access to hashing algorithms provides though

What's the best way to resume an HTTP POST?

I am trying to handle following case, I am new to web-app world so please pardon my ignorance (if any):
Assumptions/Constraints:
I don't have any control over the web-app code base whatsoever.
Authentication is being handled at http proxy level.
Scenario:
User has authenticated with a web-app using SAML and has been accessing the web-app for quite some time.
His authentication token (or cookie) expires.
He submits a form (HTTP POST).
Server needs to perform the authentication workflow again, which requires a HTTP redirect.
Now, the requirement is to somehow resubmit the original HTTP POST, after completing the authentication workflow, automatically for the user. A couple of options I could think of are:
Use javascript injection to store the POST state in browser's sessionStorage and rePOST things after completing the auth workflow. But I couldn't figure how would this work if the original POST was done using XMLHttpRequest ???
Store POST state on server (or proxy) side and do an internal rePOST and return the result to the browser.
Are there any other options ??? It'd be great if we could avoid saving state on server side. How do people usually handle such scenarios?
TIA,
praddy

Generally the second approach is implemented.
You send the request to the server running the application, it preserves the POST data (actually the again doing the authentication does this), sends you to authenticate, redirects you back and POSTs the data again. This feature may already be present in your authentication infrastructure and just needs to be enabled, but that of course depends what you are running.
This feature is sometimes also referred to 'POST data preservation'.

Public facing Authentication mechanisms for REST

I am designing a new service that would enable 'customers' to register and pay a per-use type fee for particular searches they perform. This service would be exposed using a RESTFul and SOAP interface. Typically the web service would integrate with the customer's website and then be exposed to the 'public' where anyone would be able to use the customer's website and take advantage of my web service features (which the customer would pay for but have full control of moderating the requests so they don't get charged too much).
I want to design the service that optimises the integration to make it as simple as possible. The web service API will change so creating an internal proxy to expose the web service to the public in some cases is too much of a detractor for customers. So the issue as I see it is creating a web service that balances authentication, security and integration.
Ideal
Not use OAuth
Avoid forcing the customer to create an internal proxy which re-exposes the same web service API I have already.
Be secure (token username/pass whatever and ssl)
Embed a javascript library in customer website - This would be a client Javascript library to make integration steps even easier.
The Javascript library would need to be secure enough so that the public wouldn't be able to simply grab credentials and re-purpose it themselves
Not be too hacky, if possible, so the web service doesn't have to be re-built if Firefox 87 comes out (to be released in as many minutes) and decides to fubar it.
It seems that some kinda of 3-way authentication process is needed for this to work, i.e. authenticates a particular client (in the public), the web service (the customer) and the web service.
Has anyone implemented something kind of similar and how did they tackle a situation like this?
I also understand there is a balance between what can be done, and what would violate cross-domain security, so perhaps the entire web service might be exposed by another GET only interface which would return JSONP data.
/** Addendum **/
I have since discovered a web service that does what I'm looking after. However, I am not confident I understand the implementation details entirely. So perhaps someone could also elaborate on my thinking.
The web service I discovered seems to host the Javascript on the service side. The customer would then integrate their website with the service side by including the Javascript in a script tag, but supplies a key to do so i.e.
Somehow if I add the script to my website it doesn't work. So somewhere along the line the token must be registered to a particular customer domain, and the 'client-lib.js' is actually a servlet or something similar which can somehow detect that the user from the 'public' coming in has actually originated from the 'customer' domain.
Is my thinking right? Is there some kind of http header that can be used this way? Is that safe?
Cheers

First of all - let me provide you a link to another SO question which I answered yesterday - as it gives a pretty extensive answer to a similar question-set.
I am assuming that you are going to charge the owner of the site from which the search is made, and not care so much who the individual user is who makes the search. If that's incorrect, please clarify and I will update my answer accordingly.
Clearly, in any such case, the first and foremost thing you need to do is to make sure you know which client this is on each request. And - as you said, you also want to make sure you're protecting yourself from cross-site attacks and people stealing your user's keys.
What you might consider would be the following:
Create a private key on your side - which only your service knows.
Whenever a new consumer site creates an account with you, create a new shared key which only you and they will know. I suggest creating this key by using your private key as a password, and encrypting some kind of identifier which will let you identify this particular user.
As part of your registration process, make the consumer site tell you what URI they will be using your scripts on.
Now - the way that you both do your tracking and authentication becomes fairly simple.
You mentioned providing a JS library which won't need to update every time FF updates. I suggest building that library using jQuery, or another, similarly supported cross-browser JS foundational library - and letting that wrap your AJAX.
When the client site requests your script, however, have them provide you something like:
http://www.yourdomain.com/scripts/library.js?key={shared key}
On your side, when you receive this request, check the following:
When you decrypt their shared key using your private key, you should not get gibberish. If you do - it's because their key has been altered in some way - and is not valid. This should result in a 401: Unauthorized error.
Once you decrypt the key and know which client site this is (because that's what the key contains) - check to make sure that the request is coming from the same URI that client registered with. This now protects you from someone stealing their key and injecting it into a different website.
As long as the above matches, let them download the file.
Now - when you serve the JS file, you can do so in a way that injects the key into that file - and therefore it has access to their shared key. On each of your AJAX requests, include that key so that you can identify which client this request is coming from again. In a RESTful environment, there shouldn't really be sessions - so you need this level of authentication on each post. I suggest including it as a cookie.
On your server-side - simply repeat the checks of their key on each subsequent request - and voila - you've built yourself some fairly tight security without a lot of overhead.
That said - if you expect a lot of traffic - you may want to come back to this and explore more deep security processes in the future, as rolling your own security matrix can leave unexpected holes. However - it is a good start and will get you off the ground.
Feel free to ask any questions if you need, and I will try to update my answer accordingly.

The best way to go about it is something like this (taking that you want to use javascript hosted on your server and make the include part as simple as it can be):
*user registers on your website and he receives a token for his domain
*the user can include a js file pointing to your server
the js file will be something like:
<script type="text/javascript" src="http://your.server.com/js.php?token=###&widget=xxx"></script>
or
<script type="text/javascript" src="http://your.server.com/js.js?token=###&widget=xxx"></script>
if you will use a .htaccess to redirect
*in the php file check if the token matches the requests domain, if yes echo out the js lib, if not throw a error or something
*in the js you will need to build some ajax calls to your service and stuff to manipulate the HTML (create a widget holder,show some data, etc.)
*also all the calls should have the token, and again you can use the same logic to check if token==server address
EDIT:
The REFERER is sent by the client's browser as part of the HTTP protocol, and is therefore unreliable indeed.
If you want to verify if a request is coming from your site, well you can't, but you can verify the user has been to your site and/or is authenticated. Cookies are sent in AJAX requests so you can rely on that. But this means you need to use something like oAuth
If you want to use this method, you should still check the referrer as well to prevent CSRF en.wikipedia.org/wiki/Cross-site_request_forgery
Ideally you should use a unique token per session per user (per request if you're paranoid) to prevent CSRF attacks. Checking the referrer is just security by obfuscation and not quite a real solution.

Is there a cross-browser way to ignore 401 errors triggered by XHRs?

Attempting to keep as pure of a REST approach as possible, I decided that the application would be the API - down to the last bit.
Unfortunately, I have reached one stumbling block. After writing a Digest authentication handler in PHP, I found myself unable to make the experience as friendly for users in web browsers as an form-based authentication method would be.
The reason for this is that even though I can simulate a Digest authentication response through Javascript, using username and password from a HTML form (I can give the nonce values to the script safely, due to the way they are generated), on failure, the browser still shows the standard, ugly authentication prompt.
Is there any way around this? An earlier question references mozBackgroundRequest, but that seems hardly cross-browser.
Thanks!

You could make a request with AJAX first, to make sure the authentication details provided work.
If they do, do what you have been doing. If they don't, tell the user about it in your own, nicer way...
EDIT
If you control the auth routine in PHP (and if auth fails, presumably doing header('HTTP/1.1 401 Unauthorized'); etc) you can just... not do it. Instead, respond with a normal 200 OK but put a string in the body that you can look for in ajax.responseText (like 'authFailed' or something).
You can just add a header (or a cookie, or something) to the AJAX requests that are not present in normal browser requests, to allow you to distinguish between the AJAX auth checks and a normal, authenticated session.
The exact mechanics are a little fuzzy in my head at the moment (it's been a long day) but I am sure it could be done using that method.

WCF REST Authentication - Determine Web Request

What is the best way to determine if a request being made to my REST service originated from a web client. I know I can look at the user-agent, but my concern is that is very easy to spoof.
The reason I want to know who originated the request is because of the following. It is natively built into web-browser that you can't do cross-domain requests. Therefore I don't need to worry about the authentication, because I know the request originated from my website.
My site is built entirely in HTML and Javascript, any suggestions?
Or is there a good way in Javascript to store a hidden username / password I can use just for my website, without it being displayed to the public?
Thanks,
Adam

Anything put in the javascript can be found by using a debugger, such as Firebug, so even though it isn't visible to the user, it can be found by a user.
But, if the javascript first calls to a REST service to get an encrypted token, then the token, which has a timestamp encrypted within it, could be the password, so you then pass the username and token to call the rest of the REST services.
Your server could then validate that it had created the token and that it is not expired, and that the username matches what was encrypted in the token.
But, this depends on if you have any control over the REST service.

The reason I want to know who originated the request is because of the following. It is natively built into web-browser that you can't do cross-domain requests. Therefore I don't need to worry about the authentication, because I know the request originated from my website.
This is not a good assumption. For the reasons you already gave (easily spoofed User-Agent), anyone could make a request to your application. You can even disable cross origin policy in firefox and chrome from the client side - so even if you could verify the request came from a browser, it's still possible to get around your security measures:
Disable same origin policy in Chrome
There are a couple of standard ways to implement security for this kind of service (as James mentioned, assuming you have control over the REST service).
Use Basic Authentication - If your application is communicating with the WCF service via HTTPS, basic authentication is probably the easiest method. See this question
If both your website and your WCF service are implemented using .NET, and your ASP.NET web application is using Forms Authentication, you could share the Forms Auth cookie and use that for authentication. See this question