Upload from Client Browser to Google Cloud Storage Using JavaScript - javascript

I am using Google Cloud Storage. To upload to cloud storage I have looked at different methods. The method I find most common is that the file is sent to the server, and from there it is sent to Google Cloud storage.
I want to move the file directly from the user's web browser to Google Cloud Storage. I can't find any tutorials related to this. I have read through the Google API Client SDK for JavaScript.
Going through the Google API reference, it states that files can be transferred using a HTTP request. But I am confused about how to do it using the API client library for JavaScript.
People here would require to share some code. But I haven't written any code, I have failed in finding a method to do the job.

EDIT 1: Untested Sample Code
So I got really interested in this, and had a few minutes to throw some code together. I decided to build a tiny Express server to get the access token, but still do the upload from the client. I used fetch to do the upload instead of the client library.
I don't have a Google cloud account, and thus have not tested this, so I can't confirm that it works, but I can't see why it shouldn't. Code is on my GitHub here.
Please read through it and make the necessary changes before attempting to run it. Most notably, you need to specify the location of the private key file, as well as ensure that it's there, and you need to set the bucket name in index.html.
End of edit 1
Disclaimer: I've only ever used the Node.js Google client library for sending emails, but I think I have a basic grasp of Google's APIs.
In order to use any Google service, we need access tokens to verify our identity; however, since we are looking to allow any user to upload to our own Cloud Storage bucket, we do not need to go through the standard OAuth process.
Google provides what they call a service account, which is an account that we use to identify instances of our own apps accessing our own resources. Whereas in a standard OAuth process we'd need to identify our app to the service, have the user consent to using our app (and thus grant us permission), get an access token for that specific user, and then make requests to the service; with a service account, we can skip the user consent process, since we are, in a sense, our own user. Using a service account enables us to simply use our credentials generated from the Google API console to generate a JWT (JSON web token), which we then use to get an access token, which we use to make requests to the cloud storage service. See here for Google's guide on this process.
In the past, I've used packages like this one to generate JWT's, but I couldn't find any client libraries for encoding JWT's; mostly because they are generated almost exclusively on servers. However, I found this tutorial, which, at a cursory glance, seems sufficient enough to write our own encoding algorithm.
I'd like to point out here that opening an app to allow the public free access to your Google resources may prove detrimental to you or your organization in the future, as I'm sure you've considered. This is a major security risk, which is why all the tutorials you've seen so far have implemented two consecutive uploads.
If it were me, I would at least do the first part of the authentication process on my server: when the user is ready to upload, I would send a request to my server to generate the access token for Google services using my service account's credentials, and then I would send each user a new access token that my server generated. This way, I have an added layer of security between the outside world and my Google account, as the burden of the authentication lies with my server, and only the uploading gets done by the client.
Anyways, once we have the access token, we can utilize the CORS feature that Google provides to upload files to our bucket. This feature allows us to use standard XHR 2 requests to use Google's services, and is essentially designed to be used in place of the JavaScript client library. I would prefer to use the CORS feature over the client library only because I think it's a little more straightforward, and slightly more flexible in its implementation. (I haven't tested this, but I think fetch would work here just as well as XHR 2.).
From here, we'd need to get the file from the user, as well as any information we want from them regarding the file (read: file name), and then make a POST request to https://www.googleapis.com/upload/storage/v1/b/<BUCKET_NAME_HERE>/o (replacing with the name of your bucket, of course) with the access token added to the URL as per the Making authenticated requests section of the CORS feature page and whatever other parameters in the body/query string that you wish to include, as per the Cloud Storage API documentation on inserting an object. An API listing for the Cloud Storage service can be found here for reference.
As I've never done this before, and I don't have the ability to test this out, I don't have any sample code to include with my answer, but I hope that my post is clear enough that putting together the code should be relatively straightforward from here.
Just to set the record straight, I've always found OAuth to be pretty confusing, and have generally shied away from playing with it due to my fear of its unknowns. However, I think I've finally mastered it, especially after this post, so I can't wait to get a free hour to play around with it.
Please let me know if anything I said is not clear or coherent.

Related

Where to store API key in ReactJS?

I quick question!
I have a webapp that only fetches data from an API so i dont have i backend part. My question is where do you keep your API key? Accordingly to React docs you should not store API keys in the app so how do you manage this when you have an webapp that consumes an API and you have to use the API key in your get requests?
Let's do a bit of explanation, so you connect the dots and design it more robustly. There are two (three) places where to store it in the end:
frontend (your React application)
backend (your server)
third-party service
TL;DR: use non-frontend solution + rate limiting to a registered user and have the registration step secured properly (+ captcha).
Frontend
Storing anything on the frontend side is mostly a bad idea, unless you're completely sure you can allow such data to be exposed - constants, names, icons, maybe some URLs so you don't have it hardcoded in the JS files.
Your "compiled" ReactJS (or any other framework) when built is just a slightly mangled (minified/transpiled/etc etc) JavaScript, but for that to work the client has to retrieve it and have it executed in the browser. Therefore before the ReactJS application even starts, there are 1+ requests downloading the JavaScript code and other parts of the application depending on the framework.
Those you can see in the network monitoring tab in any modern browser or simply use Wireshark (unless encrypted, then it's a little bit annoying) or a local proxy if using a less sane browser.
After retrieval you can simply try Ctrl+F or any online deminifier/deobfuscator if you don't know how to do it yourself and you can retrieve the key.
Implications when retrieved
I can impersonate you for the service that issues the API key
I can lock your key/account by calling too often (just for fun or to retrieve some info)
I can use your web for scraping while not needing to pay for the API key (if paid) or to register to such service vendor
If it's per-request API key and there's some limitation that would make it cost you money, I can just run some silly while (true) { callYourApi() } via a service to make me anonymous just to make it cost you
Depending on the API key and how serious you intend to approach this problem, you might utilize the .env file for development purposes only. Though you should never ever store an API key in the frontend unless you explicitly have to store it in there (e.g. maps) because it's mostly a very stupid idea and allows anyone to misuse it.
Backend
Your server, if properly configured and secured, will store the key anywhere which isn't accessible by simply path traversing (if in a file) or scraping (if you attempt to retrieve the key to execute on the frontend part).
Therefore the most sane and secure way would be to retrieve the data (of any service) by having either a custom API or a scheduled script collecting the data, which when your frontend gets called will be able to retrieve as pre-rendered or already fetched, thus no key needed for that case.
However! There's a trick to that. If you design your custom API as /api/<key>=123 or /api/<param> and you use that parameter for the original API to filter on frontend, the attacker couldn't care less for the API key because you've already created an API for free and made it public and unsecure.
So GET /yourapi/<my data> and API key for free without even needing to have one displayed.
How to do it safely? Two simple approaches:
pre-rendering data to HTML
You then fetch with frontend and just display - but this one can be scraped, it's just a bit annoying if more complex, but that's it. Server-side rendering sounds nice, but doesn't really work for this case. It should be mostly used to make the frontend fast or to template the views, but never for security purposes as the silver bullet solution (because it doesn't work that way).
rate limiting + CORS + account management
with rate limiting you make sure that a user (preferably you have that API called only after a user is logged in) can call that API only e.g. 10 times within 1 hour and with CORS you make sure it's callable only by your frontend.
It's not a silver bullet either, anybody with a little bit of brain can simply scrape your API locally thus go around CORS, but the rate limit will still hit hard, if you forbid registering more than 1 user from a single IP or if you require a phone number for verification. And add some annoying captcha, so it's problematic to automate for some people.
Still it can be attacked and misused, but it's painful unless you allow the same phone number (or any other ID less comfortable to get / requiring effort to get) to be used multiple times, so it'll make the most incompetent people go away... and the remaining ones, well, they'd play with your website anyway, so have a proper security assessment / harden your server if you maintain it alone.
Third-party
It's like 2., but you don't maintain the "low-level" server part, because the third-party is then managing it for you, you just need to specify conditions under which it'll be called. This applies to Firebase or Supabase which kind of behaves like a separate backend, but can have multiple modules (for FB, 1, 2).
Thus you'd use Firebase functions (or other alternatives), where you'd have your key e.g. even hardcoded and the client (browser) wouldn't have any access to that, add a limit, cors, perhaps some user registration limit and you're kind of done.
Note: Any domain, IP, region, phone number restrictions can be bypassed, so do not rely on them. It's just a mean to require effort when misusing your website for something different than you intended.
domain: curl http(s)://yourweb/path -H "Host: spoofed-domain"
region or IP: proxy, VPN, Tor, I2P, just somebody else's computer/server + ssh, some random WiFi
phone number: can go to a local shop and buy 10 fresh ones if I wanted to
It's more of a recommendation for them to keep your API keys server-side, and let your web app communicate with your server. Otherwise malicious users could easily steal your API key and use it for whatever.
If you think it isn't much of a security risk if your key gets (scratch that, is) compromised, that's fine then, you can just keep it in your webapp. Really depends on your use case and what that API key is for.
The only way to do this without exposing your API keys in your client app is to create a backend and serve the client app from the backend app as stated by Kelvin Schoofs and Peter Badida answers above (or use a third party service such as AWS Credential Vault). I suggest you use Node Express library for a backend as this will handle a lot of the boiler plate code for you. There are plenty of tutorials online for this.
Using a dotenv file as suggested by a few other users will only hide your API code from version sharing tools like Git (because you can ignore the dotenv in gitignore). It is very important that you understand the process of dotenv with a react app. Any user who opens the Dev console in their browser can view your exposed API keys in the static HTML.
Create a dotenv file and store all secret and API keys. Make sure to use REACT_APP_ before every variable.
DOCS: https://create-react-app.dev/docs/adding-custom-environment-variables/
dotenv package: https://www.npmjs.com/package/dotenv

How to secure the JavaScript API Access Token?

There are numerous online resources which provide JavaScript APIs to access their services. To be more clear, I will base my question on the example of MapBox, but this applies well to many other services in various domains.
When someone wants to use such a service in a web application (like the map imagery from MapBox for example), they typically need to Register/Sign Up and obtain an access token to access the service.
Now, if I would use the API from the server side - there is no issue: I know my token is stored securely somewhere on the server and is only exposed upon communication between my server and the service provider, which is OK as long it is HTTPS. However, in case of a JavaScript API (for example if I use Leaflet to render a map from MapBox), I am supposed to have my access token in a JavaScript which is exposed to the user's web browser - and so it makes it extremely easy to find someone's access token. My users, or in a case of a public service, literally anyone, would be able to find the token in the browser's "Dev Tools".
This token however, as for me, should be considered as a sensetive data - service usage is tracked based on the authentication this token provides. If you pay for the service based on its usage it becomes critical, but even if you don't (like, if you use a Free/Starter/Non Paid plan) - service usage is limited and I'd like to be sure it is only me who uses it.
Is my only option a proxy via my own web server?
Is there a way to secure the access token used by a JavaScript API to access an external service, provided that JavaScript is executed in a user's browser?
Restrict Access with CORS
Make your web server return the access tokens on an ajax request from you javascript with CORS setup. Token can be captured with this method visiting your app.
Provide Tokens to Authorized Users
You can also add authentication on your webserver to provide limited access to the users you allow. Token can be captured with this method but only by authorized users.
Proxy Requests
The only way to completely protect that token is to proxy the requests through your server. Token cannot be captured with this method. Note that this may be against terms of service.
Javascript API tokens (and all client tokens, in fact) are always visible to the client (unless using them only server-side, as in node). There is no way around that. As you mentioned, the only way to truly secure an API key and keep it private is to store it in the server, then request the server to make the request on the client's behalf.
5 years later, this is not necessarily for the original poster but for anyone still interested, Mapbox now allows you to easily restrict tokens by domain(s):
https://account.mapbox.com/access-tokens (assuming you are signed in)
I will speak only about map imagery APIs like Mapbox, it seems that unfortunatly only services like Google Maps, Here Maps, Bing Maps etc offer ip/domain filtering by service provider or this type of security, all offers based on OSM i met don't propose it. As Justin Poehnelt said the only reliable way is to build a proxy, but it's usually forbidden. I find this in the ToU of Mapbox:
You may not redistribute Map Assets, including from a cache, by
proxying, or by using a screenshot or other static image instead of
accessing Map Assets through the Mapping APIs.
You may like to read up on CORS headers
These allow you restrict which domain can call a remote web service.

Server-side flow for Google Drive API authorization of a javascript Chrome extension

I was reading #Nivco answer to Authorization of Google Drive using JavaScript and saw:
"...all you have to do it is use server-side code to process the authorization code returned after the Drive server-side flow (you need to exchange it for an access token and a refresh token). That way, only on the first flow will the user be prompted for authorization. After the first time you exchange the authorization code, the auth page will be bypassed automatically.
Server side samples to do this is available in our documentation."
Having read the documentation I am still pretty confused about how to process the authorization code and ultimately pass the access and refresh tokens to my Chrome extension so that it can proceed without the server for future requests. Can someone provide an example of the server-side code to do this?
As background I have a Chrome Extension with several thousand users that is built on the Google DocList API but I am trying to transition to the Drive API since the other one is being deprecated. Ideally my code would be entirely stand alone as an extension but I'm willing to accept the single authorization request through my server that Nivco's answer requires.
Thanks!
We've just ported our JavaScript application from using server to client flow. We've removed the server part entirely, it's not needed any longer.
You can see the source code that we used online, it's available uncompressed.

Public facing Authentication mechanisms for REST

I am designing a new service that would enable 'customers' to register and pay a per-use type fee for particular searches they perform. This service would be exposed using a RESTFul and SOAP interface. Typically the web service would integrate with the customer's website and then be exposed to the 'public' where anyone would be able to use the customer's website and take advantage of my web service features (which the customer would pay for but have full control of moderating the requests so they don't get charged too much).
I want to design the service that optimises the integration to make it as simple as possible. The web service API will change so creating an internal proxy to expose the web service to the public in some cases is too much of a detractor for customers. So the issue as I see it is creating a web service that balances authentication, security and integration.
Ideal
Not use OAuth
Avoid forcing the customer to create an internal proxy which re-exposes the same web service API I have already.
Be secure (token username/pass whatever and ssl)
Embed a javascript library in customer website - This would be a client Javascript library to make integration steps even easier.
The Javascript library would need to be secure enough so that the public wouldn't be able to simply grab credentials and re-purpose it themselves
Not be too hacky, if possible, so the web service doesn't have to be re-built if Firefox 87 comes out (to be released in as many minutes) and decides to fubar it.
It seems that some kinda of 3-way authentication process is needed for this to work, i.e. authenticates a particular client (in the public), the web service (the customer) and the web service.
Has anyone implemented something kind of similar and how did they tackle a situation like this?
I also understand there is a balance between what can be done, and what would violate cross-domain security, so perhaps the entire web service might be exposed by another GET only interface which would return JSONP data.
/** Addendum **/
I have since discovered a web service that does what I'm looking after. However, I am not confident I understand the implementation details entirely. So perhaps someone could also elaborate on my thinking.
The web service I discovered seems to host the Javascript on the service side. The customer would then integrate their website with the service side by including the Javascript in a script tag, but supplies a key to do so i.e.
Somehow if I add the script to my website it doesn't work. So somewhere along the line the token must be registered to a particular customer domain, and the 'client-lib.js' is actually a servlet or something similar which can somehow detect that the user from the 'public' coming in has actually originated from the 'customer' domain.
Is my thinking right? Is there some kind of http header that can be used this way? Is that safe?
Cheers
First of all - let me provide you a link to another SO question which I answered yesterday - as it gives a pretty extensive answer to a similar question-set.
I am assuming that you are going to charge the owner of the site from which the search is made, and not care so much who the individual user is who makes the search. If that's incorrect, please clarify and I will update my answer accordingly.
Clearly, in any such case, the first and foremost thing you need to do is to make sure you know which client this is on each request. And - as you said, you also want to make sure you're protecting yourself from cross-site attacks and people stealing your user's keys.
What you might consider would be the following:
Create a private key on your side - which only your service knows.
Whenever a new consumer site creates an account with you, create a new shared key which only you and they will know. I suggest creating this key by using your private key as a password, and encrypting some kind of identifier which will let you identify this particular user.
As part of your registration process, make the consumer site tell you what URI they will be using your scripts on.
Now - the way that you both do your tracking and authentication becomes fairly simple.
You mentioned providing a JS library which won't need to update every time FF updates. I suggest building that library using jQuery, or another, similarly supported cross-browser JS foundational library - and letting that wrap your AJAX.
When the client site requests your script, however, have them provide you something like:
http://www.yourdomain.com/scripts/library.js?key={shared key}
On your side, when you receive this request, check the following:
When you decrypt their shared key using your private key, you should not get gibberish. If you do - it's because their key has been altered in some way - and is not valid. This should result in a 401: Unauthorized error.
Once you decrypt the key and know which client site this is (because that's what the key contains) - check to make sure that the request is coming from the same URI that client registered with. This now protects you from someone stealing their key and injecting it into a different website.
As long as the above matches, let them download the file.
Now - when you serve the JS file, you can do so in a way that injects the key into that file - and therefore it has access to their shared key. On each of your AJAX requests, include that key so that you can identify which client this request is coming from again. In a RESTful environment, there shouldn't really be sessions - so you need this level of authentication on each post. I suggest including it as a cookie.
On your server-side - simply repeat the checks of their key on each subsequent request - and voila - you've built yourself some fairly tight security without a lot of overhead.
That said - if you expect a lot of traffic - you may want to come back to this and explore more deep security processes in the future, as rolling your own security matrix can leave unexpected holes. However - it is a good start and will get you off the ground.
Feel free to ask any questions if you need, and I will try to update my answer accordingly.
The best way to go about it is something like this (taking that you want to use javascript hosted on your server and make the include part as simple as it can be):
*user registers on your website and he receives a token for his domain
*the user can include a js file pointing to your server
the js file will be something like:
<script type="text/javascript" src="http://your.server.com/js.php?token=###&widget=xxx"></script>
or
<script type="text/javascript" src="http://your.server.com/js.js?token=###&widget=xxx"></script>
if you will use a .htaccess to redirect
*in the php file check if the token matches the requests domain, if yes echo out the js lib, if not throw a error or something
*in the js you will need to build some ajax calls to your service and stuff to manipulate the HTML (create a widget holder,show some data, etc.)
*also all the calls should have the token, and again you can use the same logic to check if token==server address
EDIT:
The REFERER is sent by the client's browser as part of the HTTP protocol, and is therefore unreliable indeed.
If you want to verify if a request is coming from your site, well you can't, but you can verify the user has been to your site and/or is authenticated. Cookies are sent in AJAX requests so you can rely on that. But this means you need to use something like oAuth
If you want to use this method, you should still check the referrer as well to prevent CSRF en.wikipedia.org/wiki/Cross-site_request_forgery
Ideally you should use a unique token per session per user (per request if you're paranoid) to prevent CSRF attacks. Checking the referrer is just security by obfuscation and not quite a real solution.

Securing my Node.js app's REST API?

I could do with some help on my REST API. I'm writing a Node.js app which is using Express, MongoDB and has Backbone.js on the client side. I've spent the last two days trying to work out all of this and not having much luck. I've already checked out:
Securing a REST API
Securing my REST API with OAuth while still allowing authentication via third party OAuth providers (using DotNetOpenAuth)
http://www.thebuzzmedia.com/designing-a-secure-rest-api-without-oauth-authentication/
http://tesoriere.com/2011/10/10/node.js-getting-oauth-up-and-working-using-express.js-and-railway.js/
I want to keep my backend and frontend as separate as possible so I thought about using a carefully designed REST API would be good. My thinking is that if I ever get round to developing an iPhone app (or something else like that), it could use the API to access data.
BUT, I want this to be secure. A user has logged into my web app and I want to ensure my API is secure. I read about OAuth, OAuth 2.0, OpenID, Hmac, hashes etc... I want to avoid using external logging in (Facebook/Twitter/etc) I want the registering and logging in to be on my app/server.
...but I'm still confused here. Maybe it's late at night or my brain is just fried, but I could really do with some steps on what to do here. What are the steps for me to create a secure API?
Any help, any information, any examples, steps or anything would be great. Please help!
In order of increasing security / complexity:
Basic HTTP Auth
Many API libraries will let you build this in (Piston in Django for example) or you can let your webserver handle it. Both Nginx and Apache can use server directives to secure a site with a simple b64encoded password. It's not the most secure thing in the world but it is at least a username and password!
If you're using Nginx you can add a section to your host config like so:
auth_basic "Restricted";
auth_basic_user_file /path/to/htpasswd;
(Put it in your location / block)
Docs: http://wiki.nginx.org/HttpAuthBasicModule
You'll need to get the python script to generate that password and put the output into a file: http://trac.edgewall.org/browser/trunk/contrib/htpasswd.py?format=txt
The location of the file doesn't matter too much as long as Nginx has access to it.
HTTPS
Secure the connection from your server to the app, this is the most basic and will prevent man in the middle attacks.
You can do this with Nginx, the docs for it are very comprehensive: http://wiki.nginx.org/HttpSslModule
A self-signed certificate for this would be fine (and free!).
API Keys
These could be in any format you like but they give you the benefit of revoking access should you need to. Possibly not the perfect solution for you if you're developing both ends of the connection. They tend to be used when you have third parties using the API, eg Github.
OAuth
OAuth 2.0 is the one to go with here. While I don't know the underlying workings of the spec it's the defacto standard for most authentication now (Twitter, Facebook, Google, etc.) and there are a ton of libraries and docs to help you get those implemented. That being said, it's usually used to authenticate a user by asking a third party service for the authentication.
Given that you doing the development both ends it would probably be enough to put your API behind Basic HTTP Auth and serve it over HTTPS, especially if you don't want to waste time messing around with OAuth.
Here's a different way of thinking about it:
Let's suppose for a moment that you're not using an API. Your user logs into the app, providing some credentials, and you give a cookie or similar token of some sort to the user, which you use to identify that user has logged in. The user then requests a page containing restricted information (or creating/modifying/deleting it), so you check that this token to ensure that the user is allowed to view that information.
Now, it sounds to me that the only thing you're changing here is the way that information is delivered. Instead of delivering the information as rendered HTML, you're returning the information as JSON and rendering it on the client side. Your AJAX requests to the server will carry that same logged-in token as before, so I suggest just checking that token, and restricting the information down to 'just what the user is allowed to know' in the same way.
Your API is now as secure as your login is - if anyone was to know the token necessary for accessing the api, they would also be logged into the site and have access to all the information anyway. Best bit is, if you've already implemented login, you've not really had to do any more work.
The point of systems such as OAuth is to provide this 'logging in' method, usually from a third party application and as a developer. This would potentially be a good solution for an iPhone app or similar, but that's in the future. Nothing wrong with the API accepting more than one authentication method!
The answers so far do a great job of explaining, but don't give any actual steps. I came across this blog post that goes into great detail about how to create and manage tokens securely with Node + Passport.
http://aleksandrov.ws/2013/09/12/restful-api-with-nodejs-plus-mongodb/
Tips valid for securing any web application
If you want to secure your application, then you should definitely start by using HTTPS instead of HTTP, this ensures a creating secure channel between you & the users that will prevent sniffing the data sent back & forth to the users & will help keep the data exchanged confidential.
You can use JWTs (JSON Web Tokens) to secure RESTful APIs, this has many benefits when compared to the server-side sessions, the benefits are mainly:
1- More scalable, as your API servers will not have to maintain sessions for each user (which can be a big burden when you have many sessions)
2- JWTs are self contained & have the claims which define the user role for example & what he can access & issued at date & expiry date (after which JWT won't be valid)
3- Easier to handle across load-balancers & if you have multiple API servers as you won't have to share session data nor configure server to route the session to same server, whenever a request with a JWT hit any server it can be authenticated & authorized
4- Less pressure on your DB as well as you won't have to constantly store & retrieve session id & data for each request
5- The JWTs can't be tampered with if you use a strong key to sign the JWT, so you can trust the claims in the JWT that is sent with the request without having to check the user session & whether he is authorized or not, you can just check the JWT & then you are all set to know who & what this user can do.
Node.js specific libraries to implement JWTs:
Many libraries provide easy ways to create & validate JWTs, for example: in node.js one of the most popular is jsonwebtoken, also for validating the JWTs you can use the same library or use express-jwt or koa-jwt (if you are using express/koa)
Since REST APIs generally aims to keep the server stateless, so JWTs are more compatible with that concept as each request is sent with Authorization token that is self contained (JWT) without the server having to keep track of user session compared to sessions which make the server stateful so that it remembers the user & his role, however, sessions are also widely used & have their pros, which you can search for if you want.
One important thing to note is that you have to securely deliver the JWT to the client using HTTPS & save it in a secure place (for example in local storage).
You can learn more about JWTs from this link

Categories

Resources