I have a restaurant locater web application that mashes up the location of restaurants to a Google Maps.
I use JQuery sliders to limit the amount of restaurant to show on the map by having Search filter such as: price, type of food, locale.
These JQuery sliders call back via AJAX to an API I created to update the map without the web page having to refresh.
JQuery calls a RESTFUL API like so:
http://example.com/search/?city=NYC&max-price:50&cuisine=french
This returns a JSON string of restaurants which match this criteria so that my web application can display on the map all the restaurants which match the search.
What I don't want to have happen is for someone to come along and figure out my API and dumps out ALL of my restaurant listings.
Is there a way that I can limit who call the above HTTP API, so that only my web server calls the URL and not spamer/hackers looking to dump my database?
Thanks
First, declare your intentions in robots.txt.
Then, send a Set-Cookie header with a nonce or some kind of unique ID on the main page, but not on your API responses. If the cookie is never sent to your API endpoint, return a 401 Bad Request response, because it's a bot, a very broken browser, or somebody is rejecting your cookies. The Referer header can also be used as an additional check, but it's trivial to fake. Keep track of how many API calls have been made by that ID. You may also want to match IDs to IP addresses. If it goes above your threshold, spit back a 403 Forbidden response. Make your threshold high enough that legitimate users don't get caught by it.
Keep good logs, and highlight 401 and 403 responses.
Realistically, if someone is determined enough, they WILL be able to dump this information. Your goal shouldn't be to make this impossible, because you will never succeed. (See all the usual adages about achieving perfect security.) Instead, you want to make it abundantly clear that:
This behavior violates the terms of service.
You are actively trying to prevent this.
You know that the offender exists and roughly who they are.
Scary lawyers might start getting involved if this continues.
(You do have a lawyer, right?)
To achieve this, be sure the body of your 403 Forbidden response conveys a scary sounding message along the lines of "This request exceeds the maximum allowed usage of the API. Your IP address has been logged. Please refer to the terms of service and obey the directives in robots.txt."
IANAL, but I believe the DMCA can be made to apply in this situation if you claim copyright on your database. This essentially means that if you can track illegal usage of your API to an IP address, you can send a nastygram to their ISP. This should always be a last resort of course.
I don't encourage the use of assigned API keys/tokens because they turn out to be a barrier to adoption and kind of a pain in the neck to manage. As a counter-point to #womp's answer, Google is slowly moving away from their use. Also, I don't think they actually apply in this case, because it sounds like your "API" is more like a JSON call that's used mainly on your own site.
All the big REST API's tend to use tokenized authentication - basically before you do a REST request, you have to send some other request to the token service to fetch a token to include with your data request. Bing Maps does this, Amazon does this, Flickr does this... etc.
I don't know too much about it other than having worked with Bing Maps. You'll need to read up on tokenized authentication with REST. Here's a blog post to get you started: http://www.naildrivin5.com/daveblog5000/?p=35
Related
I've made a simple webapp that is going to show some data in a table, which will be updated weekly.
This update it done in the backend with some python code, that scrapes and alters some data, before putting it in a SQLite database.
After doing some reading I learned that to deliver that data to my webapp I should make a API with Flask, that can take that data and deliver it to the JS in my webapp in form of JSON, which then can use the data to populate the table. However, I should secure my API with username and pw. But as its a JS frontend that will retrieve data from the API, there is really no point, as the username and pw will have to be hardcoded into JS, which then can be read by the users. (I think)
Should I expose my API to everyone, or is this not the way to go to be able to use SQLite data as a backend for my webapp? I am fine keeping the API to a GET only.
You are correct, it is pointless for you to secure your API. Securing an API is only needed in certain circumstances.
If you are accessing data that you don't want anybody to see, perhaps through a backend call, then it would make sense to add in some form of security (normally an API key or Authorisation tokens in your request headers).
However, if you are making calls from your front-end (i.e. client side) to a backend API, then there is no point putting additional security there as the user can already see the request and already has access to the data the API is returning - so by securing it you are achieving nothing.
Normally, if the page the user is visiting contains sensitive data that you don't want everyone to see, you would take steps to secure your website instead (for example protecting it with a login for username and password before you can access that page). If you were to take this approach, where the website is protected by username and password, then you can update the API to make sure it does not respond to requests where the user is not authenticated (e.g. by generating a session token or something unique for each logged in user).
If you have a look around on websites that have lots of free data available, you will find they all have front end API calls that are completely unsecured (because it is pointless if the data is already free to access). Some websites do take steps to try to make sure it is their own website that is calling the API, but even then it is a bit pointless as web scrapers can always extract the data from the HTML.
Take a look at this page which outlines authentication headers. This simpler route is to hard code the header info in Flask to make it a little more secure. You could also try the more involved route of reading header info from your db. What's currently working for me to read from postgres db is below so you may modify it slightly for sqlite.
def valid():
headers = request.headers
auth = headers.get("X-Api-Key")
user = User.query.filter_by(apikey=auth).first_or_404()
print('from search of db ',user,'',auth)
return str(user)
As you mentioned, you plan to show a public data - then it can be used even
without authentication. Otherwise I think it can take too much unnecessary time spent on that.
As you have just a simple and single table from database, I believe that you don't need an API. You can just create HTML template and render it with data. Some examples can be found here and few more here.
Just recently I was wondering, how Google manages it with analytics to track the user but avoid any fraud (as kinda everyone could insert that JS into an HTML page).
Can anyone tell me, if they just resolve the Domain on each incoming request and validate if it corresponds to the correct account and comes from a allowed server (by looking up the IP)? If that's not the case, can anyone provide me some Idea on how Google could do this?
Google does not, by and large, avoid fraud at all. They offer an option to automatically filter for known bots and spiders, and apparently they intervened in some fashion against referral spam (which used to be a huge problem but has by now almost completely disappeared at least from the accounts I have access to).
Other than that, as you say, anyone can stick the javascript code into his HTML and send data to your account, or use the measurement protocol and not even bother with JS. Checking server IPs would hardly help (relevant here is the IP of the client and there are too many possible IPs for meaningful filtering).
Of course if you notice wrong data that follows a discernible pattern you can create filters yourself, which is at least gives you some control over what data is removed.
I am trying to send transaction data to UA from a webshop which is only creating orders once it has received a "push" from a payment processing API. A success page is displayed to the customer independently from the order being created, meaning the tracking script (UA via GTM with data layer) does not have access to the order data.
This gives me the headache of trying to track transactions made through the webshop server side, instead of client side. Yet I still want to tie the transaction to the unique visitor ID who made the purchase, not just a random Client ID each time.
My site is using analyics.js (UA profile). Therefore I thought somewhere a client ID would be stored as a cookie, and I do see a _ga cookie there which I believe is the client ID, which looks like this:
GA1.2.1586737968.1429871710
The documentation for getting cookie and user identification states the following:
You should not directly access the cookie analytics.js sets, as the cookie format might change in the future. Instead, developers should use the readyCallback to wait until analytics.js is loaded, and then get the clientId value stored on the tracker.
... which is not helpful to me, as I have to do this server side. But anyway, this client ID does not even fit the description of what a client ID looks like, although it does appear to match a legacy format. Sort of.
Couple of questions then:
Is it just those last two numbers 1586737968.1429871710 that I need to be parsing from the _ga cookie and sending as a client ID? Or is the whole thing ok?
Are there any scripts/libraries that will do this for me so I don't have have to worry about Google suddenly giving new visitors the client ID based on the new UUID v4 format?
Does this approach have any obvious flaws?
Answers:
Yes the cid value should just be the last 2 numbers as you rightly point out. (this is experience from client setups that already successfully use Measurement Protocol on an ongoing basis)
Nope, not as far as I am aware, best to do it yourself. It will only be a few lines of code to detect what format you're seeing in the cookie and what, therefore you need to send through.
not really. It is a common enough scenario.
Circumstances
I develope a WebApp with AngularJS.
I've an restful API on server-side with GET and POST commands.
I want to use the API within my module (means: in JavaScript) to display and edit my data.
I want to protect the API with some kind of authentication (basic auth with an API key for example)
I don't want to protect the API when a user uses the app itself
Actual question
Okay, I guess the last point is a bit unclear.
I want that a user can use the app with his browser without any authentication
But when a third-party app want to access the API it have to use authentication
Since JavaScript is executed on client-side of course I can't write a master key into js or something similar..
Is there any kind of pattern or solution to solve this problem?
More specifications
referring to #EliranMalka and #shaunhusain
On the server-side I do use Tornado with it's builtin template engine. I do use the template engine actually just to write the index page and insert CSS, JS dynamically.
The code for authentication would just something like:
def is_authenticated(request):
if 'api_key' in request.arguments:
return sql('SELECT id FROM keys WHERE key=%S' % request.arguments['api_key']).count == 1
My AngularJS module is doing something similar to:
$http.get('/api/foo?api_key=1234')
.then(function (result) {
$scope.data = result.data
});
As you can see I'm writing my API key into js at the moment. But I wan't to avoid this.
Also, what do you mean exactly by third-party?
not a third-party request would be: Using the App on http:/ /app.example.com with a browser
A third-party request would be from an Android app for example. Something that comes from outside or remote.
A JS request from the browser on the actual page would be not from remote (again: since it's JS it is actually from remote - but I hope it gets more clear now)
Oh and before I forget...
I'm aware of that my plan is a bit weird - but it's just a learning(-web-development)-by-doing project.
Also the API key is not absolutely to avoid abusion, it is rather to log 3rd-party usage.
PS I hope my question was clear for you
Hmm, well I'll try to address the questions but here's a few things.
Question isn't really appropriate in it's current format for stackoverflow.com (should be programming questions, I tried X and Y happened) perhaps closer to a StackExchange question but is still fairly open ended.
Include more information about specifics of the languages (and/or frameworks) your using server side and any code you have that is relevant (authentication code?).
Putting the key into the client code and transmitting it from the client means anyone with a web proxy (check out Charles or Wireshark) can grab the key so just to reiterate you're right there that's not the way to go.
Check out how other organizations allow you to get access to their APIs (for example Google, LinkedIn, Facebook, Twitter) to get a feel for how it works. In all of these cases you are signed into the service to be able to make an API key, in some cases you have to specify which domain the requests with that API key will come from. If you use the same precautions and check the API key sent with a request against a database of registered API users and verify the domain in the request then I'd say you're in pretty good shape.
For my CMS component I'm implementing integration with Twitter API to fetch and display list of tweets (either connected to user or search query). I'm using Twitter Restful API v1.1, since the 1.0 version is going to be dropped in two months. Two interesting requests for me are user_timeline and search one.
Since my technology strongly relies on caching I need to avoid server-side processing as much as possible providing static html and piece of javascript. I've done it already for old version API and it worked fine. New approach however requires providing authentication data via OAuth. One of the property (oath_signature) is a hash of other properties (in which there are oauth_timestamp and oath_nonce, which should (should they?) be unique per each twitter request) and secret keys, thus make it unsecure to generate it on client side.
Is there any secure way to get list of tweets on client-side using new API?
The simple answer to your question is, "no, there is no secure way of doing this without server-side code." What I would do is set up a service to poll Twitter every xxxx seconds and retrieve the desired tweets. You should cache or store the results and then empty them each time you make your next request. If you are using C#, I have been working on a C# Twitter library that replicates Twitter's API and already has support for grabbing a user's timeline. I will be adding support for filter and search within the next two days (each one should take no more than ten minutes to implement, excluding testing, if you decide to do it yourself). You can reference this library in the service that I mentioned before.
If you do not have the server resources that you need for this, then I strongly caution you against using solutions that circumvent Twitter's intended securities, as it could leave you or your client in a vulnerable position.
You'll have to write a proxy web service on your server side. And as you say caching will be critical to avoid the 15 requests per 15 minutes for basic stuff like pulling tweets.
Definitely avoid doing any auth stuff on the front end. The new "application only" auth using OAuth 2 would allow you to embed bearer tokens in JavaScript meaning you don't need to do any of the signature stuff you're talking about. But don't. Anyone could use your bearer token and if your own users didn't exhaust the rate limit, other people stealing your token might.
If you don't have the server side resources to do this yoursefl, you might want to look at Flamingo. It'll do the auth and the caching for you, so you only need to work in JS like you used to.