I'm making interface based on 3rd party api. My app works as data analyzer, and to do that i need to download lots of data and this data keeps updating.
But api has limits, and i can't have user download entire set of data at each session. That would not scale.
So i'm thinking simplest way is to let user sync when he wish to, and save the data in local storage.
However, i don't want any user on that computer to have access to all data.
Therefore, i've come up with this scheme that will let the data be safe.
send javascript via https.
using js do oauth and retrieves data from api servers.
get opensource library like cryptojs also using https.
Encrypts data using user's (secret key + salt) and then save both in storage. So even if someone sees it then he can't understand it.
Then at every new session take stored salt and secret key to match hash(es) and get that data. Or simply start new auth process from step 2.
if hashed data is older than 30 days delete it. (which could mean user no longer uses the computer or has forgotten password, either way data should be outdated and will need to be resynced by downloading all again)
Related
Disclaimer: I'm finding my way and not sure how to ask the question. This is what I want to do:
I am getting data from twitter API in my app (this is working).
I then want to store that data (as an array), and serve it to whichever user is accessing the app (so that I don't need to query the API everytime, just poll every 10 mins).
What do I need to be able to do this? (external database? or can I just save to a file on the server in someway? or something else)
For ref I'm building with sveltekit, and deploying with vercel.
If you are using Twitter's API directly within your own app, every user of your app needs to query the API at least once to get some data. You cannot serve the results that are returned to one user to other users without having your own back-end server and handling this accordingly. However, you can save a copy of the data returned to each user to that user's localStorage so that specific user does not have to query the API every time.
You can save the data on the client's localStorage and save an expiry timestamp that allows you to query the API again after the timestamp has passed.
Here is a tutorial on how to use localStorage with SvelteKit
I'm completing an unfinished project someone else worked on and trying work out how to create a secure page for an API driven front-end.
When a user logs in successfully, a local storage variable is being created that contains user information, including user token and user secret.
I require a secure dashboard page that calls secure API's.
Am I correct in the following approach :
When secure page loads, a JS routine is executed which checks local storage for user token. Can this be a simple check for user token existence ?
If token present then the secure API's are called using the secret key. The api then returns sensitive data to populate table.
Will this work?
UPDATE:
Both server and client will run under https. As data in encrypted, secret token can be stored on client. Front-End is static html/JS making API calls for sensitive data using secret (only available to authenticated user). None of the user data is hardcoded to F/E but instead is referenced from local storage. Then tokens can be used securely to make further API calls as required for sensitive data. So basically, no-one else should be able to get to sensitive data as cookie/storage is limited to client machine and will expire anyway.
I was looking for a blog/tutorial to confirm my understanding as above.
Thanks
This approach seems OK. I don't know your exact requirements, but I would suggest using a cookie instead of localstorage, given that the token is sensitive information and should not be stored for a long time if it doesn't have to.
If the user has it's personal permanent access token go with localstorage. If the token is fetched from an auth-server upon login, use cookies instead.
I am creating a web application, and need to store some user data (like their favorites history in the app). I had considered using IndexedDB for this purpose, as it would be a little easier to implement than on the server side where I am using MySQL. But, I do want to run analytics against this user data, such as to ask what are the most popular pieces of content that users are saving to their favorites list. If I use IndexedDB, I could use some Javascript on the client side to occasionally forward user data from each IndexedDB store so that it could be analyzed. My question is: would this be scalable for > 25,000 users? Or, am I just asking for trouble with this kind of an approach, in which case I should just store all the user data in MySQL on the server side to make analytics easier?
Those are two different concepts as i understand your question.
indexDB is store locally on the user browser and it's subject to local user behaviors - clean cach , cookies refresh and browser options that are enable.
You should use the indexDB if your site/app load some data to the user browser - like a table/graphdata/friendlist etc. and you want to allow quick navigation with no server calls for some queries sending back to the server.
with this in mind you can store different user sessions and build a small key/value data base and send this when you want to you Server MySQL DB and save this on your side and allow you user to navigate this data or to use the JS API in order to load the right content for your user.
MySQL DB on your server side can do the same but you got some more latency parameters in here - user location vs Server Location , DNS ,network latency , server CPU , Disk , Table structure and Table size that can slow you down.
You should decide what you need in order to learn more on your users on the server side and which data your users need for quick and easy use with your APP . Those 2 data flows are not tied up and you can run them with no code blocking using asynchronies JS commands.
Regarding the scale Issue , IndexDB is good for 1-2MB of data and you should think on a cleansing process or counting main features on the App if you will have 1 key of favorite site . then you just need to add +1 for this key value - sitename_favorite - 5
this data is per user only and the load will be on your MySql server that will get updates from time to time for the users. the IndexDB will be managed by the local machines of the users so 25K users is not relevant for the indexDB part, the load will be on the MySQL server once users updates arrive.
I'm querying the GitHub API from the client using JavaScript (on this page).
There are 14 API calls each time the page loads, which means I will end up hitting GitHub's API rate limit of 5000 calls per hour pretty fast.
Most caching strategies I've seen assume that you have access to a server, but in my case I'm running a purely static Middleman site.
So my question is this: how can I cache API requests from the client? Are there third-party apps that provide this service?
(Note that my use case is many different clients hitting the page (e.g. it has been linked from Hacker News), not a single client refreshing. So local caching wouldn't really help much. )
Agreed with Firebase or separate data store alternative from #David so you can create a persistent cache mechanism since you don't have access to the server where the application sits. It's basically another data store and you can update your logic in Middleman to either make a fresh call to the Github api or to pull from data saved in Firebase based on some checks you do when a person visits that Translation page. Check out the logic here
You can cache a single client's page by using local storage or a cookie. This way if the user refreshes, you can have logic to see if you want to query the API again. This would be fine if your user base was small.
This type of caching is typically done on the server since you are limiting yourself to ~357 users per hour at best.
To cache on the client side, store the data in local storage and log the time of the query. Then decide on an interval (let's say 5 minutes). Then prior to any refresh or page load, look a the users local storage and see if the query was within the last 5 minutes. If it was, read from the local storage. If not, then query the API again. This only applies to each user but by querying every 5 minutes, it would allow you to say ~30 users per hour.
http://diveintohtml5.info/storage.html
No server, eh? You could use something like Parse. Make a Parse object, set the key to the particular GitHub API URI, and set the value to something like this:
{
stored: <Date>,
value: <stringified JSON returned from GitHub API call>
}
Then when someone hits your client, first call Parse to see if you already have a cached version for that particular API call. If you don't, make the call to GitHub's API and then store the results on Parse (with stored set to the current DateTime so you can check for staleness later).
If Parse does have a cached version stored, check the stored value to see how old it is - if it is stale, make a fresh call to GitHub, and store the results back into Parse. Otherwise, just parse the JSON string from value and you're good to go.
This is assuming that you want individual caching control over the 14 GitHub API calls. If you don't, then just store the compiled calls into one object on Parse under a key like cache.
I would like to import data that the user had entered into his profile on a website that I do not control. I don't want the user to hand me his login credentials to grab the data from the server-side (connecting directly to aforementioned website). I would rather prefer the login data to stay on the client's machine: It makes my service more trustworthy and I don't have to process sensitive data.
I thought that this can probably done with javascript without greater hassle. However, given the security risks, it seems to be blocked by browsers. See How to get data with JavaScript from another server?
So I think my question is already answered and can be closed/deleted.
I'm not sure I understand what you're trying to do, but there is no secure way to verify login credentials in a browser client. If you want to check login credentials, you will have to involve a server.
Some data can be stored on the client side, but then a user is tied to a particular browser on a particular computer and can't use the service from anywhere else. In older browsers data is generally limited to what can be stored in a cookie and cookies are not guaranteed to survive for any particular long lasting time frame. In the latest browsers, data can be stored in HTML5 local storage which allows for a little more structured way of storing data, but even then you're still stuck in one particular browser on one particular computer.
Based on your comments, it sounds you're trying to "cache" a copy of the data from web-site A that you can access from client-side code on web-site B on multiple visits to your website. If that's the case, then it sounds like HTML5 local storage may work to serve as a cache storage mechanism. You will have to figure out how to get the data from web-site A into the cache as the cache will be subject to same-origin access (domain X can only access the data that was put into HTML5 local storage by domain X), but if you can get manage to get the data from web-site A into your web-site B client-side page (perhaps using JSONP), then you could cache it using HTML5 local storage. You will need to realize that even HTML5 local storage is not guaranteed forever (so you need to be able to fetch it again from web-site A if required).
You said this
I don't want the user to hand me his login credentials to grab the
data from the server-side (connecting directly to aforementioned
website).
If you do that, anyone would be able to access any User's data, since you don't restrict access to data.
You also said this
I would rather prefer the login data to stay on the client's machine:
It makes my service more trustworthy and I don't have to process
sensitive data.
I'm really not sure that's a good idea. You still need to lock down personal information. But anyway, if you really want to, you can use localstorage -- modern webbrowsers support this.
Check out this link for a primer on local storage.
Storing Objects in HTML5 localStorage
Note that the user can clear the browsers local storage, so you still need to have a form to enter credentials.
EDIT -- if you want to save a user's profile information on the client, you can do that with local storage. But you still need to save the data to the server, else if the user goes to a different machine or even browser, they won't have their data. Plus, your server side model probably needs to associate a user's content with their profile in some way. I don't think there is any way around it.