Develop Reference

Storing and updating API Data in MongoDB

I am working on a web app where I have made a call to an API and stored the data using MongoDB. This data gets updated daily so I will need to be able to update the data daily by clicking a button in admin site. What is the best way to approach this?
I am new to using databases so I do not know the best approach. The reason I am wanting to store data in database is so I can store it using Redux or Context API so when someone goes to a page the data will be available faster instead of having to make a new API call (and wasting an API call) every time someone visits a page.
My database contains about 630 documents at a time.
Issue:
I need to update the 630 documents in my database to match the 630 documents coming from API that changes daily so I need to figure out what to query MongoDB to accomplish this.

You can use node-schedule.
It's very much like cron-job. But runs on the node application. Make sure the scheduler runs every 24 hours interval and put this database oprtation there.
Note that Node Schedule is designed for in-process scheduling, i.e.
scheduled jobs will only fire as long as your script is running, and
the schedule will disappear when execution completes. If you need to
schedule jobs that will persist even when your script isn't running,
consider using actual cron.

I ended up finding a way to solve my issue.
Previously I have updated a document one at a time so I used something along the lines of db.collection.update(<query>,<update>) to update one document. Issue I was facing was I needed to update all the documents in collection at one time. So by using db.collection.remove({}) I was able to remove all the documents in collection then used db.collection.create(myData) to add new updated data.

Parse-Server Make HTTP Get Request in Cloud Code

I was wondering what the best way to go about this would be and if this is even possible.
Basically I'm using Parse Server for my backend and I want to do query an API every 5 minutes for new data to populate my tables with.
I was thinking of having a cloud code function that the client could call and it would check the last updated time and if greater than 5 minutes query the API to get new data and populate the tables.
However I'm not sure it's possible to do a HTTP GET request in cloud code, if it is I'm not sure how to do it.
Also I'm wondering if this is the best way to go about solving my problem at hand? If not what would be a better alternative? The API query is very quick and returning some basic JSON data.

The simplest solution would be to create a Parse Cloud Function that makes call to external API and call it every 5 minutes from the client.
It is possible to make other API calls from the Parse Cloud code use Parse.Cloud.httpRequest or any other npm packages. Since the cloud code runs on the server, you can use any node packages that are not only supported by the browser.
Another approach would be to create a Parse Cloud Job that makes external API call, have it run in the server at the interval of 5 minutes and update a Parse table with the obtained data. Then you can use Live Query to get the up-to-date data from the Parse API on the client side.

You might be able to find a node module that is a wrapper for the API you're trying to use, even if it isn't an official one.
You'll have to set up what's called a 'cron job' to run your background job / call your cloud function that refreshes the data. But you might also be able to set up webhooks for this other service, so anytime it receives updates to information, it'll trigger this webhook on your server and you can add data in real time instead of by time intervals.
What's the API?

AngularJS and MySQL real-time communication

I have built a web application using AngularJS (front-end) and PHP/MySQL (back-end).
I was wondering if there is a way to "watch" the MySQL database (without Node.js), so if one user adds some data to it, the changes are synced to other users too.
E.g. I know Firebase does that, but it's object oriented database and I am unable to do the advanced queries there like I do with SQL.
I was thinking to use $interval and $http and do ajax requests, so that way I could detect changes in the database. Well, that's possible, but it'll then do thousands of http requests to the server everyday and plus interpret php on each request.
I believe nothing is impossible, I just need an idea to do this, which I don't have, so that's why I am asking for a help here.

If you want a form of "real-time communication" you'll likely have to incorporate some form of long-polling from the client. Unless you use web sockets, but that's a big post about a bunch of different things. You're right to be concerned about bandwidth and demand on the DB though. So here's my suggestion:
If you don't have experience with web sockets then log your events in a separate table/view and use the pub/sub method to subscribe entities to an event, and broadcast that event to the table. Then long-poll against the watcher view to see when changes may have occurred. If one did occur then you query for the exact value.
Another option would be to use some query system with "deciders" that hold messages. Take a look at Amazon's SQS platform for a better explanation of how this could work. Basically you have a queue that holds messages and a decider chooses where to store the message using some hash or sorting method (to reduce run time). When the client requests an update, the decider finds any messages that would apply based on the hash/sort and returns them. Then you just have to decide how and when to destruct the messages.
The second option would require a lot more tinkering though, so it's really about your preference. I think what you'll find the difficulty to be is that most solutions have to deal with the fact that the message has to be delivered 1 or More times and you'll need to track when someone received the message and if it can now be deleted from the queue/event table or if you still need to wait. Otherwise you'll consume a lot of memory.

Building my own database of Facebook posts by calling Graph API via Server?

I want to build my own database of Facebook posts by periodically calling the Facebook Graph API and saving the results. User would then communicate with my own database instead of directly with Facebook.
I know that the API calls require an Access token that is generated from your Facebook login. From what I understand, this means the user logging in on the clientside would be using their own access token to make the calls. However, I want to make the calls from the server, which means using my own access token.
To illustrate the process flow:
*SERVER*
myFBAccessToken ---(API call every 15 mins)---> Facebook ---(returns)---> Fb posts ---(save to)---> myDatabase
*CLIENT*
viewFbPosts ---(db call)---> myDatabase
My questions are:
----------------------
1. Is it possible to use a single access token to regularly call the API from server? (Every 15 mins)
2. Will doing so violate any usage limitations on how frequently you can call the API?
3. Does Facebook allow for storing of their content on external databases?
Alternatively, if this is not recommended, does anyone know of a way to get more than the latest 25 posts from the facebook /feed?
I am using MEAN stack (mongodb, expressjs, angularjs, nodejs) with asynchronous functions.

Yes, you can use the same token for the same user multiple times. However, once it expires, you will have to re-login your user again to get a new access token.
There is not an official limitation of number of queries that you are sending to graph API. However, being involved in this sphere for a long time, I found out that 1 query per 1 second is workable for a single user. If you try to exceed it, you will most probably get JSON with error.
You do not have to notify facebook that you are going to store its data in external database. You simply get permitted information using graph API and, afterwards, it is totally up to you what you are going to do with the data. Facebook is responsible for flow of the data from their servers and making sure that you are the person who has a right to get that information on a legal basis.

Caching JavaScript API Calls

I'm querying the GitHub API from the client using JavaScript (on this page).
There are 14 API calls each time the page loads, which means I will end up hitting GitHub's API rate limit of 5000 calls per hour pretty fast.
Most caching strategies I've seen assume that you have access to a server, but in my case I'm running a purely static Middleman site.
So my question is this: how can I cache API requests from the client? Are there third-party apps that provide this service?
(Note that my use case is many different clients hitting the page (e.g. it has been linked from Hacker News), not a single client refreshing. So local caching wouldn't really help much. )

Agreed with Firebase or separate data store alternative from #David so you can create a persistent cache mechanism since you don't have access to the server where the application sits. It's basically another data store and you can update your logic in Middleman to either make a fresh call to the Github api or to pull from data saved in Firebase based on some checks you do when a person visits that Translation page. Check out the logic here

You can cache a single client's page by using local storage or a cookie. This way if the user refreshes, you can have logic to see if you want to query the API again. This would be fine if your user base was small.
This type of caching is typically done on the server since you are limiting yourself to ~357 users per hour at best.
To cache on the client side, store the data in local storage and log the time of the query. Then decide on an interval (let's say 5 minutes). Then prior to any refresh or page load, look a the users local storage and see if the query was within the last 5 minutes. If it was, read from the local storage. If not, then query the API again. This only applies to each user but by querying every 5 minutes, it would allow you to say ~30 users per hour.
http://diveintohtml5.info/storage.html

No server, eh? You could use something like Parse. Make a Parse object, set the key to the particular GitHub API URI, and set the value to something like this:
{
stored: <Date>,
value: <stringified JSON returned from GitHub API call>
}
Then when someone hits your client, first call Parse to see if you already have a cached version for that particular API call. If you don't, make the call to GitHub's API and then store the results on Parse (with stored set to the current DateTime so you can check for staleness later).
If Parse does have a cached version stored, check the stored value to see how old it is - if it is stale, make a fresh call to GitHub, and store the results back into Parse. Otherwise, just parse the JSON string from value and you're good to go.
This is assuming that you want individual caching control over the 14 GitHub API calls. If you don't, then just store the compiled calls into one object on Parse under a key like cache.