I've got a simple app that fetches a user's complete feed from the Facebook API in order to tally the number of words he or she has written total on the site.
After he or she authenticates, the page makes a Graph call to /me/feed?limit100 and counts the number of responses and their dates. If there is a "next" cursor in the response, it then pings that next URL, which looks something like this:
https://graph.facebook.com/[UID]/feed?limit=100&until=1386553333
And so on, recursively, until we reach the time that the user joined Facebook. The function looks like this:
var words = 0;
var posts = function(callback, url) {
url = url || '/me/posts?limit=100';
FB.api(url, function(response) {
if (response.data) {
response.data.forEach(function(status) {
if (status.message) {
words += status.message.split(/ /g).length;
}
});
}
if (response.paging && response.paging.next) {
posts(callback, response.paging.next);
} else {
alert("You wrote " + words + " on Facebook!");
}
});
}
This works just fine for people who have posts a total of up to 4,000 statuses, but it really starts to crawl for power users with 10,000 lifetime updates or more. Each response from the API is only about 25Kb, but I cannot figure out what's straining the most.
After I've added the number of words in each status to my total word count, do I need to specifically destroy the response object so as not to overload memory?
Alternatively, is the recursion depth a problem? we're realistically talking about a total of 100 calls to the API for power users. I've experimented with upping the limit on each call to fetch larger chunks, but it doesn't seem to make a huge difference.
Thanks.
So, you're doing this with the JS SDK I guess, which mean this runs in the Browser... Did you try to run this in Chrome and then watch the network monitor to see about the response time etc.?
With 100 requests, this also means that the data object/JSON must be about the size of 2.5mb, which for some browsers/machines could be quite challenging I guess. Also, it must take quite a while to fetch the data from FB. What does the user see in the meantime?
Did you think of implementing this in the backend on the server side, and then just passing the results to the frontend?
For exmple use NodeJS together with SocketIO to do it on the server side and dynamically update the word count?
Related
I'm trying to overcome Call stack size exceeded error but with no luck,
Goal is to re-run the GET request as long as I get music in type field.
//tech: node.js + mongoose
//import components
const https = require('https');
const options = new URL('https://www.boredapi.com/api/activity');
//obtain data using GET
https.get(options, (response) => {
//console.log('statusCode:', response.statusCode);
//console.log('headers:', response.headers);
response.on('data', (data) => {
//process.stdout.write(data);
apiResult = JSON.parse(data);
apiResultType = apiResult.type;
returnDataOutside(data);
});
})
.on('error', (error) => {
console.error(error);
});
function returnDataOutside(data){
apiResultType;
if (apiResultType == 'music') {
console.log(apiResult);
} else {
returnDataOutside(data);
console.log(apiResult); //Maximum call stack size exceeded
};
};
Your function returnDataOutside() is calling itself recursively. If it doesn't gets an apiResultType of 'music' on the first time, then it just keeps calling itself deeper and deeper until the stack overflows with no chance of ever getting the music type because you're just calling it with the same data over and over.
It appears that you want to rerun the GET request when you don't have music type, but your code is not doing that - it's just calling your response function over and over. So, instead, you need to put the code that makes the GET request into a function and call that new function that actually makes a fresh GET request when the apiResultType isn't what you want.
In addition, you shouldn't code something like this that keeping going forever hammering some server. You should have either a maximum number of times you try or a timer back-off or both.
And, you can't just assume that response.on('data', ...) contains a perfectly formed piece of JSON. If the data is anything but very small, then the data may arrive in any arbitrary sized chunks. It make take multiple data events to get your entire payload. And, this may work on fast networks, but not on slow networks or through some proxies, but not others. Instead, you have to accumulate the data from the entire response (all the data events that occur) concatenated together and then process that final result on the end event.
While, you can code the plain https.get() to collect all the results for you (there's an example of that right in the doc here), it's a lot easier to just use a higher level library that brings support for a bunch of useful things.
My favorite library to use in this regard is got(), but there's a list of alternatives here and you can find the one you like. Not only do these libraries accumulate the entire request for you with you writing any extra code, but they are promise-based which makes the asynchronous coding easier and they also automatically check status code results for you, follow redirects, etc... - many things you would want an http request library to "just handle" for you.
I was thinking about how to make an instant messaging application, and wanted to not have to send an AJAX request so often (one every .2s), and I came across the following idea:
Send an AJAX request from the user side, to the server.
Only respond once there is a change in the MySQL database
And then send the next AJAX request once the response has been recorded and parsed
I'm aware of how to do the first and third steps, but the second one is going over my head.
I'm assuming that for step 2, I'll need to store the request somewhere, while the PHP script is continuously looping and looking for some changes, and once there is a change, the saved request would be responded to.
EDIT
Didn't know about WebSockets, should've used those.
You could use recursion and query the database every 2 seconds, until you find new data to be served to the user. So basically you could do something like
public function isDataUpdated($lastId) {
$query = "SELECT * FROM `messages` WHERE `messages`.`message_id` > $lastId";
return (bool)(count($this->executeSQL($query)) > 0);
}
public function fetchNewMessages () {
if ($this->isDataUpdated($_GET['last_id'])) {
/* We have new data! Send it to the user */
} else {
sleep(2); // wait for 2 seconds
$this->fetchNewMessages(); // we use recursion to query the database every 2 seconds to find new data
}
}
Although, it is not the best of solutions, it would hopefully work. I would recommend taking a look at Sockets in PHP to better achieve what you want
I'm developing a facebook app which searches for facebook events near your position.The only way to do so is to search for all the places id's in your zone and then for each of those check if there is an event today.The problem I have is that the computation takes like 1-1:30 min which is kinda long. This is the code I use(might not be the best, I know):
foreach (var item in allPlacesIds)
{
RunOnUiThread (() =>loading.Text = string.Format ("Loading {0} possible events out of {1}",count,allPlacesIds.Count));
string query = string.Format ("{0}?&fields=id,name,events.fields(id,name,description,start_time,attending_count,declined_count,maybe_count,noreply_count).since({1}).until({2})", item,dateNow,dateTomorrow);
JsonObject result=(JsonObject)fb.Get (query, null);
try
{
JsonArray allEvents= (JsonArray)((JsonObject) result ["events"])["data"];
foreach (var events in allEvents)
{
Events theEvent= new Events(((JsonObject)events) ["id"].ToString(),
((JsonObject)events) ["name"].ToString(),
((JsonObject)events) ["description"].ToString(),
((JsonObject)events) ["start_time"].ToString(),
int.Parse(((JsonObject)events) ["attending_count"].ToString()),
int.Parse(((JsonObject)events) ["declined_count"].ToString()),
int.Parse(((JsonObject)events) ["maybe_count"].ToString()),
int.Parse(((JsonObject)events) ["noreply_count"].ToString()));
todaysEvents.Add(theEvent);
}
}
catch(Exception ex)
{
}
count++;
}
Where the try starts I used to have an if but that made it take even longer so I replaced it with a try block, as the result comes as null.
I know this isn't exactly a technical issue but I felt maybe you guys know a faster and better implementation of this, my only other option is to create and host a web service and use that just to interrogate data. the problem with that is that I need to invest a lot of money into a server/real ip/ and then I need to create a scheduled job to update the data daily.
Each API call takes some time, the only way to make it faster is to use Batch Requests. Here´s the documentation about those: https://developers.facebook.com/docs/graph-api/making-multiple-requests
Keep in mind that this will not count as one API call, it´s still the same amount, so be careful with API limits.
Doing some experimenting to get a grasp of the basics on my road to learning how to code. Been messing around with one aspect all day but can't figure it out. I'm sure it must be something simple and would appreciate any help!
So I've created a Mongo collection:
Posts = new Mongo.Collection('posts');
I then added 3 documents into the collection, with 2 fields each (name and quantity). Now, when in the Google Chrome console I run:
Posts.find().count();
//returns '3' as expected
However, when instead I run the following code from the server (is it called the server if it is the client side code?):
if (Meteor.isClient) {
var c = Posts.find().count();
console.log("The database documents count is " + c);
}
Then it comes up on Chrome console as:
The database documents count is 0
What gives? What's going on here?
Thanks in advance!!
How it works: when you open a page, the client has no data at all. After your JS is loaded, publish functions begin to work and after some time, when publishing is over, you have data on the client.
So at the moment when browser runs:
if (Meteor.isClient) {
console.log(Posts.find().count());
}
You have no data on the client, that's why you see 0 in the console.
But then you open console manually and write Posts.find().count(). It took some time for you to open the console, to write Posts.find().count() and this time was enough for publish functions to send all data to the client, so now you have data on the client and you will see 3 in the console.
You can try this experiment:
if (Meteor.isClient) {
setTimeout(function() {
console.log(Posts.find().count());
}, 2000);
}
It will log posts count not immediately, but with 2 seconds delay, this should be enough for publishing to be over and you will see 3 in the console. You can change timeout delay and find out how much does it take for the server to send posts data to the client.
i'm currently building a website which searches an external database and brings up records which match the given search string. The search is live, so results are brought up as the user types.
now the first (and current) approach i took, is that the page actually connects to the mySQL server and retrieves content via AJAX, with EVERY letter the user types in the search box.
now i am starting to look at JSON objects (i only very recently started building websites), and was wondering if it would be a good idea, to load the entire database into a JSON object in the beginning and then look through that when searching.
is this a good idea? would it be faster? thanks in advance
It totally depends on the size of the data and the complexity of the query. If you can reasonably send the data to the client in advance and then search it locally, then sure, that's useful because it's all local and you don't have the latency of querying the server. But if you have a large amount of data, or the query is complex, it may well make more sense to do the query on the server.
There's no one-size-fits-all solution, it's data-dependent.
...and retrieves content via AJAX, with EVERY letter the user types in the search box.
That's usually overkill. Normally, you want to wait until there's a pause in the user's typing before firing off the ajax call, so that if they type "james" in rapid succession, you search for "james" rather than searching for "j", then "ja", then "jam", then "jame", and then "james".
For instance, let's say your search trigger is a keypress event. This would be a fairly common approach:
var keypressTimer = 0;
function handleKeypress() {
if (keypressTimer) {
cancelTimeout(keypressTimer);
}
keypressTimer = setTimeout(doSearch, 100); // 100ms = 1/10th of a second
}
function doSearch() {
var searchValue;
keypressTimer = 0;
searchValue = /*...get the search value...*/;
doAjaxCallUsing(searchValue);
}
This is called "debouncing" the input (from hardware engineering, related to the mechanical and electrical "bouncing" of a key as it's pressed).