I'm trying to make an HTTP GET request using the jQuery get() function, but I'm having some trouble.
Here's what my code looks like:
// get the links on the page
var pageLinks = $.find('#pageLinks');
// loop through each of the links
$(pageLinks).find('a').each(function(){
if($(this).attr('title') !== "Next Page"){
// make a GET request to the URL of this link
$.get($(this).attr("href"), function(data) {
console.log("here");
var temp = parse_page(data);
// concatenate the return string with another
bdy = bdy+String(temp);
console.log("done");
});
}
});
There are multiple pages that I need to get data from. Since the get() function is asynchronous, I get the pages in a random order. Secondly, the concatenation does not work. Even though I get each of the pages, they're not put into bdy.
Can anyone suggest how I might deal with this?
Thanks a lot!!
Construct bdy after all pages are retrieved, i.e. store get results in a dictionary or array; wait for all gets to finish; then assemble them in the correct order.
I tried this one and it works:
// get the links on the page
var pageLinks = $('a');
var bdy
// loop through each of the links
$(pageLinks).each(function(){
console.log(this);
// make a GET request to the URL of this link
$.get($(this).attr("href"), function(data) {
// concatenate the return string with another
bdy = bdy + data.toString();
console.log(bdy);
});
});
As an example of what #muratgu has said:
var results = [];
var count = 0;
function allDone() {
var bdy = results.join("");
// do stuff with bdy
}
// get the links on the page
var pageLinks = $.find('#pageLinks');
// filter the links so we're left with the links we want
var wantedLinks = $(pageLinks).find('a').filter(function (idx) {
return $(this).attr('title') !== "Next Page";
});
// remember how many links we're working on
count = wantedLinks.length;
// loop through each of the links
wantedLinks.each(function (idx) {
// make a GET request to the URL of this link
$.get($(this).attr("href"), function (data) {
console.log("here");
var temp = parse_page(data);
results[idx] = temp;
// Decrement the count.
count--;
if (count === 0) {
// All done.
allDone();
}
});
});
You could go further and abstract this into a data type that can perform N async downloads, and then notify you when all are complete.
I just found that there are modules that allow one to manage the control flow in JS. The ones I found are:
Async
Step
For help using the above modules, see my follow up question here.
Related
I'm having issues retrieving data from an object using JS on my website. I have a third party scrape Instagram posts and provides JSON to my website via a link. I've managed to retrieve this data from the link and manipulate it, but the problem comes when I try to change the displayed image every 5 seconds.
I took the solution from How to change an image every 5 seconds for example? and tried to adapt for my solution, however, I get an error where posts[index] is undefined even though it shouldn't be.
posts = [];
let index = 0;
indexx = 0
$.getJSON('posts.json', function(data) {
$.each(data, function(i, f) {
posts[indexx] = f
indexx = indexx + 1
});
});
console.log(posts) // returns all the posts
window.onload = change();
function change() {
console.log(posts) // Returns the list of posts
console.log(posts[index]) // Returns 'undefined'
console.log(posts[1]) // Returns 'undefined'
$('#instaimg').attr('src', posts[index]["mediaUrl"])
if (index == 5) {
index = 0;
} else {
index++;
}
setTimeout(change, 5000);
}
I'm not sure if I am missing something or whether my lack of JS knowledge is to blame, but if anyone could help it would be appreciated
Several issues with your code:
Your console.log(posts) will show an empty array because the ajax callback has not finished yet => move that inside the .getJSON callback function
You call change recursively every 5 sec, e.g your call stack will grow indefinitely
Use setInterval instead of setTimeout
Start the interval timer inside the .getJSON callback function, so that it starts once the fetched data is ready
Use .push() to add to an array, no need to keep track of the index
Use $(function() { to make sure the DOM is ready before you do any action
You use a hardcoded length 4 for your data length, reference the array size instead
Updated code:
let index = 0;
let posts = [];
$(function() {
$.getJSON('posts.json', function(data) {
//$.each(data, function(i, f) {
// posts.push(f);
//});
// It looks like data is the array you want to use, so:
posts = data;
setInterval(changeImage, 5000);
});
});
function changeImage() {
$('#instaimg').attr('src', posts[index++]["mediaUrl"]);
if(index > posts.length) {
index = 0;
}
}
I made a js with data in a const array as below
const messages = [
{ date: '2020-1-1', content:'message1'},
]
In order to make my file cleaner I decide to put the data in a Json file and want to call the Data in my Js in order to use it like before.
my Json is like this
[
{
"date":"2020-1-1",
"content":"message1"
}
]
In order to import my Json I put this code:
let messages = [];
$.getJSON("messages.json", function(data) {
messages = data;
console.log(messages);
});
The result is that my array is loaded in the console but the variable dont work, I tried things with Object.keys but no more result. I dont use framework also and dont find a solution on other questions here. Any help will be appreciated. Thank you very much!
I dont use framework
You are using a library, though, $ === jQuery
in order to make my file cleaner I decide to put the data in a Json file
You can just define a constants.js file and load that before your other scripts.
For example,
constants.js
const messages = [
{ date: '2020-1-1', content:'message1'},
]
main.js
alert(messages);
index.html
<script src="constants.js"></script>
<script src="main.js"></script>
#Alvin Stefanus
Because x,z and messages were undefined and didnt let the code work, I also add
let messages,x,z = [];
And now it works perfectly with your solution.
I will use it as you told also for the other operations.
Thank you very much it helped for my problem and gave me a new technique.
EDIT:
I also tried to delete this part
var start = x;
$.getJSON('messages.json', function(data) {
messages = data;
}); <-- this
var end = z;
And it also works! That means that the problem was not the async function but because I didnt put the loop within the curly bracket of
$.getJSON('messages.json', function(data) {
//The data is finished being filled here
}
Ok this is probably the issue. $.getJSON() is an async function, the code will not wait until the closing curly bracket of the method:
var start = x;
$.getJSON('messages.json', function(data) {
messages = data;
}); <-- this
var end = z;
The code will run var end = z; before the $.getJSON() finished getting the result, because it is asynchronous function. In other word, when the code is currently at var end = z, $.getJSON() is still working to get the data, and has not been finished. That is why the messages = data is not being called yet.
So here is what you want to do:
$.getJSON('messages.json', function(data) {
messages = data;
for (const item of messages) {
if (item.date === todayDay) {
console.log(item.content);
var newPara = document.createElement("p");
var textNode = document.createTextNode(item.content);
newPara.appendChild(textNode);
var nodeParent = document.getElementById("titre");
var nodeChild = document.getElementById("child1");
nodeParent.appendChild(newPara, nodeChild);
}
}
});
Do all your needed operations within the curly bracket of
$.getJSON('messages.json', function(data) {
//The data is finished being filled here
}
This is a callback function. You can learn more about callback function here
Some Info
Yes the loop has to be inside the callback function to run after the data has been retrieved. To give you better understanding about the async function, you can also put your loop outside within a timeout function, but you will never want this, because you will not know how long the operation for retrieving the data will run.
For example:
$.getJSON('messages.json', function(data) {
messages = data;
});
setTimeout(function() {
for (const item of messages) {
if (item.date === todayDay) {
console.log(item.content);
var newPara = document.createElement("p");
var textNode = document.createTextNode(item.content);
newPara.appendChild(textNode);
var nodeParent = document.getElementById("titre");
var nodeChild = document.getElementById("child1");
nodeParent.appendChild(newPara, nodeChild);
}
}
}, 2000); //run after 2 seconds
Im sure the code above will also work, the process of getting the data should not be longer than 2 seconds.
Again this is not a correct way to do it, just to give you better understanding of async function.
Now I have a jQuery function for getting the list items from SharePoint List
function getListItems(listTitle, queryText){
var ctx = SP.ClientContext.get_current();
var splist = ctx.get_web().get_lists().getByTitle(listTitle);
var camlQuery = new SP.CamlQuery();
camlQuery.set_viewXml(queryText);
var listItems = splist.getItems(camlQuery);
ctx.load(listItems);
var d = $.Deferred();
ctx.executeQueryAsync(function() {
var result = listItems.get_data().map(function(i){
return i.get_fieldValues();
});
d.resolve(result);
},
function(sender,args){
d.reject(args);
});
return d.promise();
}
And then I call this function
getListItems(listname , "").done(function(listItems){
//do something here...
}).fail(function(error){console.log(error.get_message());}); // Error message
But one of the list contains quite a large amount of records and I want to show the progress to users so that they know what is going on. Is there a way to do this with just client side scripting? Any help is appreciated. Thank you.
Using the provided example you could display only indeterminate progress bar since the request is submitted once to the server and there is no way to determine current status complete.
But since SharePoint JSOM API supports paged data retrieval, you could consider the below approach that allows to determine current status complete and therefore display determinate progress bar.
function getPagedListItems(list, queryText,itemsCount,position){
itemsCount = itemsCount || 100;
var ctx = SP.ClientContext.get_current();
var list = ctx.get_web().get_lists().getByTitle(listTitle);
var ctx = list.get_context();
var camlQuery = new SP.CamlQuery();
if(typeof position != 'undefined')
camlQuery.set_listItemCollectionPosition(position);
var viewXml = String.format("<View>{0}<RowLimit>{1}</RowLimit></View>",queryText,itemsCount);
camlQuery.set_viewXml(viewXml);
var listItems = list.getItems(camlQuery);
ctx.load(list, 'ItemCount');
ctx.load(listItems);
var d = $.Deferred();
ctx.executeQueryAsync(function() {
d.resolve(listItems,list.get_itemCount());
},
function(sender,args){
d.reject(args);
});
return d.promise();
}
function getListItems(listTitle, queryText,itemsCount,position,results){
results = results || [];
return getPagedListItems(listTitle, queryText,itemsCount,position)
.then(function(pagedItems,totalItemCount){
pagedItems.get_data().filter(function(i){
results.push(i.get_fieldValues());
});
var percentLoaded = results.length / totalItemCount * 100;
console.log(String.format('{0}% has been loaded..',percentLoaded));
var pos = pagedItems.get_listItemCollectionPosition();
if(pos != null) {
return getListItems(listTitle, queryText,itemsCount,pos,results);
}
return results;
});
}
Usage
var listTitle = 'Contacts';
getListItems(listTitle , "",20)
.done(function(results){
console.log('Completed');
})
.fail(function(error){
console.log(error.get_message());
});
Results
Short Answer: You can't
Since your query is executed as a single request, there's no way to show "real" progress, although you could fake it by showing a generic "loading" gif.
Long Answer: You can if you really want to
If you were to modify your query to be paged (with a row limit per query), and then execute one request per page until all records are loaded, then you could update something on the page indicating progress.
// Use the RowLimit element to query for only 100 items at a time
camlQuery.set_viewXml("<View>"
+ "<OrderBy><FieldRef Name=\"Created\" /></OrderBy>"
+ "<RowLimit>100</RowLimit>"
+ "</View>");
Now inside the onSuccess function of executeQueryAsync(), you can access the listItemCollectionPosition property of your list item collection and pass that back into your CAML query to get the next page of items.
var itemsCount = listItems.get_count();
// use itemCount to update the current progress as displayed to the user
camlQuery.set_listItemCollectionPosition(listItems.get_listItemCollectionPosition());
// set the query's listItemCollectionPosition so you'll get the next page of results
// reload the items with the updated query
listItems = splist.getItems(camlQuery);
ctx.load(listItems);
ctx.executeQueryAsync(... // rinse and repeat to get the next batch of items
Obviously, this approach would require you to restructure your code to allow an arbitrary number of function calls. You may want to split out your onSuccess function into a named function instead of an anonymous one, so you can execute it somewhat recursively.
When you restructure your code, I also recommend wrapping the entire code block up inside an immediately executing function expression so that your variables can be accessed as needed without polluting the global namespace.
(function(){
//your code here
})();
I'm learning FRP using Bacon.js, and would like to assemble data from a paginated API in a stream.
The module that uses the data has a consumption API like this:
// UI module, displays unicorns as they arrive
beautifulUnicorns.property.onValue(function(allUnicorns){
console.log("Got "+ allUnicorns.length +" Unicorns");
// ... some real display work
});
The module that assembles the data requests sequential pages from an API and pushes onto the stream every time it gets a new data set:
// beautifulUnicorns module
var curPage = 1
var stream = new Bacon.Bus()
var property = stream.toProperty()
var property.onValue(function(){}) # You have to add an empty subscriber, otherwise future onValues will not receive the initial value. https://github.com/baconjs/bacon.js/wiki/FAQ#why-isnt-my-property-updated
var allUnicorns = [] // !!! stateful list of all unicorns ever received. Is this idiomatic for FRP?
var getNextPage = function(){
/* get data for subsequent pages.
Skipping for clarity */
}
var gotNextPage = function (resp) {
Array.prototype.push.apply(allUnicorns, resp) // just adds the responses to the existing array reference
stream.push(allUnicorns)
curPage++
if (curPage <= pageLimit) { getNextPage() }
}
How do I subscribe to the stream in a way that provides me a full list of all unicorns ever received? Is this flatMap or similar? I don't think I need a new stream out of it, but I don't know. I'm sorry, I'm new to the FRP way of thinking. To be clear, assembling the array works, it just feels like I'm not doing the idiomatic thing.
I'm not using jQuery or another ajax library for this, so that's why I'm not using Bacon.fromPromise
You also may wonder why my consuming module wants the whole set instead of just the incremental update. If it were just appending rows that could be ok, but in my case it's an infinite scroll and it should draw data if both: 1. data is available and 2. area is on screen.
This can be done with the .scan() method. And also you will need a stream that emits items of one page, you can create it with .repeat().
Here is a draft code (sorry not tested):
var itemsPerPage = Bacon.repeat(function(index) {
var pageNumber = index + 1;
if (pageNumber < PAGE_LIMIT) {
return Bacon.fromCallback(function(callback) {
// your method that talks to the server
getDataForAPage(pageNumber, callback);
});
} else {
return false;
}
});
var allItems = itemsPerPage.scan([], function(allItems, itemsFromAPage) {
return allItems.concat(itemsFromAPage);
});
// Here you go
allItems.onValue(function(allUnicorns){
console.log("Got "+ allUnicorns.length +" Unicorns");
// ... some real display work
});
As you noticed, you also won't need .onValue(function(){}) hack, and curPage external state.
Here is a solution using flatMap and fold. When dealing with network you have to remember that the data can come back in a different order than you sent the requests - that's why the combination of fold and map.
var pages = Bacon.fromArray([1,2,3,4,5])
var requests = pages.flatMap(function(page) {
return doAjax(page)
.map(function(value) {
return {
page: page,
value: value
}
})
}).log("Data received")
var allData = requests.fold([], function(arr, data) {
return arr.concat([data])
}).map(function(arr) {
// I would normally write this as a oneliner
var sorted = _.sortBy(arr, "page")
var onlyValues = _.pluck(sorted, "value")
var inOneArray = _.flatten(onlyValues)
return inOneArray
})
allData.log("All data")
function doAjax(page) {
// This would actually be Bacon.fromPromise($.ajax...)
// Math random to simulate the fact that requests can return out
// of order
return Bacon.later(Math.random() * 3000, [
"Page"+page+"Item1",
"Page"+page+"Item2"])
}
http://jsbin.com/damevu/4/edit
I want to fetch a list online from a certain URL that is in JSON format and then use the DATA_ID from each item in that list to call a new URL. I'm just new with PhantomJS and I can't figure out why nest loops inside the page.open() acts all weird. Also the way to use phantom.exit() seems to be really weird doing what I want to achieve.
Here's my code:
console.log('Loading recipes');
console.log('===============================================================');
var page = require('webpage').create();
var url = 'http://www.hiddenurl.com/recipes/all';
page.open(url, function (status) {
//Page is loaded!
var js = page.evaluate(function () {
return document.getElementsByTagName('pre')[0];
});
var recipes = JSON.parse(js.innerHTML).results;
//console.log(recipes[0].name.replace('[s]', ''));
for (i = 0; i < recipes.length; i++) {
console.log(recipes[i].name.replace('[s]', ''));
var craft_page = require('webpage').create();
var craft_url = 'http://www.hiddenurl.com/recipe/' + recipes[i].data_id;
craft_page.open(craft_url, function (craft_status) {
//Page is loaded!
var craft_js = craft_page.evaluate(function () {
return document.getElementsByTagName('body')[0];
});
var craftp = craft_js.innerHTML;
console.log('test');
});
if (i == 5) {
console.log('===============================================================');
phantom.exit();
//break;
}
}
});
The thing that happens here is that this line:
console.log(recipes[i].name.replace('[s]', ''));
..prints the following:
===============================================================
Item from DATA_ID 1
Item from DATA_ID 2
Item from DATA_ID 3
Item from DATA_ID 4
Item from DATA_ID 5
..then it just prints the next:
===============================================================
..followed by:
'test'
'test'
'test'
'test'
'test'
Why is this not happening serial? The data from the innerly called page() request gets heaped up and dumped at the end, even after phantom.exit() should actually already be called.
Also when I free-loop a normal data-set I get this error:
QEventDispatcherUNIXPrivate(): Unable to create thread pipe: Too many open files
2013-01-31T15:35:18 [FATAL] QEventDispatcherUNIXPrivate(): Can not continue without a thread pipe
Abort trap: 6
Is there any way I can set GLOBAL_PARAMETERS or direct the process in some way so I can just handle 100's of page requests?
Thanks in advance!
I've made a workaround with Python by calling PhantomJS separately through the shell, like this:
import os
import json
cmd = "./phantomjs fetch.js"
fin,fout = os.popen4(cmd)
result = fout.read()
recipes = json.loads(result)
print recipes['count']
Not the actual solution for the PhantomJS issue, but it's a working solution and has less problems with memory and code-structure.