Okay so i've spent the last day trying to figure something out, I'm relatively new to coding so if its a mess I'm sorry. I'm currently working on a bot that requests a JSON, here is the code I have so far
const request = require('request');
const bodyParser = require('body-parser');
global.count = 10;
for (var i = 1; global.count === 10; i++) {
var options = {
url: 'https://www.the100.io/api/v1/groups/2127/users?page=' + i, //Returns 10 entries per page, so loop is to navigate pages
headers: {
'Authorization': 'Token token="Hidden for Privacy"'
}
}
function callback(error, response, body) {
if (!error && response.statusCode == 200) {
var info = JSON.parse(body); //Also need a way to append to info as to add on as the loop progresses, still need to look that up though
console.log(JSON.stringify(info, null, 1)); //Logs the body
global.count = info.length; //Will return the value 10, until there are no more entries and then will terminate loop
}
}
request(options, callback);//Sends request
}
//It just keeps running the loop and doesn't execute the request at the bottom which is what will make the loop terminate, I've tried many things with
//callbacks and nothing has worked so far
I can't seem to be able to make the loop run properly, I didn't wanna ask for help but I am stuck I'm sad to say. Thanks ahead of time.
I think Some what confusion in the question and explain clearly???
In Your writing for loop is continuously repeating You want like that are some thing else..
in my thought it should be for navigation of page loading( every page contains 10)
global.count = 10;
for( var i = 1; i< =global.count; i++)
{
-- Write Your code here ---
}
Related
I am having an interesting issue. The general idea of what I am doing is pulling data from a Firebase database, and populating a table based on that data. Everything runs perfectly during initial population--cells and rows are populated as they should be, but the weird issue is that the scripts seem to execute again randomly. I've logged the incoming data to the console, and can see it print twice after some amount of time.
This second execution does not happen if I am to navigate between pages, or reload the page--in either of those cases everything works as it should. The problem SEEMS to happen when I log back into my computer after locking it??? Does anybody have ANY idea what could be going on here? Relevant portion of script below:
const table = document.getElementById('myTable');
firebase.auth().onAuthStateChanged(firebaseUser => {
if (firebaseUser) {
let user = firebase.auth().currentUser;
let uid = user.uid;
const dbRef = firebase.database().ref().child("data/" + uid);
dbRef.once('value', snap => {
var dataCount = snap.child("secondData").numChildren();
var datalist = snap.child("secondData").val();
var dataArray = Object.keys(datalist).map(function(k) {
return datalist[k]
});
pullAllInfo(dataCount, dataArray);
});
}
});
function pullAllInfo(count, array) {
let k = 0;
let dataArray = [];
for (i = 0; i < count; i++) {
let specificRef = firebase.database().ref().child("secondData/" + array[i]);
specificRef.once('value', snap => {
var optionsTag = array[k];
k++;
var dataId = snap.child("id").val();
var dataName = snap.child("name").val();
var dataCount = snap.child("data").numChildren();
dataArray.push(dataId, dataName, dataCount, optionsTag);
if (k == count) {
buildTable(dataArray);
console.log(dataArray);
}
});
}
}
As you can see from the code above I AM calling .once() for each reference, which would prevent data duplication from the typical .on() call. Just cant seem to figure this one out. ALSO I have an iMac, just for anyone curious about my potential computer unlock diagnosis.
Thanks all!
Most likely, the auth state is changing and setting off your function. Try throwing a log under firebase.auth().onAuthStateChanged like this:
firebase.auth().onAuthStateChanged(firebaseUser => {
console.log( 'auth state changed', firebaseUser );
if (firebaseUser) {
My guess is that you'll see that the AuthState is changing when you log out/log in from your computer.
I solved this issue by creating another global boolean called preLoaded. At the beginning, it is set to false and, once the data is loaded and passed off to build the table, it is set to true. It now looks like this:
if(k == count && preloaded == false){
preloaded = true;
console.log(dataArray);
buildTable(dataArray);
}
All set!
Main purpose: I'm trying to scrape data off of around 10,000 different pages using Node.js.
Problem: It scrapes through the first 500~1000 very fast and then turns into a turtle (its variable where it slows down) beyond that, and then eventually just seems stuck forever.
I'm using the request module in Node.js to make the requests I then use cheerio to start scraping,
This code replicates my problem:
var request = require('request');
var requestsCalledCounter = 0;
var requestsCompletedCounter = 0;
var MAX_REQUESTS = 500;
var start = function () {
while (requestsCalledCounter < MAX_REQUESTS) {
request("http://www.google.com", function (error, response, html) {
requestsCompletedCounter++;
});
requestsCalledCounter++;
}
};
start();
Output:
Test 1:
447/500
89.4%
Timed out: No requests completed after 5 seconds
447 Completed
Test 2:
427/500
85.39999999999999%
Timed out: No requests completed after 5 seconds
427
Extra details that might help:
I have an array of URL's that I am going to scrape, so I am looping through them making a request to every URL in the array. It has about 10,000 URL's.
I agree with #cviejo in the comments. You should use an existing project. However to increase the understanding, here is an implementation that will only have 10 requests outstanding at a time.
var request = require('request');
var requestsCalledCounter = 0;
var requestsCompletedCounter = 0;
var pending = 0;
var MAX_PENDING = 10;
var MAX_REQUESTS = 500;
var doreq = function () {
request("http://www.google.com", function (error, response, html) {
requestsCompletedCounter++;
pending--;
});
pending++;
requestsCalledCounter++;
}
var start = function () {
while (pending < MAX_PENDING && requestsCalledCounter < MAX_REQUESTS) {
doreq();
}
if (requestsCalledCounter < MAX_REQUESTS) {
setTimeout(start, 1);
}
};
start();
So, I'm using nodejs to parse json, and I'm wanting to get an object that's an array and use the values in a for each loop.
JSON I'm getting: http://tmi.twitch.tv/group/user/loneztar/chatters
var cronUsers = client.utils.cronjobs('1 * * * * *', function() {
console.log('API Test');
console.log('Response from Twitch Chat:');
request('http://tmi.twitch.tv/group/user/loneztar/chatters', function (error, response, body) {
if (!error && response.statusCode == 200) {
data = JSON.parse(body);
console.log(data.chatters.viewers);
//for each string in the viewers
}
})
});
Above is the code I'm using to retrieve the data. And when I log to the console I get the following (at time of asking):
[ 'duckziller72',
'adanaran',
'nyckeln',
'diabolicalsheep69',
'audery101',
'gone_nutty',
'k4unl',
'eidokan',
'mattesolo',
'siland',
'nullvoid8',
'troy00114',
'sixdev',
'jake_evans',
'doctoranguus',
'juicegraip',
'k4rush' ]
Now I want to use each of them strings in a for each loop. I've tried using the data var I'm using but I can't get anything working. Would appreciate some help! Feel free to ask for other stuff, I'll happily edit.
There are a few different ways to do this.
Using a loop counter:
var viewers = data.chatters.viewers;
var numViewers = viewers.length;
for (var i = 0; i < numViewers; i++) {
var viewer = viewers[i];
console.log(viewer);
}
Using 'for .. in':
var viewers = data.chatters.viewers;
for (var i in viewers) {
var viewer = viewers[i];
console.log(viewer);
}
You may have been trying to use this, but the loop variable contains the indices of the array, not the values.
Using Array.prototype.forEach():
data.chatters.viewers.forEach(function(viewer) {
console.log(viewer);
});
Note that none of this is specific to Node.js or JSON responses.
I'm trying to write a performance tool using node.js so I can automate it, and store the results in MySQL. The tool is supposed to gather how long took for the browser to load a particular webpage. I'm using HttpWatch to measure the performance, and the result is displayed in seconds. The browser utilized is Firefox.
Below is a piece of script I'm using to run the performance test:
var MyUrls = [
"http://google.com",
"http://yahoo.com"
];
try {
var win32ole = require('win32ole');
var control = win32ole.client.Dispatch('HttpWatch.Controller');
var plugin = control.Firefox.New();
for (var i=0; i < MyUrls.length; i++) {
var url = MyUrls[i];
console.log(url);
for(var j=0; j < 14; j++) {
// Start Recording HTTP traffic
plugin.Log.EnableFilter(false);
// Clear Cache and cookier before each test
plugin.ClearCache();
plugin.ClearAllCookies();
plugin.ClearSessionCookies();
plugin.Record();
// Goto to the URL and wait for the page to be loaded
plugin.GotoURL(url);
control.Wait(plugin, -1);
// Stop recording HTTP
plugin.Stop();
if ( plugin.Log.Pages.Count != 0 )
{
// Display summary statistics for page
var summary = plugin.Log.Pages(0).Entries.Summary;
console.log(summary.Time);
}
}
}
plugin.CloseBrowser();
} catch(e) {
console.log('*** exception cached ***\n' + e);
}
After the second iteration of the inner loop, I'm getting the following error:
C:\xampp\htdocs\test\browser-perf>node FF-load-navigation.js
http://localhost/NFC-performance/Bing.htm
[Number (VT_R8 or VT_I8 bug?)]
2.718
[Number (VT_R8 or VT_I8 bug?)]
2.718
OLE error: [EnableFilter] -2147352570 [EnableFilter] IDispatch::GetIDsOfNames Au
toWrap() failed
Have someone seen this before? Can you help me?
You have to remember that node is asynchronous
So that for loop runs simultaneously to plugin.CloseBrowser();, which is obviously not what you want because thats causing it to close, which will cause problems in the for loop.
rather you want that to run after the for loop finishes.
Look at async for a simple way to do this.
async.each(MyUrls, function (callback) {
...
callback()
}, function(err){
plugin.CloseBrowser();
});
The same has to be done for your inner for loop.
Here's the code:
exports.index_post = function(req, res) {
var nicks = [];
if (req.body.nick) {
for (var nick in nicks) {
if (nick == req.body.nick) {
res.redirect("/");
} else {
nicks.push(req.body.nick)
req.session.nick = req.body.nick;
res.redirect("/msg");
console.log(nicks);
}
}
} else {
res.redirect("/");
}
};
What it's meant to do is check if req.body.nick is one of the items in the nicks dictionary, and if if is redirect it back to the root. If it's not in the dictionary, it should add it to the dictionary and set it as a session variable, then redirect to /msg. However, this code doesn't seem to be working for me and instead it causes Express to hang. Could someone tell me what I'm doing wrong? Thanks.
First off, you're creating a new nicks array every time the function is run; it sounds like you want this to persist throughout the life of the server, each request potentially adding to the nicks array. Also, though you're iterating over the array looking for req.body.nick, the way the if/else statement is written inside the loop, it will always redirect on the very first iteration through the loop. Try something like this:
var nicks = []; // don't reset `nicks` every call
exports.index_post = function(req, res) {
if (req.body.nick) {
if (nicks.indexOf(req.body.nick) != -1) { // better checking for nick in nicks
res.redirect("/");
} else {
nicks.push(req.body.nick)
req.session.nick = req.body.nick;
res.redirect("/msg");
console.log(nicks);
}
} else {
res.redirect("/");
}
};
Your nicks is empty
so your for loop won't do anything
you don't even have a chance to push anything into it
For each request you'll have a new nicks, it's not global