Develop Reference

How do I prevent duplicate API Calls in JavaScript FOR Loop

I have an array of URLs I scraped from a webpage and then make an API call to validate the URLs to see if they are malicious. The only problem is I am limited to a certain amount of API calls per day and the array contains duplicate URLs. I am trying to loop through the array and used a saved API call for duplicate values and I am struggling to find the best way to do it since there could be multiple duplicates. If the loop encounters a duplicate value I want it to not make the API call and just return the already saved values from the previous API call. I included some basic sudo code inside the code below and I am unsure of what to populate the sudo code with.
/* urlToValidate is a list of URLS */
urlToValidate = ["ups.com", "redfin.com", "ups.com", "redfin.com", "redfin.com", "redfin.com"];
var isValid = false;
/* API Overview https://www.ipqualityscore.com/documentation/malicious-url-scanner-api/overview */
for (let i = 0; i < urlToValidate.length; i++) {
if (i == 0 || Is Not A DUPLICATE) {
$.getJSON('https://ipqualityscore.com/api/json/url/<API_KEY>/' + urlToValidate[i], function( json ) {
if (!json.phishing && !json.spamming && json.risk_score < 80) {
isValid = true;
returnMessage(isValid, json.risk_score, i)
} else {
isValid = false;
returnMessage(isValid, json.risk_score, i)
}
});
} else {
returnMessage(alreadySaved duplicateValue, alreadySaved duplicate risk_score, i)
}
}
Desired Output:
URL Valid: true Risk Rating: 0 Position: 7
or
Duplicate URL: true Risk Rating: 0 Position: 7

This is a simple matter of caching.
Outside of your for loop, maintain some kind of mapping of URLs to their corresponding fetch results. That way, you can store not only whether that URL has been called but also the result, if it exists. An easy way to do that is with a basic object, where the "keys" are strings corresponding to the URLs, and the "values" are the results of your fetches.
const resultCache = {};
Inside of your loop, before you do a fetch you should first check whether the cache already has a result for that URL.
let result;
if (resultCache[urlToFetch]) {
result = resultCache[urlToFetch];
} else {
// use the previous result
result = await fetch(/* whatever */);
// remember to also store result in cache
resultCache[urlToFetch] = result;
}

You have a few options.
First you could convert your urls to a Set which prevents any duplicates from occurring at all.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Set
Another option would be to store the return in an object with the key being the url and in your if statement check to see if the value is not null.
*** UPDATE using a set ***
/* urlToValidate is a list of URLS */
urlToValidate = ["ups.com", "redfin.com", "ups.com", "redfin.com", "redfin.com", "redfin.com"];
var urls = new Set(urlToValidate);
var isValid = false;
/* API Overview https://www.ipqualityscore.com/documentation/malicious-url-scanner-api/overview */
for (let i = 0; i < urls.length; i++) {
$.getJSON('https://ipqualityscore.com/api/json/url/<API_KEY>/' + urls[i], function( json ) {
if (!json.phishing && !json.spamming && json.risk_score < 80) {
isValid = true;
returnMessage(isValid, json.risk_score, i)
} else {
isValid = false;
returnMessage(isValid, json.risk_score, i)
}
});
}
}

Trouble creating unordered list with number of list items controlled by user input

I tried to make a function that would generate a number of list items based on the user input from a prompt. It does not work although I believe it should.
I'm looking for an explanation of what's wrong with my code even if an alternate solution is also provided, if possible.
On the HTML side I have entered <div class="freshList"></div> in the body so that it can be picked up by the function and have the list placed in that location
Code is below:
function makeAList()
{
var freshList = document.getElementsByClassName("freshList");
var listLength = prompt("Enter number of list items");
var listString = "<ul>";
for (var i=0; i < listLength; i++)
{
listString+= "<li>"+"</li>"
}
listString += "</ul>"
document.innerHTML = listString;
}
makeAList();
// end code
Now the only way I have been able to get this to work was by accident when using the document.Write method at various points in the code to see what was working (I tried console log first which said that the function was called and the loop was proceeding but no output was coming so I switched to doc.write instead). I used document.Write(listString); and this was able to forcibly print the bullet points onto the screen but that is not my desire. I want it in the HTML not just printed on the screen (so that I can manipulate it with other functions I have made).
Altogether I wanted to make a series of functions to perform the following action: Ask if the user would like to make a new list. Call the makeNewList function which would prompt the user for the number of items. Then ask the user if they would like to edit the list and call the editList function with new prompts for each list item. Finally leaving an output of # of bullet points with user input on each point. I am sure this is a ridiculous idea that nobody would use but it was more a lesson for myself to try an idea I had rather than something functional. Full (attempted) code below:
function makeAList()
{
var freshList = document.getElementsByClassName("freshList");
var listLength = prompt("Enter number of list items");
var listString = "<ul>";
for (var i=0; i < listLength; i++)
{
listString+= "<li>"+"</li>"
}
listString += "</ul>"
document.innerHTML = listString;
}
makeAList();
function editAList() {
var list = document.getElementsByTagName("li");
for (var i = 0; i < list.length; i++)
{
list[i].innerHTML = prompt("Place list text below","")
}
function checkList(){
var resp1 = confirm("Would you like to make a new list?")
if(resp1 == true)
{
makeAList();
}
else
{
}
if(resp1 === false){
var resp2 = prompt("Would you like to edit an existing list instead?")
}
else if(resp2 === true){
editAList();
}
else{
alert("You have chosen not to make a new list or edit an existing one")
}
}
checkList();

My friend looked at my code and made some changes as well as detailed comments with the places I went wrong. For anyone who views this question in the future here is his response. All credit to him but I don't know his stack overflow handle to tag him.
Here is his js bin updated and heavily commented code
Code below in case that link dies:
// hi
// i've changed a few things, i've left the original code in comments (//)
function makeAList()
{
// what does the following code return? a single element? a list of elements?
//var freshList = document.getElementsByClassName("freshList")
var freshList = document.getElementById("freshList");
var listLength = prompt("Enter number of list items");
// var listString = "<ul>";
// you can create a 'ul' element and append the list string later
// https://developer.mozilla.org/en-US/docs/Web/API/ParentNode/append
var ul = document.createElement('ul');
ul.setAttribute('id', 'theList');
// there's an even shorter way of doing all this, but since you're starting out, we can save that for later
for (var i=0; i < listLength; i++)
{
//i would probably append here too, but just demonstrating insertAdjacent
ul.insertAdjacentHTML('beforeend', '<li></li>');
}
// document.innerHtml = listString //this was the reason why this function didn't work
// document has no inner html, instead, you want to append the list to the .freshList div that you created
// and then append that to the listOfLists that you queried
// the reason why we don't want to manually set innerHTML is because the DOM has to be reparsed and recreated
// every time innerHTML is set. if you have 1000s of lists, this would be extremely slow
// there are DOM apis that create and insert html elements much more faster and efficient (appendChild)
// if you want to create html elements as strings, as you have done previously, use insertAdjacentHTML: https://developer.mozilla.org/en-US/docs/Web/API/Element/insertAdjacentHTML
// it is faster and more efficient
freshList.appendChild(ul);
}
makeAList();
function editAList() {
var list = document.getElementsByTagName("li");
// there's a much more efficient way to do this, but keep this here for now
var insertText = function(i) {
var input = prompt("Place list text below", "");
console.log(i);
list[i].append(input);
}
for (var i = 0; i < list.length; i++)
{
// why would we use settimeout? http://www.w3schools.com/jsref/met_win_settimeout.asp
setTimeout(insertText.bind(null, i), 1000); // why bind? https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Function/bind
}
}
editAList();
// function checkList(){
// var resp1 = confirm("Would you like to make a new list?")
// if(resp1 == true)
// {
// makeAList();
// }
// else
// {
// }
// if(resp1 === false){
// var resp2 = prompt("Would you like to edit an existing list instead?")
// }
// else if(resp2 === true){
// editAList();
// }
// else{
// alert("You have chosen not to make a new list or edit an existing one")
// }
// }
// checkList();

Skipping multiple elements in a FOR loop, Javascript

I have some file contents I'd like to pass on to my server using a javascript function. In the file, there are many empty lines, or those which contain useless text. So far I read every line in as a string stored in an array.
How do I then loop through that content skipping multiple lines such as lines 24,25, 36, 42, 125 etc. Can I put these element id's into an array and tell my for loop to run on every element except these?
Thanks

you can't tell your for loop to iterate all, but skip certain elements. it will basically just count in any direction (simplified) until a certain critera has been met.
you can however put an if inside your loop to check for certain conditions, and chose to do nothing, if the condition is met. e.g.:
(pseudo code below, beware of typing errors)
for(var line=0; line < fileContents.length; line++) {
if(isUselessLine(line)) {
continue;
}
// process that line
}
the continue keyword basically tells the for loop to "jump over" the rest of the current iteration and continue with the next value.
The isUselessLine function is something you'll have to implement yourself, in a way, that it returns true, if the line with the given linenumber is useless for you.

You can try this its not much elegent but will suerly do the trick
<html>
<body>
<p>A loop which will skip the step where i = 3,4,6,9.</p>
<p id="demo"></p>
<script>
var text = "";
var num = [3,4,6,9];
var i;
for (i = 0; i < 10; i++) {
var a = num.indexOf(i);
if (a>=0) {
continue;
}
text += "The number is " + i + "<br>";
}
document.getElementById("demo").innerHTML = text;
</script>
</body>

You could use something like this
var i = 0, len = array1.length;
for (; i < len; i++) {
if (i == 24 || i == 25) {
array1.splice(i, 1);
}
}
Or you can have an another array variable which got all the items that need to be removed from array1

Another method:
var lines = fileContents.match(/[^\r\n]+/g).filter(function(str,index,arr){
return !(str=="") && uselessLines.indexOf(index+1)<0;
});

If you have many indices to skip, and this depends on the elements of the array, you could write a function that returns the number of elements to skip over for each index in that array (or returns 1, if no skipping required):
for ( let i = 0;
i < array.length;
i += calcNumberOfIndicesToSkip( array, i )){
// do stuff to the elements that aren't
// automatically skipped
}
function calcNumberOfIndicesToSkip( array, i ){
// logic to determine number of elements to skip
// (this may be irregular)
return numberOfElementsToSkip ;
}
In your case:
// skip the next index (i+1)?
for ( let i=0; i<array.length; i+=skipThisIndex(i+1) ){
// do stuff
}
function skipThisIndex(i){
const indicesToSkip = [ 24, 25, 36, 42, 125 ];
return 1 + indicesToSkip.includes(i);
}
// returns 1 if i is not within indicesToSkip
// (there will be no skipping)
// => (equivalent to i++; normal iteration)
// or returns 1 + true (ie: 2) if i is in indicesToSkip
// => (index will be skipped)

Looping through arrays and checking their values

What I have done is that I have firstly I have parsed them with the jQuery function, $.parseJSON() and so now I am having 2 arrays with 3 objects inside them. What I want to do is that I want to loop over both the arrays at the same time so that I could check the Post_IDs of the current object in the loop of both the arrays that if they are equal or not and then if I find that the Post_ID is not present there already or there is a mismatch then I want to do something.
Here's what I have done:
var posts = <? php echo $posts; ?> ;
setInterval(function () {
$.get("admin/post/get_new_posts.php", {
userId: <? php echo User::GetUserID($_SESSION["username"]); ?>
}, function (data) {
data = $.parseJSON(data);
for (var i = 0; i < data.length; i++) {
for (var j = 0; j < posts.length; j++) {
if (posts[j].Post_ID != data[i].Post_ID) {
//do something...
} else {
}
}
}
});
}, 10000);
The posts variable has its value and I am sure about it by logging them into the console. Now what happens here is that every ten seconds a get request is sent and checks if the Post_ID meets of the posts variable object does not equal the Post_ID of the data variable then do something in there. But that's not the case. The condition is always going true.
And yes, here are my objects. The first one is the posts variable and the second one is the data variable.
Any idea of how can I loop through both the variables or arrays and check for a property at the same time while looping?

You're comparing every entry in posts to every entry in data. Most of those won't match - posts[0] won't match data[1], data[2], etc.
It sounds like you want to look at the corresponding members of each list (posts[0] vs. data[0], etc.)
In that case, you need one loop, not two.
// find the longest array's length
var maxi = Math.max(posts.length, data.length, 0);
for ( var i = 0; i < maxi; ++i )
{
if (i >= posts.length)
{
// an entry in data, but not posts
}
else if (i >= data.length)
{
// an entry in posts, but not data
}
else if (data[i].Post_ID != posts[i].Post_ID)
{
// mismatched IDs
}
else
{
// matched IDs
}
}

You are executing two loops and this creates a problem. because they have the following meaning (I'm gonna translate them into pure english):
for each item of array 'data', check any item in array 'posts' that is not equal to the current item of 'data'. This is why it is always "doing something".
Since both arrays have the same length, you should do one loop like this :
for (var i = 0; i < data.length; i++) {
if (posts[i].Post_ID != data[i].Post_ID) {
//do something...
} else {
}
}

JavaScript: Improving the performance of a FOR Loop to stop the browser locking up?

I have an array of objects, I loop through this array one object at a time and I make a couple of checks to see if each object in that array meets certain criteria, if that object meets this criteria I then copy a property of this object in to an array (that property also contains another object).
for(var v = 0; features.length > v; v++){
var feature = features[v];
//If the feature is on the map then we want to add it
if(feature.onScreen()){
dataArray.push(feature.attributes);
}
}
Now for some reason if this array of objects is big (5000+) in size this operation becomes very expensive and the browser just locks up for a second or two (Some times more).
I cant go in to much more info of what the code does but I was wondering given this loop, what would be the best way to give the browser a break lets say every 500 iterations and so it doesn't lock up and then continue on etc.
Thanks

Not sure... but what about putting that code into a function, then breaking out and recalling the function (say every xMs)?
eg,
var LastPos = 0;
function DoLoop()
{
for(var v = LastPos; features.length > v; v++){
var feature = features[v];
//If the feature is on the map then we want to add it
if(feature.onScreen()){
dataArray.push(feature.attributes);
}
if(v > 500)
{
LastPos = v;
break;
}
}
setTimeout('DoLoop()', 10);
}

setTimeOut() can solve this.
var maxStatement = 1000, currentIndex=0, timeoutVar = ''
var sLimit = features.length
function multiStepLoop(){
for (var i=currentIndex; i<currentIndex+maxStatement; i++){
//---DO YOUR STUFFS HERE
}
var a = sLimit-i;
currentIndex += maxStatement;
if (maxStatement >= a){ maxStatement=a }
if (a<=0){
callBackFunction() //-- function to call when your peocess finish.
}else{
timeoutVar = setTimeout('multiStepLoop()',1)
}
}
multiStepLoop();
The drawback is that you need to cover all you want to do after this process complete into a function, then run it like a callBack_Function.
Note : Jus need to set the time for setTimeout to 1 millisecond & the browser won't display the waiting-cursor.

Change the structure of the loop to :
var index = 0;
function myfunc() {
while(features.length > index){
var feature = features[v];
//If the feature is on the map then we want to add it
if(feature.onScreen()){
dataArray.push(feature.attributes);
}
index++;
//You can do this
var waitingTime = 0;
If (index% 500=0) waitingTime=100; //or 10, 20...
//Or just with a short interval
var waitingTime = 10;
setTimeOut('myfunc', waitingTime);
}
}
Or with a parameter : [I prefer this one]
function myfunc(index) {
if(!index) {
index=0;
}
while(features.length > index){
var feature = features[v];
//If the feature is on the map then we want to add it
if(feature.onScreen()){
dataArray.push(feature.attributes);
}
//You can do this
var waitingTime = 0;
If (index% 500=0) waitingTime=100; //or 10, 20...
//Or just with a short interval
var waitingTime = 10;
setTimeOut('myfunc', waitingTime);
}
}
[EDIT]
Change setTimeOut calls...
And in your code, when you call the function, don't give the param, to initialize the index!

What if you did something with a little burst and a little recursion:
Fiddle Code
Main Function
What happens is a given index is passed with a burst amount of intervals (i.e. 500), upon ending the for loop at the given number of burst iterations, it recalls itself with the same burst number. The function ends when the end of the features array is completed.
var transferFeatures = function (index, burst) {
for (var z = 0; z <= burst; z++) {
if (index === features.length) {
return;
}
var feature = features[index];
if (feature.onScreen) {
dataArray.push(feature.attributes);
}
index++;
}
if (index !== features.length) {
transferFeatures(index, burst);
}
};
Testing
To simulate load, I created an array of objects using various key-value pairs (most importantly, your attributes inner object):
//To simulate a random boolean
var randomBoolean = function () {
return Math.random() <= 0.5;
};
//A random integer
var getRand = function () {
return (Math.floor(Math.random() * 10).toString());
};
// Create a bunch of dummy objects
var randomFeatures = function (arr, i) {
for (var p = 0; p < i; p++) {
arr.push({
onScreen: randomBoolean(),
attributes: {
width: getRand(),
height: getRand(),
someAtt: "I'm just an attribute",
coolKidsRideBikes: true,
foo: "bar",
bar: "baz"
}
});
}
};
Granted, it's not the same onScreen() test you will be using, but either way, it evaluates to a boolean value. I think if you apply this concept with your code, you could have amazing results.
Everything in the Fiddle I linked too at the top is called like this:
randomFeatures(features, 5000);
console.log(features.length);
transferFeatures(0,500);
console.log(dataArray.length);
Load Testing
I simulated 5000000 (5 million) random objects being pushed onto features with a burst of 1000 and the script completed in around 3.29 seconds.