Develop Reference

Recursive Facebook Page Webscraper with Selenium & Node.js

What I try to do is to loop through an array of Facebook page IDs and to return the code from each event page. Unfortunately, I only get the code of the last page ID in the array but as many times as elements are in the array. E.g. when I have 3 ID's in the array I get 3 times the code of the last page ID.
I already experimented with async await but I had no success.
The expected outcome would be the code of each page.
Thank you for any help and examples.
//Looping through pages
pages.forEach(
function(page) {
//Creating URL
let url = "https://mbasic.facebook.com/"+page+"?v=events";
//Getting URL
driver.get(url).then(
function() {
//Page loaded
driver.getPageSource().then(function(result) {
console.log(result);
});
}
);
}
);

you faced the same issue i did when i created a scraper using python and selenium. Facebook has countermeasure on manual URL change, you cannot change it , i receive the same data again and again even though it was automated. in order to get a good result you need to have access of face books Graph API which provides a complete object of Facebook page with its pagination URL.
or the second way i got it write was i used on click button of selenium browser automation to scroll down the next page.it wont work like you are typing , i prefer the usage of graph API

Google Tag Manager Virtual Page View

I have a little problem setting up a virtualPageView which should override the URL which is sent to google when no result is present.
Heres what I have as JavaScript code:
function returnNoSearchResultsGoogleTagManagerCode($searchterm){
if ($searchterm == "") return "";
$requestUri = $_SERVER['REQUEST_URI'] . "&no_result=".$searchterm;
$js = "<script>
$(document).ready(function(){
dataLayer.push({
'event':'empty_result',
'virtualPageURL':'".$requestUri."'
});
});
</script>";
return $js;
}
As you can see, I want to use an event trigger (empty_result).
In google, I use a Trigger to determine if the Page is a no result Page. First i created a custom Variable with custom JS
function(){
if (document.getElementsByClassName('ga-no-result').length > 0){
return true;
}else{
return false
}
}
The class is set, if the SearchEngine can't find a result. So far so good.
I also created a dataLayer variable to hold the virtualPageURL
Now I need an event which is triggered if the variable is true.
Finally I created a Tag with type PageView which is fired when the event occurs:
Until now it seems okay, the Tag is properly configured (i guess) but if I do a search which has no result, the Page URL is not overridden
The Tag is properly fired and the variables are filled. The overview of the dataLayer shows a correct dataLayer event.
But the PageURL is not overridden... Even if I wait a whole day, the category isn't sent to google.
What am I doing wrong?
I would be very thankful if someone would have an idea or even a solution :)
Thanks in advance
exa.byte
UPDATE:
Hey, I forgot to mention, that I want to use the new page variable as the string which google should use to determine the searchterm and the searchcategory
In Google Analytics I configuered the search as the "q" parameter and the "no_result" as the category.
Is it even possible to change the string which google will parse in the end?

To send a virtual pageview to Google Analytics, the field you need to change is page not {{Page Url}} , also the title field is often used.
That's really the only two things you need to do to send a simple virtual pageview.
Extra: I always start my pagepath with /virtual/ to be able to recognize which ones are virtual pageviews easily in GA

For virtual page view you have to change Field "page" - in your GTM-OnSearchEmptyResult you are changing "{{Page URL}}" - I don't think that's correct way to send virutal pageview. Also if you need to change hostname use Fieldname "hostname".
In preview mode you will not see Page URL changed in Variables Tab, you have to go to the actual GA tag that is fired and check it's values. You can either do this in GTM's preview tool or you can use standard developer tools - Network Tab and see what values are being sent to GA:
You can see "dl" parameter is the current page, if you set up virtual page you should also see parameter called "dp" this is going to be the new value of page in your GA.

If you want to setup virtual pageview you have to use page instead of {{Page URL}} in your fieldname and for Document title use title in you fieldname.
for more field reference of google analytics follow below link
https://developers.google.com/analytics/devguides/collection/analyticsjs/field-reference#hitType.

If you don't want to mess around with custom Tag Manager events it's still possible to use the good old ga method, even if you don't include the Analytics code on the page. You just need to fetch the right tracker dynamically, as explained by Simo Ahava in this thread.
if (typeof ga === "function") {
ga.getAll().forEach((tracker) => {
tracker.set('page', '/my/path'); // <- change here
tracker.send('pageview');
});
}
I also put it in a gist here.

thanks for your help. I think I got rid of the Problem and have solved it now. I will describe my solution below:
The solution is quite simple.
I had an error/ spelling error # google backend. I set the search_category parameter to "no_results", but used "no_result" for implementation...
Pretty dumb, but sometimes you just won't see the wood for the trees...
I created a new Trigger as helper "HelperDomReady" to trigger the only if DOM is ready and the variable "isEmptySearch" equals "(bool)true"
Now I can see the searchterms which have no result in google backend in the "sitesearch categories" summary. Since I won't set the parameter at all, if the search had at least one hit, the site-search category shows "not-set" for successful results. Therfore the category-section will only show searches without a hit. Problem solved :)
Disadvantage: The searchterm is also listed in the normal list. But I think this is negligible

onclick: do something after changing location

Suppose I want to do the following: when clicking a 'div', change to another location (as if it was ...) and then do something. I have tried with this code:
<div onclick="location.reload();location.href='...';togglebox('#ident')">Title</div>
where ident is the identifier of an object in the new location href='...' and togglebox is a function defined in the script. But it seems that the browser tries to accomplish togglebox('#ident') before changing the location, so the output is not the expected one. For instance, if I try
<div onclick="location.reload();location.href='...';console.log(5)">Title</div>
then a 5 appears instantaneously before changing the location, while I expect the browser first changing the location and then showing 5.
Is there any solution to this reversal-order solution?
Any suggestion is welcome

One way would be to use a query string to tell the destination page what to do. However, if you want a hack that will do almost exactly what you ask, you can do this:
(This goes in your current page's click event handler)
window.location.href = '...'; // Go to another page in your site
localStorage.setItem('exec',"yourCodeInQuotes()");
Now the destination page should either be prepared to handle the ident you mentioned from localStorage or just evaluate the code directly. Either way you need the destination page to anticipate the data that's coming.
(This goes in the destination page)
var message = localStorage.getItem('exec'); // Will get whatever you stored
eval(message); // Will attempt to run it as JavaScript
$('#'+message); // Or find it if it was your ident and do stuff...

Change URL data on page load

Hello I have a small website where data is passed between pages over URL.
My question is can someone break into it and make it pass the same data always?
For example let say, when you click button one, page below is loaded.
example.com?clicked=5
Then at that page I take value 5 and get some more data from user through a form. Then pass all the data to a third page. In this page data is entered to a database. While I observe collected data I saw some unusual combinations of records. How can I verify this?

yes. as javascript is open on the website, everyone can hack it.
you will need to write some code on you backend to validade it.
always think that you user/costumer will try to hack you sytem.
so take precautions like, check if user is the user of the session, if he is logged, if he can do what he is trying to do. check if the record that he is trying get exists.
if u are using a stand alone site, that u made the entire code from the ashes, you will need to implement this things by yourself.
like using the standard php session, making the data validation etc.
or you can find some classes that other people have made, you can find a lot o this on google. as it is a common problem of web programing.
if u are using a backed framework that isnt from another world, probably already has one. sp, go check its documentation.

html:
<a id = 'button-one' name = '5'> Button One </a>
javascript:
window.onload = function() {
document.getElementById('button-one').onclick = function() {
changeURL(this.attributes.name.value);
};
};
function changeURL(data) {
location.hash = data;
}

Pass div and value from one page to another

I have assigned an id of the user's name to each div generated by loop. This is fine. Each div/button has the username as its identity. However, using the examples here Load content of a div on another page and in other posts I am trying to send that div and its contents to another page and also pass the name value with it. My attempt is not working although the alert is alerting the name. Am I correctly passing the value? How can I recreate the div on the next page? Finally, is this even the correct approach because eventually I want to attach a table to the person. Long-winded enough?
identity = results.rows.item(i).username;
userDiv.id = identity;
console.log(identity);
(function (identity) {
userDiv.addEventListener("click",
function () {
window.open("whatever.html");
$('#singleUser').load('index.html #userDiv');
//var singleUser = document.getElementById("singleUser");
//singleUser.appendChild(userDiv);
alert(identity);
console.log(identity);
}, false);
})(identity);