Develop Reference

Chrome Extension - read source code with dynamic content

I have written code for a small Chrome Extension that scrapes content based on selectors. Once I have the content I need, I push it to my PHP server in order to store it in a database. The code works well except when content is loaded dynamically, I can only see what is visible the source code from the browser.
I thought I could solve the issue by waiting until the page was fully loaded (setInterval) but even when I see that the content is loaded, the source code I'm getting is still the initial one.
I use the following code to create a tab and retrieve the source code:
function CreateTab (createProperties)
{
chrome.tabs.create(createProperties, tab =>
{
if (chrome.runtime.lastError)
{
console.log(chrome.runtime.lastError);
}
else
{
container = "h2";
var i = 0;
var checkClass = setInterval (function()
{
if (i<=3)
{
console.log('Iteration :'+i);
chrome.tabs.executeScript(tab.id,
{
file: 'GetSource.js',
}, async function(results)
{
let e = chrome.runtime.lastError;
if(e !== undefined)
{
console.log(tab.id, e);
}
// GETTING HTML
parser = new DOMParser();
content = parser.parseFromString(results, "text/html");
console.log(content);
// CAPTURE TITLES
try
{
datatable = content.querySelectorAll(container);
}
catch(err)
{
console.log('Iteration '+i+' table not found');
datatable = "";
}
if (datatable.length>0)
{
clearInterval(checkClass);
removeTab(tab.id,3)
// DATA FOUND!
i=10;
}
});
i++;
}
else
{
clearInterval(checkClass);
console.log('Container not found');
removeTab(tab.id,5)
}
},10000);
}
});
}
The script I'm calling to get the source is the following (GetSource.js)
function DOMtoString(document_root) {
var html = '',
node = document_root.firstChild;
while (node) {
switch (node.nodeType) {
case Node.ELEMENT_NODE:
html += node.outerHTML;
break;
case Node.TEXT_NODE:
html += node.nodeValue;
break;
case Node.CDATA_SECTION_NODE:
html += '<![CDATA[' + node.nodeValue + ']]>';
break;
case Node.COMMENT_NODE:
html += '<!--' + node.nodeValue + '-->';
break;
case Node.DOCUMENT_TYPE_NODE:
// (X)HTML documents are identified by public identifiers
html += "<!DOCTYPE " + node.name + (node.publicId ? ' PUBLIC "' + node.publicId + '"' : '') + (!node.publicId && node.systemId ? ' SYSTEM' : '') + (node.systemId ? ' "' + node.systemId + '"' : '') + '>\n';
break;
}
node = node.nextSibling;
}
return html;
}
sourcecode = DOMtoString (document);
sourcecode
How can I make sure that the GetSource.js script is getting the final rendered source code? There are more and more sites using dynamic content loaded after the initial load and with this script I'm stuck on those sites.
There are ready to use scrapers in the Chrome store that can get dynamic content so there must be a solution for my script too. I don't want to use to solutions as I need to retrieve the selectors I want to catch from my server and once the scraping is done pushing them back to the server.
Do you have any idea of how I could achieve this?
Thanks

Validating a directory from a javascript page using $.post to call a webservice PHP function

I am trying without success to use the $.post function to test (via a webservice that calls a PHP function "is_dir") if a folder already exists on a server and then I want it to return a string or boolean value back to my javascript page before I proceed to dynamically write the new files that will be placed there. The file path of the folder to be tested is "built" using jQuery which captures form data. I need to define (in a variable) if the directory exists and then be able to access that variable from outside of the $.post function (not from within, using success callback). This is so I can proceed in javascript as follows:
if {directory exists} then
capture more form data (via jQuery) and
$.post to webservice that calls PHP to update database
Outside of the $.post function, the value of my return variable is undefined.
I think I may be over-complicating this. Any suggestions? Thank you, in advance.

Please see my comment to #Steve above:
<script type='text/javascript'>
//function gathers form data, validates constructed file path and then writes to DB
function post_FormData() {
var week_number = $("#form_week_number").val();
var program = $("#form_program").val();
var course = $.trim($("#form_course_number").val());
var form_content_type = $("input:radio[name=content_type]:checked").val();
var content_type = "";
var activity_title_Val = $.trim($("#form_activity_name").val());
var activity_title_Split = activity_title_Val.split(" ");
var activity_title_Clean = new Array();
//this for-loop constructs a valid directory folder name from form data
for(var i=0, l=activity_title_Split.length; i<l; i++) {
activity_title_Split[i] = activity_title_Split[i].replace(/[^a-z0-9\s]/gi,"");
activity_title_Clean[i] = activity_title_Split[i];
activity_title_Split[i] = activity_title_Split[i].replace(/\b[a-z]/g, function(letter){return letter.toUpperCase();});
}
var activity_title = activity_title_Split.join("");
var file_path = "";
file_path += "/CourseFiles/" + program + "/" + program + course + "/" + content_type + "/Week" + week_number + "/activity-" + activity_title;
var message = "<div id=\"confirmation_container_contents\"><p><b>Confirm Content Repository file path: </b><br></p>";
//begin web service call to PHP function
$.post('webservices/create_PA_webservices.php', {web_service: "go_check_if_exists", data_file_path: file_path}, function(data){
var exists = data.does_exist; //json_encoded RESPONSE FROM ASYNC REQUEST
if(exists == "Y") {
message += file_path;
message += "<br><br><br><center><b>An activity folder with this name already exists.</b></center>";
message += "<br><br><center>Please edit the activity title and resubmit.</center>";
message += "<br><br><br><center><input type=\"image\" src=\"pa_images/editButton.jpg\" id=\"editButton\" value=\"edit\"></center></div>";
$("#confirmation_container").empty();
$("#confirmation_container").append(message);
}
else if(exists == "N") {
message += file_path;
message += "<br><br><center><input type=\"image\" src=\"pa_images/editButton.jpg\" id=\"editButton\" value=\"edit\">";
message += "&nbsp\;&nbsp\;&nbsp\;<input type=\"image\" src=\"pa_images/confirmButton.jpg\" id=\"confirmButton\" value=\"confirm\"></center></div>";
$("#confirmation_container").empty();
$("#confirmation_container").append(message);
}
$(function(){//edit proposed file path
$("#editButton").click(function() {
$("#confirmation_container").empty();
});//end function edit path button
});//end anonymous function
$(function(){//confirm proposed file path and write to DB
$("#confirmButton").click(function() {
go_post_FormData(activity_title_Val, file_path, week_number, program, course, content_type);
$("#create_practice_activity").hide();
$("#build_practice_activity").show();
$("#activity_is_new").val("N");
});//end function confirm path button
});//end anonymous function
}, "json").fail(function() {alert("The go_check_if_exists webservice call has failed");}); //end web service call
}//end function post_FormData declaration
</script>

How to save the current webpage with casperjs/phantomjs?

Is there a way to save the current webpage by using casperjs or phantomjs?
I tried to get the html and save it into a file. But the resulting file was a lot different from the screenshot of that time (with casper.capture). Is there a way to save the current webpage?

Andrey Borisko suggested to use the disk cache to retrieve the resources. My solution is not that efficient, but you don't need to decompress text files.
I use XMLHttpRequest to retrieve all resources after I registered them with the resource.received event handler. I then filter the resources into images, css and fonts. The current limitation is that remote resource paths that contain something like ../ or ./ are not handled correctly.
I retrieve the current page content with getHTML and iterate over all captured resources to replace the path used in the markup, that is identified by a portion of the complete resource URL, with a randomly generated file name. The file extension is created from the content type of the resource. It is converted using mimeType from this gist.
Since CSS files may contain background images or fonts, they have to be processed before saving to disk. The provided loadResource function loads the resource, but does not save it.
Since XMLHttpRequest to download the resources the script has to be invoked with the --web-security=false flag:
casperjs script.js --web-security=false
script.js
var casper = require("casper").create();
var utils = require('utils');
var fs = require('fs');
var mimetype = require('./mimetype'); // URL provided below
var cssResources = [];
var imgResources = [];
var fontResources = [];
var resourceDirectory = "resources";
var debug = false;
fs.removeTree(resourceDirectory);
casper.on("remote.message", function(msg){
this.echo("remote.msg: " + msg);
});
casper.on("resource.error", function(resourceError){
this.echo("res.err: " + JSON.stringify(resourceError));
});
casper.on("page.error", function(pageError){
this.echo("page.err: " + JSON.stringify(pageError));
});
casper.on("downloaded.file", function(targetPath){
if (debug) this.echo("dl.file: " + targetPath);
});
casper.on("resource.received", function(resource){
// don't try to download data:* URI and only use stage == "end"
if (resource.url.indexOf("data:") != 0 && resource.stage == "end") {
if (resource.contentType == "text/css") {
cssResources.push({obj: resource, file: false});
}
if (resource.contentType.indexOf("image/") == 0) {
imgResources.push({obj: resource, file: false});
}
if (resource.contentType.indexOf("application/x-font-") == 0) {
fontResources.push({obj: resource, file: false});
}
}
});
// based on http://docs.casperjs.org/en/latest/modules/casper.html#download
casper.loadResource = function loadResource(url, method, data) {
"use strict";
this.checkStarted();
var cu = require('clientutils').create(utils.mergeObjects({}, this.options));
return cu.decode(this.base64encode(url, method, data));
};
function escapeRegExp(string) {
// from https://stackoverflow.com/a/1144788/1816580
return string.replace(/([.*+?^=!:${}()|\[\]\/\\])/g, "\\$1");
}
function replaceAll(find, replace, str) {
// from https://stackoverflow.com/a/1144788/1816580
return str.replace(find, replace);
}
var wrapFunctions = [
function wrapQuot1(s){
return '"' + s + '"';
},
function wrapQuot2(s){
return "'" + s + "'";
},
function csswrap(s){
return '(' + s + ')';
}
];
function findAndReplace(doc, resources, resourcesReplacer) {
// change page on the fly
resources.forEach(function(resource){
var url = resource.obj.url;
// don't download again
if (!resource.file) {
// set random filename and download it **or** call further processing which in turn will load ans write to disk
resource.file = resourceDirectory+"/"+Math.random().toString(36).slice(2)+"."+mimetype.ext[resource.obj.contentType];
if (typeof resourcesReplacer != "function") {
if (debug) casper.echo("download resource (" + resource.obj.contentType + "): " + url + " to " + resource.file);
casper.download(url, resource.file, "GET");
} else {
resourcesReplacer(resource);
}
}
wrapFunctions.forEach(function(wrap){
// test the resource url (growing from the back) with a string in the document
var lastURL;
var lastRegExp;
var subURL;
// min length is 4 characters
for(var i = 0; i < url.length-5; i++) {
subURL = url.substring(i);
lastRegExp = new RegExp(escapeRegExp(wrap(subURL)), "g");
if (doc.match(lastRegExp)) {
lastURL = subURL;
break;
}
}
if (lastURL) {
if (debug) casper.echo("replace " + lastURL + " with " + resource.file);
doc = replaceAll(lastRegExp, wrap(resource.file), doc);
}
});
});
return doc;
}
function capturePage(){
// remove all <script> and <base> tags
this.evaluate(function(){
Array.prototype.forEach.call(document.querySelectorAll("script"), function(scr){
scr.parentNode.removeChild(scr);
});
Array.prototype.forEach.call(document.querySelectorAll("base"), function(scr){
scr.parentNode.removeChild(scr);
});
});
// TODO: remove all event handlers in html
var page = this.getHTML();
page = findAndReplace(page, imgResources);
page = findAndReplace(page, cssResources, function(cssResource){
var css = casper.loadResource(cssResource.obj.url, "GET");
css = findAndReplace(css, imgResources);
css = findAndReplace(css, fontResources);
fs.write(cssResource.file, css, "wb");
});
fs.write("page.html", page, "wb");
}
casper.start("http://www.themarysue.com/").wait(3000).then(capturePage).run(function(){
this.echo("DONE");
this.exit();
});
The magic happens in findAndReplace. capturePage is completely synchronous so it can be dropped anywhere without much head ache.
URL for mimetype.js

No, I don't think there is an easy way to do this as phantomjs doesn't support rendering pages in mht format (Render as a .mht file #10117). I believe that's what you wanted.
So, it needs some work to accomplish this. I did something similar, but i was doing it the other way around I had a rendered html code that I was rendering into image/pdf through phantomjs. I had to clean the file first and it worked fine for me.
So, what I think you need to do is:
strip all js calls, like script tags or onload attributes, etc..
if you have access from local to the resources like css, images and so on (and you don't need authentication to that domain where you grab the page) than you need to change relative paths of src attributes to absolute to load images/etc.
if you don't have access to the resources when you open the page then I think you need to implement similar script to download those resources at the time phantomjs loads the page and then redirect src attributes to that folder or maybe use data uri.
You might need to change links in css files as well.
This will bring up the images\fonts and styling you are missing currently.
I'm sure there are more points. I'll update the answer if you need more info, once I see my code.

fetching xml data into a div via ajax and javascript

Building a chat app and I am trying to fetch all logged in user into a div with ID name "chat_members". But nothing shows up in the div and I have verified that the xml file structure is correct but the javascript i'm using alongside ajax isn't just working.
I think the problem is around the area of the code where I'm trying to spool out the xml data in the for loop.
XML data sample:
<member>
<user id="1">Ken Sam</user>
<user id="2">Andy James</user>
</member>
Javascript
<script language="javascript">
// JavaScript Document
var getMember = XmlHttpRequestObject();
var lastMsg = 0;
var mTimer;
function startChat() {
getOnlineMembers();
}
// Checking if XMLHttpRequest object exist in user browser
function XmlHttpRequestObject(){
if(window.XMLHttpRequest){
return new XMLHttpRequest();
}
else if(window.ActiveXObject){
return new ActiveXObject("Microsoft.XMLHTTP");
} else{
//alert("Status: Unable to launch Chat Object. Consider upgrading your browser.");
document.getElementById("ajax_status").innerHTML = "Status: Unable to launch Chat Object. Consider upgrading your browser.";
}
}
function getOnlineMembers(){
if(getMember.readyState == 4 || getMember.readyState == 0){
getMember.open("GET", "get_chat.php?get_member", true);
getMember.onreadystatechange = memberReceivedHandler;
getMember.send(null);
}else{
// if the connection is busy, try again after one second
setTimeout('getOnlineMembers()', 1000);
}
}
function memberReceivedHandler(){
if(getMember.readyState == 4){
if(getMember.status == 200){
var chat_members_div = document.getElementById('chat_members');
var xmldoc = getMember.responseXML;
var members_nodes = xmldoc.getElementsByTagName("member");
var n_members = members_nodes.length;
for (i = 0; i < n_members; i++) {
chat_members_div.innerHTML += '<p>' + members_nodes[i].childNodes.nodeValue + '</p>';
chat_members_div.scrollTop = chat_members_div.scrollHeight;
}
mTimer = setTimeout('getOnlineMembers();',2000); //Refresh our chat members in 2 seconds
}
}
}
</script>
HTML page
<body onLoad="javascript:startChat();">
<!--- START: Div displaying all online members --->
<div id="chat_members">
</div>
<!---END: Div displaying all online members --->
</body>
I'm new to ajax and would really appreciate getting help with this.
Thanks!

To troubleshoot this:
-- Use an HTTP analyzer like HTTP Fiddler. Take a look at the communication -- is your page calling the server and getting the code that you want back, correctly, and not some type of HTTP error?
-- Check your IF statements, and make sure they're bracketed correctly. When I see:
if(getMember.readyState == 4 || getMember.readyState == 0){
I see confusion. It should be:
if( (getMember.readyState == 4) || (getMember.readyState == 0)){
It might not make a difference, but it's good to be absolutely sure.
-- Put some kind of check in your javascript clauses after the IF to make sure program flow is executing properly. If you don't have a debugger, just stick an alert box in there.

You must send the xmlhttp request before checking the response status:
function getOnlineMembers(){
getMember.open("GET", "get_chat.php?get_member", true);
getMember.onreadystatechange = memberReceivedHandler;
getMember.timeout = 1000; //set timeout for xmlhttp request
getMember.ontimeout = memberTimeoutHandler;
getMember.send(null);
}
function memberTimeoutHandler(){
getMember.abort(); //abort the timedout xmlhttprequest
setTimeout(function(){getOnlineMembers()}, 2000);
}
function memberReceivedHandler(){
if(getMember.readyState == 4 && getMember.status == 200){
var chat_members_div = document.getElementById('chat_members');
var xmldoc = getMember.responseXML;
var members_nodes = xmldoc.documentElement.getElementsByTagName("member");
var n_members = members_nodes.length;
for (i = 0; i < n_members; i++) {
chat_members_div.innerHTML += '<p>' + members_nodes[i].childNodes.nodeValue + '</p>';
chat_members_div.scrollTop = chat_members_div.scrollHeight;
}
mTimer = setTimeout('getOnlineMembers();',2000); //Refresh our chat members in 2 seconds
}
}
To prevent caching response you can try:
getMember.open("GET", "get_chat.php?get_member&t=" + Math.random(), true);
Check the responseXML is not empty by:
console.log(responseXML);
Also you might need to select the root node of the xml response before selecting childNodes:
var members_nodes = xmldoc.documentElement.getElementsByTagName("member"); //documentElement selects the root node of the xml document
hope this helps

Call MVC action method by javascript but not using AJAX

I have a MVC3 action method with 3 parameters like this:
var url = "/Question/Insert?" + "_strTitle='" + title + "'&_strContent='" + content + "'&_listTags='" + listTags.toString() + "'";
and I want to call this by normal javascript function not AJAX (because it's not necessary to use AJAX function)
I tried to use this function but it didn't work:
window.location.assign(url);
It didn't jump to Insert action of QuestionController.
Is there someone would like to help me? Thanks a lot
This is more detail
I want to insert new Question to database, but I must get data from CKeditor, so I have to use this function below to get and validate data
// insert new question
$("#btnDangCauHoi").click(function () {
//validate input data
//chủ đề câu hỏi
var title = $("#txtTitle").val();
if (title == "") {
alert("bạn chưa nhập chủ đề câu hỏi");
return;
}
//nội dung câu hỏi
var content = GetContents();
content = "xyz";
if (content == "") {
alert("bạn chưa nhập nội dung câu hỏi");
return;
}
//danh sách Tag
var listTags = new Array();
var Tags = $("#list_tag").children();
if (Tags.length == 0) {
alert("bạn chưa chọn tag cho câu hỏi");
return;
}
for (var i = 0; i < Tags.length; i++) {
var id = Tags[i].id;
listTags[i] = id;
//var e = listTags[i];
}
var data = {
"_strTitle": title,
"_strContent": content,
"_listTags": listTags.toString()
};
// $.post(url, data, function (result) {
// alert(result);
// });
var url = "/Question/Insert?" + "_strTitle='" + title + "'&_strContent='" + content + "'&_listTags='" + listTags.toString() + "'";
window.location.assign(url); // I try to use this, and window.location also but they're not working
});
This URL call MVC action "Insert" below by POST method
[HttpPost]
[ValidateInput(false)]
public ActionResult Insert(string _strTitle, string _strContent, string _listTags)
{
try
{
//some code here
}
catch(Exception ex)
{
//if some error come up
ViewBag.Message = ex.Message;
return View("Error");
}
// if insert new question success
return RedirectToAction("Index","Question");
}
If insert action success, it will redirect to index page where listing all question include new question is already inserted. If not, it will show error page. So, that's reason I don't use AJAX
Is there some one help me? Thanks :)

Try:
window.location = yourUrl;
Also, try and use Fiddler or some other similar tool to see whether the redirection takes place.
EDIT:
You action is expecting an HTTP POST method, but using window.location will cause GET method. That is the reason why your action is never called.
[HttpPost]
[ValidateInput(false)]
public ActionResult Insert(string _strTitle, string _strContent, string _listTags)
{
// Your code
}
Either change to HttpGet (which you should not) or use jQuery or other library that support Ajax in order to perform POST. You should not use GET method to update data. It will cause so many security problems for your that you would not know where to start with when tackling the problem.
Considering that you are already using jQuery, you might as well go all the way and use Ajax. Use $.post() method to perform HTTP POST operation.
Inside a callback function of the $.post() you can return false at the end in order to prevent redirection to Error or Index views.
$.post("your_url", function() {
// Do something
return false; // prevents redirection
});
That's about it.

You could try changing
var url = "/Question/Insert?" + "_strTitle='" + title + "'&_strContent='" + content + "'&_listTags='" + listTags.toString() + "'";
to
var url = "/Question/Insert?_strTitle=" + title + "&_strContent=" + content + "&_listTags=" + listTags.toString();
I've removed the single quotes as they're not required.
Without seeing your php code though it's not easy to work out where the problem is.
When you say "It didn't jump to Insert action of QuestionController." do you mean that the browser didn't load that page or that when the url was loaded it didn't route to the expected controller/action?

You could use an iframe if you want to avoid using AJAX, but I would recommend using AJAX
<iframe src="" id="loader"></iframe>
<script>
document.getElementById("loader").src = url;
</script>