Develop Reference

Missing data in PhantomJS screenshots

Im working in a script that get an screenshot of a website every day. I already did it for other sites and it worked correctly but for the first time i have the next problem... my phantomjs script capture almost all the data in the website, but not all (in fact it doesn't print the most important for my case).
Until now i was using this simple script adapted:
var page = require('webpage').create();
page.open('http://www.website.com', function() {
setTimeout(function() {
page.render('render.png');
phantom.exit();
}, 200);
});
But when i run the same script for this site its losing some data. Take the screenshot but miss the prices...
Screenshot of the site with phantomjs
After exploring a bit i saw that if i make a DOM capture (for example using a PHP Simple HTML DOM parser) i can get most of the data but not the prices.
$html = file_get_html('https://www.falabella.com.ar/falabella-ar/category/cat10178/TV-LED-y-Smart-TV');
$prods = $html->find('div[class=fb-pod-group__item]');
foreach ($prods as $prod) {
// For example i can get the title
$title = $prod->find('h4[class=fb-responsive-hdng-5 fb-pod__product-title]',0)->plaintext;
// But not the price
$price = $prod->find('h4[class=fb-price]',0)->plaintext;
}
Exploring the console log i found the javascript objects where these values are. If i return the object fbra_browseProductListConfig.state.searchItemList.resultList[0].prices[0].originalPrice;
i see the price of the first product and so on and so on...:
Console log of the site
also i can get it with a phantomjs script like this:
var page = require("webpage").create();
page.open("https://www.falabella.com.ar/falabella-ar/category/cat10122/Cafeteras-express", function(status) {
var price = page.evaluate(function() {
return fbra_browseProductListConfig.state.searchItemList.resultList[0].prices[0].originalPrice;
});
console.log("The price is " + price);
phantom.exit();
});
In other posts (like this) i read about changing the timeout intervals but its not working for me (i tried all the scripts shared in the quoted post). The problem is not that the website doesn't fully charge. But it seems that this data (the prices) is not printed in the DOM. I even downloaded the full site from the terminal with wget command and the prices are not there o_O.
Edited
When i execute the script i get the next errors:
./phantomjs fala.js
ReferenceError: Can't find variable: Set
https://www.falabella.com.ar/static/assets/scripts/react/vendor.js?vid=111111111:22
https://www.falabella.com.ar/static/assets/scripts/react/vendor.js?vid=111111111:1 in t
https://www.falabella.com.ar/static/assets/scripts/react/vendor.js?vid=111111111:22
https://www.falabella.com.ar/static/assets/scripts/react/vendor.js?vid=111111111:1 in t
https://www.falabella.com.ar/static/assets/scripts/react/vendor.js?vid=111111111:22
https://www.falabella.com.ar/static/assets/scripts/react/vendor.js?vid=111111111:1 in t
https://www.falabella.com.ar/static/assets/scripts/react/vendor.js?vid=111111111:1
TypeError: undefined is not an object (evaluating 't.componentDomId')
https://www.falabella.com.ar/static/assets/scripts/react/productListApp.js?vid=111111111:3
https://www.falabella.com.ar/static/assets/scripts/react/vendor.js?vid=111111111:22
Maybe the problem is there because the script "productListApp.js" executes the prices?

How to calculate page load time of a website which includes the total time and not just document.ready() time? [duplicate]

This question already has answers here:
Chrome: API for performance data
(2 answers)
Closed 4 years ago.
I was trying to write a function either in Javascript or in Python using Selenium to calculate the page load time of a website. document.ready() will only give the DOM load time but there might be some AJAX calls which cannot be detected using document.ready().
There is even an extension in chrome web store named 'Page Load Time', which will calculate the total time, as per my requirements. How do I replicate same kind of functionality?

You can use load like as follows.
$(window).load(function() {
//code in here
});
See jQuery docs here. Also, another answer that will show you how to set up a page timer can be found here.

driver.execute_script("return $.active == 0") should help you.
$.active returns the number of active Ajax requests. Link

You can try it with selenium and with execute_script method, we will get information from window.performance.timing, it will return the milliseconds, but if we divide it by 1000 we will get the seconds of the loaded page.
from selenium import webdriver
driver=webdriver.Chrome()
driver.get("https://example.com")
load_time = driver.execute_script(
"""
var loadTime = ((window.performance.timing.domComplete- window.performance.timing.navigationStart)/1000)+" sec.";
return loadTime;
"""
)
print(load_time)
Output
# something like this
0.803 sec.

Reload updated java<script> code without fully reloading the html page

I am developing a single page web application, that has many different features and forms. When developing a deep (I mean something that is not on the home page) feature, I go through this cycle:
develop the code, editing classes and functions
refresh the whole page
clicking all the way till I get to the part that I need to test (that adds up to about a minute sometimes)
testing the new code
back to the (1) code editor doing updates
doing about 15 minor edits, can take a frustrating 30 minutes of repeated reloading and clicking
Is there any plugin, piece of javascript, or method, that allows to reload the updated javascript without reloading everything, so one can skip the 2. and 3. from the cycle above and continue doing live tests?
If there's no such thing, I am planning on developing a little javascript plugin that will reload the scripts, and probably with socket.io connection to a backend node.js server that will watch the files for any updates and push the load events to the browser.
So, I am interested in any idea about this, any thing that I should take into consideration when writing the plugin.
Thanks : )

You could do something like this.
function LoadMyJs(scriptName) {
var docHeadObj = document.getElementsByTagName("head")[0];
var dynamicScript = document.createElement("script");
dynamicScript.type = "text/javascript";
dynamicScript.src = scriptName;
docHeadObj.appendChild(newScript);
}
Call the LoadMyJs function on page load
<body onLoad="LoadMyJs()">
Then reload with the click of a button (or from your console)
<input type="button" name="reloadjs" value="Reload JavaScript" onclick="LoadMyJs('my_live_loading_script.js')">
This could be simplified using e.g jQuery
Thanks to:
http://www.philnicholas.com/2009/05/11/reloading-your-javascript-without-reloading-your-page/

Here's what I came up with: a Node.js module that watches for changes in .js & .coffee scripts, and pushes the changes to the browser upon editing the files.
It works standalone, even if you are developing on filesystem file:/// without using a web server.
It works with any framework, just launch the standalone script and point it to your js/ directory.
It has an express.js helper, that make it run using the same server instance.
It is as easy as
adding a single line of <script> tag to your existing code, and
running the live script, pointing it to the html root.
code: 🐱/etabits/live.js

That's may be not the best answer but for local developments I use that firefox plugins:
https://addons.mozilla.org/en-US/firefox/addon/auto-reload/
This reload the css, js or anything present in a directory
For dev which really needs to be remotely , I use that small js code you can adapt for reloading js.
function refreshCss(rule){
if (rule == null)
rule = /.*/;
var links = document.getElementsByTagName("link");
for(var i=0;i<links.length;i++)
{
if (!links[i].href.match(rule))
continue;
if (! links[i].href.match(/(.*)time=/)){
if (links[i].href.match(/\?/))
var glue = '&';
else
var glue = '?';
links[i].href += glue+"time="+new Date().getTime();
}
else{
links[i].href.replace(/time=\d+/, "time"+new Date().getTime());
}
}
if (!no_refresh)
{
setTimeout(function(){refreshCss(rule)}, 5000);
}
};
// and then call it refreshCss("regex to match your css, or not"); var no_refresh=false;
Edit: this is a version with "setTimeout", but you can easily made a "keypress" version of it

Replace with dynamic script.
function LoadMyJs(scriptName)
{
var docHeadObj = document.getElementsByTagName("head")[0];
var dynamicScript = document.createElement("script");
dynamicScript.type = "text/javascript";
dynamicScript.src = scriptName;
docHeadObj.appendChild(dynamicScript);
}

How to measures script execution and *parsing* time?

As far as I know, scripts are downloaded and executed synchronously in javascript.
Hence if we write the following code:
<script type='text/javascript'>console.time('core')</script>
<script type='text/javascript' src="guicore.js"></script>
<script type='text/javascript'>console.timeEnd('core')</script>
we'll see in console total time for download, parse and execute js.
How we can exclude parsing time? Just add similar file, but with all code commented out. More or less, this technique should work.
The problem is this just doesn't work =)
I optimized that code, reduce execution time from 90ms to 25ms, but see the same ~100±10ms time for Chrome and ~160±15ms for Firefox.
Ok, I know I could use profiler, but the question is: "how to measure js parsing time correctly" and what did I measured btw. Research.reverse-engineering is very fun, but maybe there's someone who knows that field in depth.

You cannot accurately measure script parse time independent of execution time using web APIs alone. Different browsers have different strategies for when they do parsing, and some of them will parse progressively as the script is executed, or even do "partial parsing" in cases where it's assumed a block of code is unlikely to immediately be executed (e.g. a function that is not an IIFE, see some details in the optimize-js README).
Using separate <script> tags is the most accurate way to at least capture both parsing and execution time. The only modification I would make to your snippet is this:
<script>
performance.mark('start');
</script>
<script src="myscript.js"></script>
<script>
performance.mark('end');
performance.measure('total', 'start', 'end');
</script>
Note the use of the high-precision User Timing API which, as an added bonus, will show visualizations in the Chrome/Edge/IE dev tools (and tools like Windows Performance Analyzer and WebPageTest if you're so inclined).
Technically the 3rd <script> is not necessary, as you can just append the mark/measure to the end of the 2nd script. But the 1st <script> is certainly necessary to capture all parse time. You can verify in the dev tools that the marks/measures encompass all initial parsing and execution time.

I know this is kind of an old question but I came across it while looking for a solution to this myself. You can use the dev tools in the browser of your choice to look at this but if you'd like to do it in code this is the method I ended up using.
The scriptLoadParseDuration function below will take a URL to a .js file, place it into a <script> element, and log the load/parse duration to the console.
Keep in mind that this will execute the <script> you are profiling within the current DOM context. So in the example below: jQuery is still accessible in the global scope even though the script was removed. The script could be extended to do all of this in an <iframe> to isolate it though.
function scriptLoadParseDuration(url) {
var start;
var script = document.createElement('script');
// <script> must be attached to the document to actually load the file
document.querySelector('html').appendChild(script);
// Calculate load/parse duration once script has loaded
script.addEventListener('load', function scriptLoad() {
// Calculate load/parse duration
console.log('Duration: ' + (Date.now() - start) + 'ms');
// Remove <script> from document
script.parentElement.removeChild(script);
}, false);
// Get current time in milliseconds
start = Date.now();
// Setting the `src` starts the loading. Math.random is used to make sure it is an uncached request
script.src = url + '?' + Math.floor(Math.random() * 9e9);
}
var url = 'https://code.jquery.com/jquery-3.0.0.min.js';
scriptLoadParseDuration(url);
Here is an example showing that jQuery is still in the global scope after the <script> removal.
function scriptLoadParseDuration(url) {
var start;
var script = document.createElement('script');
console.log('`jQuery` before attaching: ' + typeof jQuery);
// <script> must be attached to the document to actually load the file
document.querySelector('html').appendChild(script);
// Calculate load/parse duration once script has loaded
script.addEventListener('load', function scriptLoad() {
// Calculate load/parse duration
console.log('Duration: ' + (Date.now() - start) + 'ms');
console.log('`jQuery` once attached: ' + typeof jQuery);
// Remove <script> from document
script.parentElement.removeChild(script);
console.log('`jQuery` after detach: ' + typeof jQuery);
}, false);
// Get current time in milliseconds
start = Date.now();
// Setting the `src` starts the loading. Math.random is used to make sure it is an uncached request
script.src = url + '?' + Math.floor(Math.random() * 9e9);
}
var url = 'https://code.jquery.com/jquery-3.0.0.min.js';
scriptLoadParseDuration(url);

Open up Chrome and open the developer tools, the go to the "Timeline" tab. If you press the record button (filled in circle, bottom left) then reload the page it'll give you a fairly detailed timeline, broken down into specific types of activity (Send Request, Parse, Evaluate), timed down to the microsecond.

Very old question with a relatively new answer.
Date.now() returns a timestamp with millisecond accuracy. For an application to run at 60FPS, it must update the frame every 16ms. Our millisecond meter may not be accurate enough.
Introducing the Performace API in modern JS browsers, this allows for floating-point timestamps accurate to the microsecond.
Instead of Date.now() use window.performance.now() for measurements, there's a good guide on using the Performance API on HTML5Rocks.

Chrome DevTools actually has a hidden flag that shows V8 Parse and Compile time!
https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/javascript-startup-optimization#parsecompile
Outcome looks like this:
A quick guide is also available in the doc in the blue section below:
After you enable the feature, you can profile a page and then click on "Bottom-Up" tab on the Performance tab in Chrome DevTools, and then make sure you "Group by Activity", and you should see the Compile and Parse time now.
Enjoy!

How can I delay the execution of a script block until after an external script has loaded?

I'm trying to dynamically insert and execute a couple of scripts, and I think I'm hitting a race condition where the second is trying to execute before the first is loaded.
The project I'm working on has an unusual requirement: I am unable to modify the page's HTML source. It's compiled into an app for localization purposes.
Therefore, I'm unable to insert <script> tags like I normally would to link in JavaScript files.
It turns out that the client wants to use a hosted web font, so I decided to build and append the two required <script> tags dynamically in an already-linked JavaScript file.
The <script> blocks are appending correctly in the head of the document, but function in the second block seems to be firing before the external script linked in the first <script> tag is fully loaded, and it's throwing an undefined error.
Here's the relevant piece of code:
document.getElementsByTagName("head")[0];
var tag = document.createElement('script');
tag.setAttribute("src", "http://use.typekit.com/izj3fep.js");
document.getElementsByTagName("head")[0].appendChild(tag);
try {
Typekit.load(); // This is executing too quickly!
} catch(e){
console.log("Hosted fonts failed to load: " + e);
}
I tried moving the try block to the window.onload event, but that fires before any of this code is called.
I guess I could dynamically load jQuery and then use it's ready event, but that seems pretty heavy-handed. I'm hesitant to pull in a library on this project, as the client has a lot of custom JavaScript that could potentially clash with it.
What else can I try?

You need to hook into the script element's onload event and execute your code there:
document.getElementsByTagName("head")[0];
var tag = document.createElement('script');
tag.onload = onTagLoaded;
tag.setAttribute("src", "http://use.typekit.com/izj3fep.js");
document.getElementsByTagName("head")[0].appendChild(tag);
function onTagLoaded() {
try {
Typekit.load(); // This is executing too quickly!
} catch(e){
console.log("Hosted fonts failed to load: " + e);
}
}

You can load it with yepnope ( http://yepnopejs.com/ ). I know it's a library, but it's very light (free if your client is already using modernizr). It's well worth it. Hopefully the client doesn't have another yepnope function, and you don't have to worry about the clash.

Are you using jQuery? If not, I highly recommend it. It'll make your life so much easier:
$.getScript('http://use.typekit.com/izj3fep.js', function(data, textStatus){
try {
Typekit.load(); //executes properly now!
} catch(e) {
console.log("Hosted fonts failed to load: " + e);
}
});

Combining the scripts into one big seems to be the easiest solution.