I have a website that loads very slow. Today I have done a test with the Timeline included in the Chrome Tools for Developers and I have seen that during about 9 seconds there is a Timer firing in a JS called v8natives.js. I have checked my Wordpress plugins and none of them are loading this library. I don't know why this library is loading and what it is used for. Anyone knows something about this JS?
I have tried also deleting some Javascript calls, deactivate plugins... But nothing happens.
The website is http://www.laparabolica.es
Thanks!
Based on these results:
http://www.webpagetest.org/result/120704_SN_10Z5/1/details/
It would appear that the web server is taking a long time (11.388 seconds in this case) to respond. (See Time to First Byte). I can't see v8natives.js in this equation ( Pretty sure this is something to do with Chrome's Javascript engine).
Since it's taking 11.388 seconds to receive the first byte of your site it can't be any javascript slowing this down...
Related
I wish to create a script that can interact with any webpage with as little delay as possible, excepting network delay. This is to be used for arbitrage trading so any tips in this direction are welcome.
My current approach has been to use Selenium with Python as I am a beginner in web scraping. However, I have investigated some approaches and from my findings it seems that Selenium has a considerable delay (See benchmark results below) I mention other things besides it in this question but the focus is on Selenium.
In general, I need to send a BUY or SELL request to whatever broker I am currently working with. Using the web browser client, I have found a few approaches to do this:
1. Actually clicking the button
a. Use Selenium to Click the button with the corresponding request
b. Use Puppeteer to Click
c. Use Pynput or other direct mouse input manipulation
2. Reverse-engineering the request and sending it directly without clicking any buttons
Now, the delay in sending this request needs to be minimized as much as possible. Network delay is out of our control for the purpose of this optimization.
I have benchmarked the approaches for 1. by opening a page in the standard way you would, either with puppeteer and selenium then waiting for a few seconds. While the script waits, I injected into the browser the following code:
$x('//*[#id="id_demo_webfront"]/iframe')[0].contentDocument.body.addEventListener('click', (data => console.log(new Date().getTime())), true);
The purpose of this code is to log in console the current time when the click is registered by the browser.
Then, I log in my python(Selenium, pynput)/javascript(Puppeteer) script the current time right before issuing the click. I am running on Ubunutu 18.04 as opposed to Windows so my system clock should have good resolution. Additional reading about this here and here.
Actual results, all were run on the same webpage multiple times, for roughly 10 clicks each run:
1. a. ~80ms
1. b. ~10-30ms
1. c. ~5-10ms
For 2., I haven't reliably tested the delay. The idea behind this approach is to inject a function that when fired will send a request that exactly resembles a request that would be sent when the button is clicked. I have not tested the delay between issuing the command to run such an injected function and it actually being run, however I expect this approach to be even faster, basically creating my own client to interact with whatever API the broker has on the backend.
From the results, it is pretty clear that issuing mouse commands seems to be the quickest, but it also is the hardest for me to implement reliably. Also, I seem to find puppeteer running faster across the board, but I prefer selenium in python for ease of development and was wondering if there are any tips and ideas to speed up those delays I am experiencing.
Summary:
Why does Selenium with Python have such a delay to issue commands and can it be improved?
Why does Puppeteer seem to have a lower delay when interacting with the same browser?
Some code snippets of my approach:
class DMMPageInterface:
# These are part of the interface script, self.page is the webpage after initialization
def __init__(self, config):
self.bid_price = self.page.get_element('xpath goes here')
...
# Mostly only the speed of this operation matters, sending the trade request
def bid_click(self):
logger.debug("Clicking bid")
if USE_MOUSE:
mouse.position = (720,390)
mouse.click(Button.left, 1)
else:
self.bid_price.click()
Quite a few questions there!
"Why does Selenium with Python have such a delay to issue commands"
I don't have any direct referencable evidence for this but the selenium delay is probably down to the amount of work it has to do. It was created for testing not for performance. The difference in that is that the most valuable part of a test is that MUST run reliably, if it's slower and does more checks in order to make it more reliable than so be it. You still get a massive performance gain over manual testing, and that's what we're replacing.
This is where you'll need the reliability - and i see you mention this need in your question too. It's great if you have a trading script that can complete an action in <1s - but what if it fails 5% of the time? How much will that cause problems?
Selenium also needs to render the GUI then, depending on how you identify your objects it needs to scan the entire DOM for what you want. Generic locators will logically take longer.
Selenium's greater power is the simplicity and the provided ability synchronisation. There's lots of talk in lots of SO articles about webdriverwait (which is great for in-page script sync), but there's also good recognition of page load wait-times. i.e. no point pushing the button before all the source is loaded.
"Why does Puppeteer seem to have a lower delay when interacting with the same browser?"
Puppeteer works with chrome devtools - selenium works with the DOM. So the two are interacting in a different way. Obviously, puppeteer is only for chrome, but selenium can go to almost any browser. It might not be a problem for you if you don't care about cross-browser capability.
However - how much faster is it really?
In your execution timings, you might also want to factor in the whole life cycle of what you aim to do - namely, include: browser load time, and page render time. From your story, you want to buy or sell something. So, you kick off the script - how long does it take from pressing GO to the end result. You most likely will need synchronisation and that's where puppeteer might not be as strong (or take you much longer to get a handle on).
You say you want to "interact with any webpage with as little delay as possible", but you have to get to the page first and get it in the right state. I wouldn't be concerned about milliseconds per action. Kicking open chrome isn't the quickest action in itself and your overall millisecond test becomes >seconds to minutes.
When you have full selenium script taking 65 seconds and a puppeteer script taking 64 seconds, the overall consideration changes :-)
Some other options to think about if you really need speed:
Try running selenium headless - no gui, less resources, runs faster. (google it) :-)
Try other browsers
If you're all about speed - Drop the GUI entirely. Create an API script or start to investigate the use of performance scripts (for example jmeter or loadrunner).
Another option that isn't selenium or puppeteer is javascript directly in the browser
Try your speed tests with other free web tools like cypress or Microsoft's playwright - they interact with JS in the browser directly and brag about some excellent inbuilt stability features.
If it was me, and it was about a single performant action - my first port of call would be doing it at the API level. There's lots of resources and practice sites out there to help get you started and there's no browser to render, there's barely anything more than the network delay. That's the real way to keep it as a sub-second script.
Final thought, when you ask "can it be improved": if you have some code, there may be other ways to make it faster. Share your script so far and i'm sure folks will love to chip in their 2 cents on how you can streamline it :-)
On some of my websites, Google's Search Console tool and PageSpeed show a Max First Input Delay of 300 seconds on mobile and 900 seconds on PC on my homepages.
Strange phenomenon because even on a development environment where I disable all javascript and css code, the result does not change.
Moreover, I installed on my website the following script, developed by Google: https://github.com/GoogleChromeLabs/first-input-delay
It allows you to send data to Google Analytics on the FID. On my website, I added a line in the code so that the Browser Console returns the FID (console.log(delay);).
And the result is not surprising : Less than 5 or in some cases 10 milliseconds for the browser to respond when you click anywhere on the page.
So I think that google doesn't do a simple click on the page to calculate the FID. By the way, I also know that the FID is calculated by real users on Google Chrome, it can't be done by a bot.
But in this case, how do you identify the element that could take half a second or even a whole second before the browser responds?
Despite my long researches on the internet on this subject, all give the same explanations to make a diagnosis, but this does not help me in my problem.
Since you already tried removing CSS and JavaScript links, you could also try:
Taking a look at any images (img) and see how long those take to load
Taking a look at any iframes in your site and seeing how long those take to load
Checking how long any favicon(s) take to load
Any dns-prefetch or preconnect meta tags.
Taking a look at the Sources tab in DevTools to see what HTTP requests are made while loading
Sorry I couldn't be of more help in terms of specific solutions to your problem — I didn't really have much information on your specific site and what HTTP requests are made while loading it.
I hope this helps.
Im working on a college project thats going well apart from a problem with slowness. It seems to take forever to load.
I have a feeling that im not deploying it properly or something else is slowing it down. Its deployed to a Heroku Hobby server which $7 a month and never has to go to sleep.
How would i properly check through chrome inspect or somethign similar whats causing it to take 20 seconds to see anything?
For me, the page took 9.85s to load the DOM content and 33.91s to fully load. I should also note that I'm in Korea, so it may take slightly longer for me.
Like the others have said, you have far too many script tags. Each is making their own request and they have to wait until the previous request is finished before the next one can fire off.
To drastically speed up the time, I would suggest to concatenate and minify your scripts into as few scripts as possible. Maybe one bundle for third party scripts and another for all of your scripts. Also, cache them if you can.
The networking, and auditing tabs on Chrome DevTools (inspect) can help find your issue, which definitely seems to be the amount of scripts you have loading.
Networking
Audit
We use PhantomJs 2.0 to take screenshots of web pages. We've found that one particular page takes several minutes to process. This page does not appear to have this issue (or at least not of any comparable magnitude) when loaded in Chrome.
I believe that this is because the javascript is hanging/running very slowly. During the hang, Phantom is using a lot of CPU (although only one core). It does not appear to be taking up an abnormal amount of memory. I am fairly confident that javascript is the culprit because I can see from logging that all requests complete quickly, but then after the page loads Phantom hangs for awhile and won't run anything (I think this is because Phantom is all single-threaded so if the page is still running javascript my Phantom script won't run anything).
I'd like to debug and try to understand what part of the JS is taking so long, but I can't figure out how to get at this in Phantom. For example, I can't seem to collect any output from console.profile/console.profileEnd. How can I profile the javascript running in Phantom to find the bottleneck?
I use Phantomas, via grunt-phantomas. It's a tool that integrates with PhantomJS to profile a wide variety of performance-related metrics. Definitely worth checking out. If it doesn't give you exactly what you need, you can look at the source and see how they integrate with PhantomJS and get data out.
I have a one page web app so there is no page refreshing. Sometimes I leave the page open overnight and I come back in the morning and start interacting with it again. Usually I find I have to refresh. Javascript performs incorrectly - edit-in-place loads weird data, ajax calls don't fire... It's nothing to do with the backend, it seems to just be the browser dumping it's memory, or something. There're no sessions involved.
How does Google calendar stay open for 3 days and still fire event alerts?
I have a 'keep alive' call that fires every 5 seconds, in an attempt to keep the browser on its toes, but it hasn't helped. What's the trick? IS there a way to tell the browser to hold everything in memory forever?
(I'm sure this is addressed in numerous places on the web, but I can't figure out what to search for.)
Possible things to look at:
Test on a couple different browsers to see if they have the same problem.
If they do then its almost 100% certain something in your code.
Otherwise its probably something with your current browser and perhaps some interaction with some portion of your code.
Seems trivial but a lot of people over look this in javascript, but make sure you are deleteing/freeing anything you allocate with new.
If you use any 3rd party libraries consider updating them or checking their forums.
Good luck!
When you refresh the page do you need to do any authentication because of a timed-out session or anything ? Because if all you really need to do is hit F5 and you're good to go, I would suggest you create an 'idle' timer in your app that does a window.reload() every hour if there is no interaction (IOW reset that timer each time there is an interaction)
Hope this helps