I'm developping a NodeJS application, and sometimes I get my CPU usage at 100%, with no memory variation (I check this with pm2 monit) and with my Event Loop blocked by this operation.
So I remember that browsers do something interesting when a script seems to have a problem, and they show a popup like: "Do you want to continue running script on this page or stop it".
Is there a way to have the same behavior with NodeJS? I want to detect an operation which takes too many time, and trigger something that I can catch by pm2 to force restart, or (better) by node-inspector where it could break on the bad script and so I could understand what is the problem.
Thanks a lot.
your can stop the node server or check the task list for windows and stop the services whatever services you want.
Related
I have a web app that basically can be looked at as a messaging system - people can submit a message, and someone else can receive it. This all works via AJAX and the Javascript front end interacts with a PHP backend. All of this works completely fine and there's no issue.
I have also implemented the Notification system that sends the desktop or android app a push notification when a new message is received. This also works completely fine.
The notification system works using setTimeout to periodically check the PHP AJAX system. But this is where the deal breaking issues arise.
When out of focus on Android, settimeout becomes completely unreliable - sometimes it will work, sometimes it will not work at all, sometimes it is very late.
To fix this, I then moved everything into a support worker as I thought that would work independent of the browser being focused, but this is even worse - seems it is even less consistent than just running settimeout on the browser.
So is there some way to rectify this? Is there some special directive within the supportworker that I can put so that it does not sleep?
thank you.
This API does not guarantee that timers will run exactly on schedule. Delays due to CPU load, other tasks, etc, are to be expected which may result in the actual delay may be longer than intended.
You can read more about the setTimeout delay and reasons it may be delayed longer here on MDN.
If you need immediate messaging like capabilities you should look into an architecture and protocol meant for such. Something like WebSockets with events would better suffice for this use-case.
I wish to create a script that can interact with any webpage with as little delay as possible, excepting network delay. This is to be used for arbitrage trading so any tips in this direction are welcome.
My current approach has been to use Selenium with Python as I am a beginner in web scraping. However, I have investigated some approaches and from my findings it seems that Selenium has a considerable delay (See benchmark results below) I mention other things besides it in this question but the focus is on Selenium.
In general, I need to send a BUY or SELL request to whatever broker I am currently working with. Using the web browser client, I have found a few approaches to do this:
1. Actually clicking the button
a. Use Selenium to Click the button with the corresponding request
b. Use Puppeteer to Click
c. Use Pynput or other direct mouse input manipulation
2. Reverse-engineering the request and sending it directly without clicking any buttons
Now, the delay in sending this request needs to be minimized as much as possible. Network delay is out of our control for the purpose of this optimization.
I have benchmarked the approaches for 1. by opening a page in the standard way you would, either with puppeteer and selenium then waiting for a few seconds. While the script waits, I injected into the browser the following code:
$x('//*[#id="id_demo_webfront"]/iframe')[0].contentDocument.body.addEventListener('click', (data => console.log(new Date().getTime())), true);
The purpose of this code is to log in console the current time when the click is registered by the browser.
Then, I log in my python(Selenium, pynput)/javascript(Puppeteer) script the current time right before issuing the click. I am running on Ubunutu 18.04 as opposed to Windows so my system clock should have good resolution. Additional reading about this here and here.
Actual results, all were run on the same webpage multiple times, for roughly 10 clicks each run:
1. a. ~80ms
1. b. ~10-30ms
1. c. ~5-10ms
For 2., I haven't reliably tested the delay. The idea behind this approach is to inject a function that when fired will send a request that exactly resembles a request that would be sent when the button is clicked. I have not tested the delay between issuing the command to run such an injected function and it actually being run, however I expect this approach to be even faster, basically creating my own client to interact with whatever API the broker has on the backend.
From the results, it is pretty clear that issuing mouse commands seems to be the quickest, but it also is the hardest for me to implement reliably. Also, I seem to find puppeteer running faster across the board, but I prefer selenium in python for ease of development and was wondering if there are any tips and ideas to speed up those delays I am experiencing.
Summary:
Why does Selenium with Python have such a delay to issue commands and can it be improved?
Why does Puppeteer seem to have a lower delay when interacting with the same browser?
Some code snippets of my approach:
class DMMPageInterface:
# These are part of the interface script, self.page is the webpage after initialization
def __init__(self, config):
self.bid_price = self.page.get_element('xpath goes here')
...
# Mostly only the speed of this operation matters, sending the trade request
def bid_click(self):
logger.debug("Clicking bid")
if USE_MOUSE:
mouse.position = (720,390)
mouse.click(Button.left, 1)
else:
self.bid_price.click()
Quite a few questions there!
"Why does Selenium with Python have such a delay to issue commands"
I don't have any direct referencable evidence for this but the selenium delay is probably down to the amount of work it has to do. It was created for testing not for performance. The difference in that is that the most valuable part of a test is that MUST run reliably, if it's slower and does more checks in order to make it more reliable than so be it. You still get a massive performance gain over manual testing, and that's what we're replacing.
This is where you'll need the reliability - and i see you mention this need in your question too. It's great if you have a trading script that can complete an action in <1s - but what if it fails 5% of the time? How much will that cause problems?
Selenium also needs to render the GUI then, depending on how you identify your objects it needs to scan the entire DOM for what you want. Generic locators will logically take longer.
Selenium's greater power is the simplicity and the provided ability synchronisation. There's lots of talk in lots of SO articles about webdriverwait (which is great for in-page script sync), but there's also good recognition of page load wait-times. i.e. no point pushing the button before all the source is loaded.
"Why does Puppeteer seem to have a lower delay when interacting with the same browser?"
Puppeteer works with chrome devtools - selenium works with the DOM. So the two are interacting in a different way. Obviously, puppeteer is only for chrome, but selenium can go to almost any browser. It might not be a problem for you if you don't care about cross-browser capability.
However - how much faster is it really?
In your execution timings, you might also want to factor in the whole life cycle of what you aim to do - namely, include: browser load time, and page render time. From your story, you want to buy or sell something. So, you kick off the script - how long does it take from pressing GO to the end result. You most likely will need synchronisation and that's where puppeteer might not be as strong (or take you much longer to get a handle on).
You say you want to "interact with any webpage with as little delay as possible", but you have to get to the page first and get it in the right state. I wouldn't be concerned about milliseconds per action. Kicking open chrome isn't the quickest action in itself and your overall millisecond test becomes >seconds to minutes.
When you have full selenium script taking 65 seconds and a puppeteer script taking 64 seconds, the overall consideration changes :-)
Some other options to think about if you really need speed:
Try running selenium headless - no gui, less resources, runs faster. (google it) :-)
Try other browsers
If you're all about speed - Drop the GUI entirely. Create an API script or start to investigate the use of performance scripts (for example jmeter or loadrunner).
Another option that isn't selenium or puppeteer is javascript directly in the browser
Try your speed tests with other free web tools like cypress or Microsoft's playwright - they interact with JS in the browser directly and brag about some excellent inbuilt stability features.
If it was me, and it was about a single performant action - my first port of call would be doing it at the API level. There's lots of resources and practice sites out there to help get you started and there's no browser to render, there's barely anything more than the network delay. That's the real way to keep it as a sub-second script.
Final thought, when you ask "can it be improved": if you have some code, there may be other ways to make it faster. Share your script so far and i'm sure folks will love to chip in their 2 cents on how you can streamline it :-)
I was hoping to set up some sort of script (preferably) or small program that would detect and automatically end/restart Chromium's GPU process when it exceeds a certain memory threshold.
Turning off hardware acceleration prevents the bloating of course, but I was interested in a way to use the feature while keeping it in check, without having to launch the Task Manager > End Process every hour or so.
A bit late to the question, but it's the top Google search hit combining "chrome" "gpu process" and "script". There don't seem to be any answers out there, so here's mine.
At least on Windows, it looks as if every chromium-based product with GPU-assisted rendering spins up a process with --gpu-process in the command-line. This applies to any Electron apps as well, as I see this on my main PC for apps like Slack, Discord and Skype, as well as other downstream Chromium-based browsers. The actual process name will match the product's main image name, so you can differentiate that way.
This should allow one to use any program language that can inspect the command line of running process, find their PID, and monitor the process's memory usage, killing it when it's over some threshold. Search for the expected image name (such as chrome.exe) and check for --gpu-process in the arguments or full command-line.
When you terminate it, it will be automatically restarted ... as long as you don't kill it too often. The chromium code includes a check meant to protect against the gpu-render process "flapping" up and down when it actually crashes. It unlikely to know the difference in an external tool killing the gpu render process and the process crashing due to some issue. Based on what I'm seeing, I believe the control process will stop restarting the gpu-render process if it dies more than once an hour.
I have a macro (Macro Recorder), which runs 24/7 without pause.
I need to view an item in one tab, order a specific quantity in another tab, write a message and repeat.
Can I achieve this result with Puppeteer?
What happens if I do not the close browser (because I don't need to)?
Yes, it is possible to run puppeteer 24/7 without any problems. Simply don't close the browser and keep using it. I have run puppeteer instances myself for weeks without any issues. You can see some statistics of one of the runs here.
The only thing you need to watch out for is error and memory handling. You need to make sure your script does not crash due to some minor JavaScript error. Additionally, you need to clean up any memory (by closing the page or even the browser) that might be leaked due to other kind of errors (network errors, SSL errors, timeouts, ...). See the linked run above to see a full list of errors that might happen during crawling process.
puppeteer-cluster: Pool of puppeteer instances
Depending on the complexity of your use case you might want to use the library puppeteer-cluster (disclaimer: I'm the author), which takes care of these kind of problems and also gives you a simple monitoring overview, which is very handy when running a task for a long time.
The project I'm working on involves a "planning" screen, which is entirely made with backbone.js (the other pages of the app are not).
My issue is that from times to times, chrome freezes and the web view stop responding to any interaction. Sometimes, I can manage to quit chrome itself, but usually the controls does not answer either.
I'm pretty convinced this is related to the js code. It seems to me that when a script takes too much time, or loops indefinitely, Chrome can detect this and interrupt the script. However, since this is not the case, I'm thinking that too many js objects stay in memory.
Whatever the cause is, I would like to know which chrome dev tools can help me here. While I'm not a beginner in js, asides setting breakpoints and calling console.log, I have no idea how to debug JS apps. I'm not opposed to use another browser if the dev tools are more suited.
Thanks a lot for your time !
FTR : This is a rails 3.2.8 app, using mongodb, & Backbone.js 0.9.2. The js code is written in coffeescript. This issue happened on my macbook air 2012 running mountain lion, as well as on the client machine which runs on windows 7. The issue appeared at least on chrome 22 & 23.
Using the Javascript CPU profiler, I was able to find the group of functions that seems to be responsible for the freezing.
I'm still open to any advice/ressource on debugging javascript code.
Make a console.log in the loop and check if its the same point freezing on all chrome versions. There is a limit see Browser Javascript Stack size limit.
Maybe add some code? Because there could be some memory leaks especially with event handlers!
What I would do is the long and weary approach to getting your hands dirty with console.log. --- in this case.
In your case, and THX for the hint by the way (finding the offender with the CPU profiler, I'll try that next time round), I guess the offenders might be some functions "callbacking" themselves .. probably via some sort of event-handling/bubbling/callback-combination.
What happens then usually is that it just doesn't recognize the endless-loop, because the callback-stack is kinda "broken". Therefor it will never throw a browser-error.
If you have any luck, it doesn't kill the browser quick enough to kill the console. I did have that at times, the console killed (remaining silent), the coffeescript files not even loaded in the debugger (I use a JIT-coffee-to-js-translator as in JS-MVC) ... the page frozen or not doing anything ...
So, if indeed you are lucky and the debugger spits out your console.logs you can thereby guess where your unwanted loop is hiding. Just by looking at the repeated order of the output statements.
of course you know you can set breakpoints right? even conditional breakpoints
I'm currently struggling a bit untangling a similar sounding loop - i guess in case of emergency some alert() calls in your code would at least slow the loop down
or maybe some wait calls (bad idea)
my loop is running so fast I loose my console log!