Is there a node / npm module which would scrape the contents of a web page or a given URL and execute and evaluate it's JavaScript as a real-life web browser, including XMLHttpRequest object and alike?
I thought that scraperjs was the solution, but sadly it isn't.
All help very appreciated!
Headless browsers based on rendering engine:
Chrome (blink) => Playwright or Puppeteer
Firefox (gecko) => Playwright
Safari (webkit) => Playwright
Puppeteer is the official automation tool for Chrome. It only supports Chrome/Chromium browsers, (with experimental support for Firefox).
Playwright is an alternative to Puppeteer, with support for more browsers, written by the original creators of puppeteer, who jumped ship from Google to Microsoft. It supports Chrome, Firefox and Safari browsers
Previous versions of my answer, included PhantomJS, SlimerJS,CasperJS and Nightmare all of which are no longer maintaned. For modern browsers, just use one of the above
Related
In a .NET Framework 4.6.2 Windows App, I am attempting to run some WebRTC capabilities, in JavaScript, within the WebView control. The control is from Microsoft.Toolkit.Forms.UI.Controls.Webview (v6.1.1).
When I try to use RTCPeerConnection.AddTransceiver(), or RTCPeerConnection.GetTransceivers(), I receive the error:
Object doesn't support property or method 'addTransceiver'
When run directly within Edge, the code executes as expected.
Why the difference, and how can I see the exact API that is used within the WebView control context?
Which version of Edge browser are you using? I assume you're using Edge Chromium so the APIs can run well in the browser.
From the browser compatibility of RTCPeerConnection.addTransceiver(), we can see that it only supports Edge Chromium. Webview control uses EdgeHTML as the rendering engine which is Edge Legacy engine, so you can't use the APIs in WebView. I also tried RTCPeerConnection.GetTransceivers() and it doesn't support Edge Legacy as well.
If you want to use the APIs in Microsoft Edge webview control, I suggest you to use WebView2 which uses Microsoft Edge Chromium as the rendering engine to display the web content in native applications.
Firefox has the JSShell for SpiderMonkey. Do the other major browsers Chrome and Safari have a command-line interface to their javascript engines? If so, this would be a great lightweight way to test javascript to ensure that it runs cross-browser without having to open a full browser.
I want to develop a browser plugin for RTSP streaming on web browser, I read about NPAPI, which can execute native code(C++). But Google Chrome will not support NPAPI in future and no guaranty about Firefox also.
Please somebody can suggest me to "How can I develop plugin which can execute native code(OR any other language) using other framework instead of NPAPI?".
How can I add RTSP Streaming support for web browsers?
I'm afraid you cannot implement universal plugin at the moment. It's possible to write extensions for Chrome and Firefox which support TCP and UDP sockets (Chrome API, Mozilla API ). No such thing for Microsoft Edge or Explorer or Safari as far as I know.
If using different streaming protocol is an option, consider WebRTC. Its support is still not very good, but at least WebRTC is an official standard and has better chances of being adopted.
If you plan to develop an extension for the major browsers (IE, Firefox, Chrome and Safari) then I'd suggest looking into the kango framework. It takes a common code base and converts it into plugins for 3 of the 4 major browsers (for IE you'll have to contact them)
I am using camera with getUserMedia and I get a few errors I want to fix. The thing is Visual Studio only allows me to debug JavaScript (I mean hitting breakpoints) with IE, and IE does not support getUserMedia.
If you want to debug in a browser other than Internet Explorer then you will need to use the Developer Tools of that individual browser and set your breakpoints accordingly. Alternatively you could use console.log() and/or console.warn() to get the state of your objects in code. Visual Studio does not and cannot understand the implementations of different JavaScript and rendering engines (V8, Blink, Gecko etc.) - nor should it ever attempt to do so. These engines move so rapidly with six week release cycles the IDE would be a totally misleading proxy for the actual the platform you're supposed to be testing against.
Both Firefox and Chrome allow you to retain console contents beyond a page reload/navigation and there are tools like Browser Link in Visual Studio will allow you to test in multiple browsers simultaneously - inspecting the console results afterwards for errors (interactions will have to be tested individually in each browser).
If you are testing mobile devices Adobe Shadow that would potentially allow you to test on multiple tablets and/or phone simultaneously whilst using Dev Tools (I believe Chrome) on your laptop or desktop. Otherwise desktop browsers dev tools allow you to debug page running on a connected mobile device using the same browser, or - in the case of Firefox - also Chrome instances on the device.
It is well worth getting to know browser dev tools for front end debugging and troubleshooting, as these are much more powerful for this purpose than those provided with Visual Studio. The free CodeSchool course Discover DevTools (focussed on the Chrome DevTools and sponsored by Google/Chrome) may help you discover some of the features of client developer tooling, and many of the interfaces and techniques will be broadly applicable to other browser dev tools.
If you want to test on devices that are not available to you, browsers that you cannot install on your system (e.g. Safari on Windows) or simply a newer or legacy version which you cannot install alongside the one already installed locally then there are services like Browserstack which allow you to do so in hosted virtual machines. There is a Visual Studio extension available for BrowserStack to help connect to projects running locally.
I'm looking for a way to test an app or a website, which is not based on angularjs, on real browsers and real devices. I found CasperJS, but CasperJS only runs with headless browsers.
Are there test-runner/testing-framework combinations out there which support:
grunt
all major browser (Chrome, Firefox, Safari, IE)
headless browsers (PhantomJS)
(optional) simulate touch gestures
I would choose protractor. I know its main target is angular, but it works fine without angular as well. It has nice shortcut features. We use it to test non-angular sites and it looks stable enough. Otherwise it is based on webdriverjs, and it can work with phantomjs as well.
There is a good starter setup: https://github.com/angular/angular-seed
And this page can help to tune it for your requirements:
http://ng-learn.org/2014/02/Protractor_Testing_With_Angular_And_Non_Angular_Sites/
And the main urls:
https://github.com/angular/protractor
https://code.google.com/p/selenium/wiki/WebDriverJs
I didn't test the phantomjs part, but google says it should be fine.
About main browsers Protractor can connect to Saucelabs as well.
http://www.saucelabs.com/
And they have 301 different Device/OS/Browser Platforms at now. https://saucelabs.com/platforms
Selenium is certainly the most widely used test automation tool that meets your requirements. I personally used Sahi most of the times which does the same but afaik has no grunt integration.