I have to execute code of a web page (action of uploading files, a button for a popup, ..) without the need to open the browser, automate it in a code (JavaScript for example). Do you know if its possible and how?
Thank you
Learn how to use Selenium... You can use it in many languages including Node.JS (https://www.npmjs.com/package/selenium-webdriver).
I think program still needs to open web-browser or atleast start it in background but seems this is what you are looking for.
Related
I have some user tasks on the browser that I want to automate. These tasks involve page load/re-load. I am wondering if it is possible to write a javascript code using browser capabilities (like dev tool / local storage) to perform this.
Is it possible to write an extension to do that ?
My flow is the following :
Be on a page that has multiple listings of my own
Loop through each listing to modify it.
Modifying each listing involves
clicking on a button "Modify", which brings user to https://xxxxxx/ modify.
Then do a bunch of clicks on that page (easy).
Then click on confirm which bring user to another page https://xxxxxx/ modify/confirmation.
Please note I am not looking for solutions that involve automation frameworks like Puppeteer or Selinium. I need to have javascript code or chrome extension.
Looking forward for your help
searched the web for two weeks no solutions
So I have this web app http://goo.gl/qxqhfm and I would like to save the progress (all client side) and restore it next time user opens it, but restore should run without JavaScript, that is users without JavaScript support should be able to load their content.
The app is a bit buggy, e.g. you cant write on orange note before you move it and you cant move the purple one before you enter image url, I'm working on it.
Edit:App is tested only on Chrome
Edit:Is it even posible, to do it wihout JS, and if is, then what language do I use and how is it usualy done?
While that is possible, it will be very difficult.
Using the server-side language of your choice, and honestly, any of them could do it, you would simply load the data that was saved and embed it on the page.
If you want to render it, as if the JavaScript ran, you might want to look at node.js which is a server-side JavaScript engine and see if you can generate the needed HTML. PHP also has some kind of HTML tools to generate markup, but I know very little about it.
All in all, I really doubt it is worth the effort. Who is running without JavaScript in today's world?
This is part of a project I am working on for work.
I want to automate a Sharepoint site, specifically to pull data out of a database that I and my coworkers only have front-end access to.
I FINALLY managed to get mechanize (in python) to accomplish this using Python-NTLM, and by patching part of it's source code to fix a reoccurring error.
Now, I am at what I would hope is my final roadblock: Part of the form I need to submit seems to be output of a JavaScript function :| and lo and behold... Mechanize does not support javascript. I don't want to emulate the javascript functionality myself in python because I would ideally like a reusable solution...
So, does anyone know how I could evaluate the javascript on the local html I download from sharepoint? I just want to run the javascript somehow (to complete the loading of the page), but without a browser.
I have already looked into selenium, but it's pretty slow for the amount of work I need to get done... I am currently looking into PyV8 to try and evaluate the javascript myself... but surely there must be an app or library (or anything) that can do this??
Well, in the end I came down to the following possible solutions:
Run Chrome headless and collect the html output (thanks to koenp for the link!)
Run PhantomJS, a headless browser with a javascript api
Run HTMLUnit; same thing but for Java
Use Ghost.py, a python-based headless browser (that I haven't seen suggested anyyyywhere for some reason!)
Write a DOM-based javascript interpreter based on Pyv8 (Google v8 javascript engine) and add this to my current "half-solution" with mechanize.
For now, I have decided to use either use Ghost.py or my own modification of the PySide/PyQT Webkit (how ghost works) to evaluate the javascript, as apparently they can run quite fast if you optimize them to not download images and disable the GUI.
Hopefully others will find this list useful!
Well you will need something that both understands the DOM and understand Javascript, so that comes down to a headless browser of some sort. Maybe you can take a look at the selenium webdriver, but I guess you already did that. I don't hink there is an easy way of doing this without running the stuff in an actually browser engine.
many webpages use onload JavaScript to manipulate their DOM. Is there a way I can automate accessing the state of the HTML after these JavaScript operations?
A took like wget is not useful here because it just downloads the original source.
Is there perhaps a way to use a web browser rendering engine?
Ideally I am after a solution that I can interface with from Python.
thanks!
The only good way I know to do such things is to automate a browser, for example via Selenium RC. If you have no idea of how to deduce that the page has finished running the relevant javascript, then, just a real live user visiting that page, you'll just have to wait a while, grab a snapshot, wait some more, grab another, and check there was no change between them to convince yourself that it's really finished.
Please see related info at stackoverflow:
screen-scraping
Screen Scraping from a web page with a lot of Javascript
I'm connecting to a web site, logging in.
The website redirects me to new pages and Mechanize deals with all cookie and redirection jobs, but, I can't get the last page. I used Firebug and did same job again and saw that there are two more pages I had to pass with Mechanize.
I took a quick look at the pages and saw that there is some JavaScript and HTML code but couldn't understand it because it doesn't look like normal page code. What are those pages for? How they can redirect to other pages? What should I do to pass these?
If you need to handle pages with Javascript, try WATIR or Selenium - those drive a real web browser, and can thus handle any Javascript. WATIR Classic requires either IE or Firefox with a certain extension installed, and you will see the pages flash on the screen as it works.
Your other option would be understanding what the Javascript on the offending page does and bypassing it manually, but that seems onerous.
At present, Mechanize doesn't handle JavaScript. There's talk of eventually merging Johnson's capabilities into Mechanize, but until that happens, you have two options:
Figure out the JavaScript well enough to understand how to traverse those pages.
Automate an actual browser that does understand JavaScript using Watir.
what are those pages for? how they can redirect to other pages. what should i do to pass these?
Sometimes work is done on those pages. Sometimes the JavaScript is there to prevent automated access like what you're trying to do :). A lot of websites have unnecessary checks to make sure you have a "good" browser, so make sure that your user_agent is set to something common, like IE. Sometimes setting the user_agent to look like an old browser will let you get past without JavaScript.
Website automation is fun because you have to outsmart the website and its software developers, using multiple strategies. Like the others said, Watir is the best tool for getting past JavaScript at the moment.