I want to parse a html-page that unfortunately requires JavaScript to show any content. In order to do so I use a small python-script that pulls the html-code of the page, but after that I have to execute the JavaScript in a DOM-context which seems pretty hard.
To make it even harder I want to use it in a server environment that has no X11-server.
Note: I already read about http://code.google.com/p/pywebkitgtk/ but it seems to need a X-server.
You can simulate a browser environment using EnvJS. However, in order to make use of it, you will have to embed some kind of JavaScript runtime (e.g. Rhino) in your program (or spawn one as an external process).
You could try using Xvfb to have a fake frame buffer, so you won't need to run X11 (though it may be a dependency of Xvfb on your system). Most rendering engines don't have a headless mode, so something like Xvfb is necessary to run them. I used this technique successfully using XULRunner to navigate web pages, though not from python.
I'm still trying to figure this out myself, so take my answer with a grain of salt.
So far, I found http://blog.motane.lu/2009/06/18/pywebkitgtk-execute-javascript-from-python/, which describes the use and the quirks of Pywebkitgtk by someone who has similar needs to what we do.
Later, however, the writer of that blogpost discovered that he can't get it to work with Xvbf, so he hunted some more and found a Qt webkit (possibly in Qt itself, if I understand correctly) http://blog.motane.lu/2009/07/07/downloading-a-pages-content-with-python-and-webkit/. Apparently it's a much better solution than PywebkitGTK.
Naturally, I'll be looking into the other solutions offered here--but I wanted to bring up the Qt solution, because to me, it seems the most likely candidate for what I want to do...and if not, then perhaps it will be for someone else, looking for an answer to this question! :-)
I use VNC or Xvfb for this purpose, combined with Firefox. After experimenting with the two, I settled on XTightVNC. We use it to create screenshots on demand for various test purposes. It's nice to use one of these because you're executing it in an actual browser, same as a user would be (though most users probably won't be using the same OS as your server).
The handy thing about using VNC is that you can connect remotely to set up and test the browser when needed.
This might help: http://code.google.com/p/pyv8/
Related
I don't have a lot of hope for this one, but I have to ask. I am hoping, for didactic purposes, to come up with some means by which a student could load a simple javascript program into a browser, and have it interact with them in the old-fashioned command line manner, where it prints a line and then reads a line of input. This works fine if you use prompt(), but the fact that it creates popups is aesthetically annoying, and today's browsers cheerfully volunteer to stifle the scripts which overuse it. The problem is, prompt() appears to be the only way browser Javascript has of actually pausing a script to wait for input. If we avoid it, that throws us immediately into having to deal with a real-time GUI event input model.
I've been looking for a way to fake it -- to set up some kind of environment in which it is possible for a Javascript method to wait for input and then return when it's given. The best possibility I've got so far is to connect it to a Java applet, but the java applet brand is kind of poisoned now and I doubt people would want to install the plugin. Could there be another way? Worker threads? A browser add-on? Some server-side trick? Does anyone have an idea?
This question is now years old... I wonder if the addition of promises and async/await to Javascript makes this any more possible now?
...I sort of got it to work. If the read-eval-print loop is in an async function and uses await for the line that reads input, you can indeed make an old fashioned linear read-eval-print loop in an event-driven browser environment. But I don't think you can make the main thread wait on input without an async function, except with the grandfathered prompt method (which they're now getting ready to deprecate).
The technical term for what you want is a REPL: a Read-Evaluate-Print-Loop. There are many REPLs out there.
The Best Browser-Based Solution
Use Google Chrome's inbuilt console! It's a Javascript REPL that can also let you interact with a web page. You can access it by using Ctrl+Shift+I on Windows and Unix inside Chrome, or the equivalent command on a Mac (Google it!).
To load a file and play with it, all your students need to do is create a directory structure like this:
project
|--> index.html
|--> javascript.js
make sure index.html has a script tag that points to javascript.js, and then open index.html with Chrome. Voila! You have loaded a Javascript file, and can now play around with it in Chrome.
It's the best solution because you get the full power of the DOM, can do REPL stuff, and is virtually painless - everyone uses Chrome, and your students can even go home and mess around with it completely.
A Browser-Based Alternative
Rather than reinvent the wheel, you can also use Repl.it.
It's a browser-based Javascript REPL website that supports inserting an arbitrary Javascript program, and interacting with its contents. This is the closest you'll probably get to meeting your requirements - it'll be unable to interact with the DOM (obviously), but it'll more than work.
Non-Browser-Based Alternative
If the requirement for a web-based solution can be relaxed, simply using Node.js' inbuilt REPL on a terminal can be more than sufficient.
You could install Node on your lab's computers, and have people play with Javascript in that capacity. There'll be no DOM, but you could certainly have them write functions and algorithms to solve simple problems. Plus, it'd be a good way to introduce them to the fact that Javascript is no longer client-side-bound.
The fact that you'll be interacting with Javascript in an actual terminal, rather than an emulated one in the browser, is another neat bonus.
If I've Misunderstood...
Some of the question comments make it sound like you want a way to be able to interact with terminal utilities using Javascript from a browser. If that is the case, this is impossible.
There is no way for Javascript to evaluate, parse or do anything on the command line that isn't written using Javascript. You cannot expect the equivalent of a bash ls using a browser-based solution - that's because browsers don't have access to your underlying filesystem, which is a good thing. You cannot run sed, awk, grep, etc. for the same reason - Unix utilities are inaccessible to a browser. There are ways to run Unix utilities using Node, of course, but then you will be teaching them how to use Node, rather than how to play with the Unix console.
If all you want, however, is a way to SSH from a browser into a common environment, there are certainly browser-based ways to do that. FireSSH is a Firefox plugin (now also ported to Chrome) that lets people SSH into a common server. They can then do ls, vim, etc. and have it run in the server, with the results piped back to their browser screen. You'll have to think carefully about security in this case, of course, but I think simply giving people user permissions for this server should more than suffice.
Note that FireSSH doesn't use Javascript to parse or do anything - all it is doing is relaying commands you type to a server, having the server execute those commands remotely, and then piping the results back to your screen.
Alternatives to Prompt
I added this after understanding OP's requirements in more detail.
This is a question that has been asked before. I am fond of library solutions, and in 2013, iocream.js was developed for just this sort of browser functionality. You can embed it in a page, and use the jin function to assign values.
If going with a Node.js solution, by far the best approach is to make use of the prompt library. I personally find it very useful for embedding within Node.js applications.
SpiderMonkey is Mozilla's C++-based Javascript engine, and supports a function called readline(). Unfortunately, there doesn't appear to be a mainstream browser implementation.
I've recently gotten hold of a Rexster server with a REST API and I've been wanting to play around with it in an interactive way. I've never had to work with REST before, most of my work has been around SOAP and .NET. I'm able to trivially perform all of the GET operations just by navigating to the correct URL through the browser and reading the response with JSONView on Firefox, but anything requiring PUT and DELETE appears to require actual code.
Given that I'm still prototyping and testing the API, I'd like to have a way to interactively throw a bunch of requests at it and see what happens. I'm sure I could concoct something in Java or C# fairly fast, but I'm guessing there's a better way (which is why I was thinking javascript above) and was hoping you folks might recommend one.
Please advise, thanks!
If you are trying to debug requests and responses, then Fiddler or HttpScoop is great. However, for just interacting with the API, I've found that the Poster plugin for Firefox is the easiest of all. That sounds like what you're describing.
If you are on Windows, then Fiddler is your best option. I believe the Mac has an equivalent called Charles.
Recent versions of SoapUI allow you to interact with REST interfaces.
Maybe it won't affect you, but there's a bug in the current open source version that causes every query to be executed twice - took me some time to discover while testing a service that returns an incrementing number :-)
I'm pretty new to workign with Javascript.
In most languages you can run the code quickly locally on your machine. From what I've seen, in JS you generally only use it via the browser, and so I've been uploading my code an viewing its effects in the browser. This has proven very tiresome. Also, if I mak one error, it seems like my JS/JQuery will just do NOTHING, instead of giving me a useful error, message, which is making it painfully slow to code in.
IS there some way to run JS locally to see that it is working as I go? And then only upload it to the web when I'm mostly done? What ways are there for me to do this? What ways aer there for me to unit test the Javascript locally? Say I have some JAML that should render as <p>HI</p>, how do I run this locally in a unit test?
Thanks for the help,
Alex
EDIT:
Thanks for all the great suggestions. I'll have to take a bit of time and go through them to see which ones best help me in my situation.
Since you're using jQuery, I assume that you actually want to manipulate the various elements on your page. So depending on your specific development enviroment, uploading it each time is probably the way to go anyway. If you can set up a dev enviroment on your local machine (not always possible) then go with that.
As an actual answer to your question, I suggest using Chrome's developer tools, it doesn't just have the console, but an element inspector, and a resource tracker (resource tracker is invaluable when working with JSON and AJAX, since invalid json will fail silently)
As far as I know, the firebug plugin for firefox (dont use it myself) has a similar feature set, so if you're more comfortable with that go with it.
Just remember, as a developer, your development (and debuggin) enviroment is just as important as the code that you are writing.
EDIT: Noticed that you mentioned unit testing. There are several unit testing frameworks out there for JS including one that integrates with firebug called FireUnit. Do a quick google search to find more if you want.
You don't need to upload the JS file to a server to test it. Just write an html and declare the js binding
<script
src="js/yourJSFile.js"
type="text/javascript"></script>
Edit the JS file in your favorite editor and then refresh the page to test it.
For unit testing the best option is Selenium. It allows you to record an interaction with the browser and then play it back.
You can use Firebug with Firefox to debug JS, and Google Chrome has a debugger built-in (use the Tools -> Developer Tools menu).
You can run Javascript from the local file on your machine in your browser, so you can skip the uploading step.
Also, I'd recommend using Firefox/Firebug combo for developing Javascript as it will be very handy, especially for the part you mentioned about not seeing what's going wrong with your code.
Even if you upload your javascript it gets downloaded back to you as soon as you visit the webpage that invoques it. Its run client side always. So stick to local and use firebug as the others have said. Google`s developer tool is quite nice too.
In the browser if you open the developer tools, follow the following steps:
1) Navigate to sources
2) Under sources, click snippet and open run.js
3) You can use run.js to write as much code as you want and run it locally only to see if your code is working or not (it will give you output on the console)
4) Also you can get used to some keyboard shortcuts to make it faster for you.
5) For small javascript codes, you can navigate to console and run your code there
If you want to do unit testing with Javascript there are extension of Firebug that can help you with that. I haven't try any of them, so I can't really tell you which one are worth considering, but you can easily find them if you search for the keyword "Firebug unit testing" on Google.
What seems to be comming on top is FireUnit. You can find some information about how it works here.
Consider Spider Monkey, which is a javascript engine separate from a browser. If what you are developing does not involve rendering to a webpage or can be separated from the rendering code (good practice!), then this could be useful.
I prefer Chrome to Firefox and I just found Web Server for Chrome.
It's just a Google App that quickly sets up a web server for you and will be set up anywhere you are logged into Chrome. It only allows file access to your current devices, or if you specify, other devices only on the current LAN.
You just point it to the directory with your index.html file and type http://127.0.0.1:8887 in your browser.
Additionally to the answers given you can use Jasmine for automated testing.
A tutorial that seems to help get started with automated testing on Jasmine is provided by Evan Hahn.
I used it and for me it works like a charm. Especially if test driven development is what you are going for!
Firefox has the Sandbox and evalInSandbox(). Chrome has sandboxed execution in their content scripts (they call it isolated execution). I'm looking for the same thing in an IE browser extension.
I can load a javascript file, then call evalScript(), but the code executes in the same environment as javascript that exists on the page. I need a way to run my library (which includes and is based on jQuery) in an sandboxed/isolated environment, but still allow it to modify the DOM as if it were running on the page.
Jint looks promising, but cannot currently evaluate jQuery. (They can parse it.)
How can I do this?
Looks like you are in a pickle with this one.
Some Microsoft people say that IE doesn't offer this functionality
Some who say they have been able to pull this off hold it close and protect it as their secret sauce
"Professional" IE extension shops, when contacted about building what you are talking about turn down the job
I really wish I had better news but it looks like it might take a small miracle to accomplish what you are looking for... or maybe a lot of money. : )
Your best bet is probably going to be finding one of the few who claim they have been able to do it and pay them a lot to share the secret or reconsider why you really want what you want and see if you can't accomplish it another way.
You might find it worthwhile looking at the Microsoft Web Sandbox at the Live Labs:
http://websandbox.livelabs.com/
Although it's more fully fledged and related to mashups and the like it might point you in the right direction.
Thing you are looking for is ActiveScript engine (http://en.wikipedia.org/wiki/Active_Scripting)
I can`t find any usefull links now, MSDN contains only interfaces definition (http://msdn.microsoft.com/en-us/library/ccd0zt2w(v=vs.85).aspx)
Try to Google "Active Scripting" (not "Action scripting"!)
You need to implement interface IActiveScriptHost in your extension, create an object of «JScript», call SetSite and pass your host object, then you can load your js code into this engine and run it.
This technique is a quite difficult due to lack of documentation about it.
If you are still interested in this, I can send you some examples in C++/ATL.
Can you elaborate on your goals a bit?
I don't think you've clearly defined what you're trying to accomplish. If your code has the ability to modify the page's DOM, then it effectively is executing in the context of the page. It can create new script blocks to perform any unsafe or unreliable action that it could accomplish were it operating in the same execution environment.
IE doesn't offer a feature to do what you're asking, and I'm not convinced that the Firefox and Chrome features work the way that you expect that they do.
There are lots of widgets provided by sites that are effectively bits of JavaScript that generate HTML through DOM manipulation or document.write(). Rather than slow the browser down even more with additional requests and trust yet another provider to be fast, reliable and not change the widget output, I want to execute* the JavaScript to generate the rendered HTML, and then save that HTML source.
Things I've looked into that seem unworkable or way too difficult:
The Links Browser (not lynx!)
Headless use of Xvfb plus Firefox plus Greasemonkey (yikes)
The all-Java browser toolkit Cobra (the best bet!)
Any ideas?
** Obviously you can't really execute the JavaScript completely, as it doesn't necessarily have an exit path, but you get the idea.
Wikipedia's "Server-side JavaScript" article lists numerous implementations, many of which are based on Mozilla's Rhino JavaScript-to-Java converter, or its cousin SpiderMonkey (the same engine as found in Firefox and other Gecko-based browsers). In particular, something simple like mod_js for Apache may suit your needs.
If you're just using plain JS, Rhino should do the trick. But if the JS code is actually calling DOM methods and so on, you're going to need a full-blown browser. Crowbar might help you.
Is this really going to make things faster for users without causing compatibility issues?
There's John Resig's project Bringing the Browser to the Server: "browser/DOM environment, written in JavaScript, that runs on top of Rhino; capable of running jQuery, Prototype, and MochiKit (at the very least)."