Writing a non-GUI bot using Mozilla Framework - javascript

I'm looking for a way to write a non-GUI bot using Mozilla Framework. The bot should be able to work like normal browser (automatically download relevant JS files, make XMLHTTPRequests, run JS operations, modify DOM), except no GUI will be needed.
I wonder if it is possbile to build XULRunner without X, GTK/KDE (without any GUI dependencies), as I will run the bot on FreeBSD server 6.4.
It may sound a bit weird but I need a bot with capacity to operate like browser, runs JS, modifies DOM, submit forms running on non-GUI environments.
I've looked into other browsers such as Lynx, Links, Hulahop, Chrome V8 engine, WebKit JavascriptCore but yet to find desirable output.
It's a part of school project, thesis. We will use to observe price change of budget airlines and after one year long data collection, we need to deduce pricing strategy and customer behavior. It is a serious Final Year Project.
Any hint or help is greatly appreciated! Thank you in advance!
Regards.

You should be able to make progress with selenium. It's a record/test/play tool but its core is manipulating the DOM.
Update from Grundlefleck's comment: As for launching the actual tests there is selenium remote-control, which allows you to write your tests in Java, Ruby, plain HTML and other possible drivers.

Yes, it is possible (but it might very well require LOTS of code changes).
No, I do not know any of the details.
I would not recommend this approach for your purposes. From your comment, it sounds like you are trying to scrape webpages. If you really need to use JavaScript, you can use a stand-alone JavaScript-engine (Mozilla's is available here). Otherwise, I would use Beautiful Soup with Python or Twill. You might also want to read this question.

Related

Call JavaScript (3rd party library) from Python

I've already searched quite a bit but came to now clear conclusion as some projects (pyv8) seem to be dead and I'm not sure if that is suitable at all. The 3rd part lib requires a DOM, eg. a container element in which it runs. It also uses web assembly and in general is pretty heavy.
Not sure if libs like pyv8 would actually be suitable for that. Other approach would be to go with selenium and headless chrome or a local node.js service but both of these sound very heavy. Oh, and the lib must work in windows as that's simply company policy, windows servers so PyMiniRacer is out.
What are my other options?
Consider taking a look at this post: How do I call a Javascript function from Python?.
However, if your objective is to access JS code in a webpage for reasons such as webscraping, you could also consider using selenium webdriver + python to do so. Take a look at this medium.com post: How to Run JavaScript in Python | Web Scraping | Web Testing
Other Resources:
https://www.quora.com/How-do-we-use-JavaScript-with-Python
Python to JS: https://pypi.org/project/javascripthon/
P.S: I am not sure if this would help you. There is another library (PyExecJS) which is maintained no longer; but I think you have looked it up already.

JavaScript basics/instructional

I have just begun giving JavaScript a try on a programmer friend's suggestion (though I have used it to some extent before with websites using jQuery without fully understanding the intricacies of the language itself) and was wondering how certain features work.
Question:
What other uses of JavaScript are there aside from websites (by itself and/or with jQuery framework)? Is there a difference between making a site interactive and an actual "web app"? I've heard the term and don't know the difference.
To add to my main question:
I've read and done some tutorials on prototyping and am not sure on its actual application (since what little I've dabbled in, by making a website more interactive, I've never seen them used). Can someone link a website with extensive deep/heavy JavaScript use so I can check it out?
Gist:
In essence, what I'm trying to understand is where heavy use of JavaScript comes into play because so far, all JavaScript I've experienced is superficial.
Appreciate in advance any advice/help in this matter!
What other uses of Javascript are there aside from websites (by itself and/or with jQuery framework)?
Check out node.js. It runs JavaScript programs not intended for web use. Node offers a bunch of imports for things like file system IO, socket IO and a bunch of others. I personally use it for a WebSocket server because I don't like my alternatives.
Further, JavaScript has native support in many operating systems, including all Windows versions after and including XP (not sure about before). Windows Script Host runs .js files as JScript and these can be used to achieve things like those node can. It's a convenient way to do things batch can't.

JavaScript library development [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
I am a java-programmer and a few months ago I discover frontend programming with javascript and html, and it is absolutely cool. So here is a js newbie's question.
Is there any IDE, default workflow or maybe best practices for javascript libs development? I didn't mean coverage and unit testing or something like this. I mean is there any tools or techniques which should I use while developing html control with some js-logic, for example? Should I write small test page in text editor and open it in dozen of browsers pressing F5 each minute and watching at browser's console or maybe there is some magic IDE which will on button press reload all browser instances and collect reports from browsers?
JavaScript Fundamentals
Be sure that you know the language. JavaScript is a bit tricky in this regard: you can easily think that you understand the language, but there are weird things that can sneak up on you. I highly recommend JavaScript: The Good Parts by Douglas Crockford. He also has some talks on Youtube with the same name that are definitively worth watching.
JSLint
Integrate JSLint or a similar tool in your workflow. It is a style checker and static analysis tool for Javascript and helps catching subtle bugs.
Avoiding F5
Check out live.js:
Just include Live.js and it will monitor the current page including
local CSS and Javascript by sending consecutive HEAD requests to the
server. Changes to CSS will be applied dynamically and HTML or
Javascript changes will reload the page. Try it!
Browser Developer Tools
Get familiar with them. I personally strongly prefer Chrome's developer tools over Firefox/Firebug, but regardless which one you choose, learn how to use the debugger.
Node.js
You should also be aware that you don't have to test Javascript logic in your browser: you can use it as any other scripting language using node.js.
As a seasoned programmer here are the steps you should follow for JS lib production.
Separate the UI from the application logic. In this case create one component for the application logic that is completely separated from all API access. API access, which means DOM and WSH and Node.js, should be separated out. I go so far as to use different files to force and ensure separation.
Create UI environments to access your logic. Have a production UI control for access by your audience and also a separate internal UI control for your sandboxed development. For example I have written my application at http://prettydiff.com/ to work from commandline, and from a browser. I have also written access methods to the application access similar to the published HTML but different for my own development for more rapid unit testing. In a sandboxed UI can use a recursive setTimeout to refresh the page in intervals for automated test plan verification as you are writing and saving code.
Focus on availability and distribution. People can get my libraries directly from my website, Github, and NPM for Node. I have a regimented process where by I upload code to the site during a production release and then perform a quick verification in the browser. If the release did not break the application then push to Github and then push this exact same location to NPM.
Distributed access to code operation is even better. Its nice having rapid and even automated access to libraries online, but being able to access unit testing via request is even better. I can remotely unit test my application's code by accessing code samples online directly by telling my application request a code sample via URI. This means that not only is my static code available for distribution and testing, but so is its operation.
Good documentation is everything. I will never even bother to examine a lib if the documentation is weak. I am not talking about code comments, though these are important. I am talking about end user documentation. I want to be able to read documentation and know everything about the subject. Node.js became popular because even before the codebase was stable it's documentation kicked ass. Get other people to QA your documentation before they QA your code. Without stunning documentation your lib is less than worthless to me.
Know your mission. Each and every lib should have a clear, simple, and well stated purpose. If this is not established then the lib is not ready for release. If an enhancement confuses the lib's purpose then it may be time to divide one library into multiple smaller libraries. Focus on precision, clarity, directness, and only the sole stated mission of the lib.
Independence is vital to adoption. I don't like libs dependent upon other libraries or frameworks. Its great that your jQuery library may be the best thing since slice bread, but I won't be looking at it. Independence means greater portability and freedom it mix and match it with other libraries that you are not aware of.
Style is important. This is touchy subject, but style is important. Keep the logic in your lib as simple and declarative as possible. If your code is absolutely declarative in nature then your algorithmic patterns are awesome. Avoid use of the new keyword unless you are an experienced JavaScript badass and severely limit use of this keyword as it will fail you in future maintenance operations. Do not algorithmically build out large strings with concatenation and continually watch the execution speeds of your code. Because even trivial changes to style or seemingly minor enhancements to the logic can destroy execution efficiency I put a timer on all my UI controls. In your development use profilers, such as Chrome's web tools to track down long operations in JS execution.
Be open about your failures. Software is never perfect and other developers will always respect this. If you encounter a bug before anybody else then be open about the existence of the bug. If it will take more than a week to get the solution out then don't delay notification. Notify your users immediately so that they are aware. I recently rolled back a major enhancement to the logic of my diff algorithm because additional unit testing showed heavy slippage. I rolled back the same day I made a decision the prior release or two was flawed and was open about the rollback. If you want people to contribute to your code base or provide bug reports then openness is critical.
I know you may have explicitly said that you didn't mean unit testing, but that is precisely the way I recommend you write javascript libraries.
If you're a Java developer, you may be familiar with jUnit. If so, qUnit may be more natural to you. Otherwise, I recommend you take a look at Jasmine or Mocha. While I prefer Mocha, my experience with Jasmine has generally been better for in-browser development thanks to the awesome Jasmine-Jquery plugin.
If you're writing qUnit tests, take a look at IntelliJ IDE which allows you to execute tests in addition to providing code coverage.
If you're developing on the browser, take a look at LiveReload. It'll watch files for you and auto-refresh your browser - great for instant feedback.
For browser compatibility, I would recommend you just get it to work on one first reasonably before worrying about the others. Check in from time to time to see if you find issues. See if jQuery can abstract some of that mess for you. Otherwise, take a look at Adobes BrowserLab.

JavaScript object browser?

I was recently tasked to document a large JavaScript application I have been maintaining for some time. So I do have a good knowledge of the system.
But due the sheer size of the application, it will probably take a lot of time even with prior knowledge around the code and the source code itself in uncompressed form.
So I'm looking for tools that would help me explore classes and methods and their relationships in JavaScript and if possible, document them along the way, is there one available?
Something like object browser in VS would be nice, but any tools that help me get things done faster will do.
Thanks!
Firebug's DOM tab lets you browse the contents of the global window object, and you can inspect a particular object by entering inspect(whatever) in the command line.
You won't be able to use it to detect relationships unless an instance of one object holds an instance of a related object, but it's a start.
You can also use the Options menu on the DOM tab to restrict what's shown to user-defined functions and properties, which should help reduce clutter.
Take a look at Aptana, they have an outline that can help you to determine what are the objects and somtetimes their relationship.
Firebug + uneval(obj) is a simple trick that is often helpful.
I see a lot of people talking about examining the DOM within Firebug. However, from your question it looks like you want something like jsdoc? just add type and class information through comments and jsdoc generates documentation including class relationships. http://jsdoc.sourceforge.net/
Google has a fork of it with added functionality http://code.google.com/p/jsdoc-toolkit/
UPDATE: It's not a fork, it's a rewrite by the developer that wrote jsdoc originally as a perl script. It aims at being more adaptable so you can use whatever js inheritance/events/properties style you'd like. Another feature is that it lets you modify the templates used to generate the HTML in a much simpler way.
We don't know if this JS application is designed to run in a Web browser...
If yes, as advised, Firebug (a Firefox extension) is excellent at debugging JS and exploring Dom.
On the IE side, you have some tools like IEDocMon, Web Accessibility Toolbar (it does more than its name) or Fiddler (unrelated to your question, but still a good tool to have).
Firebug (Firefox) / Dragonfly (Opera) can help you with viewing objects in realtime
Aptana / JS/UML(Eclipse) can help with relationships of objects
This is an old question, but let me answer it anyway.
Use an IDE. Integrated Development Environments were made for jumping around rapidly among the code. The key features you will exercise during exploration are viewing the file structure or outline, jumping to a declaration or usage, and searching the entire project for all instances of a string. If you are using WebStorm, set up a custom scope for files except generated files and node.js to aid in searching.
Run 'npm la | less' which lists all your dependent modules with one line descriptions. You may have never seen moment.js and never need to read the documentation, but taking the time to read a one line summary of it is worthwhile. If you need more information on a tool than one line summary, search for the term on SlideShare. Slides are faster than ReadTheDocs.
Document a little as you go. I'm a fan of forcing people to use notebooks constantly rather than scratch paper. Also, I find adding a one line comment to each JavaScript file is worthwhile. You want to know what should be in each directory of your project. I also recommend building a glossary of the exact meaning of domain terms in your system, e.g., what does "job" in your system.
Finally, you may need to just fire up the application in a debugger and start stepping through parts of it. Most large projects have accreted worth from programmers of various skill levels and motivations.
You are aiming for a level of "conceptual integrity" (to quote Yourdon) or to "grok" the software (to quote Heinlien). It does take some time, cannot be bypassed, and can be done efficiently.

Building Standalone Applications in JavaScript

With the increased power of JavaScript frameworks like YUI, JQuery, and Prototype, and debugging tools like Firebug, doing an application entirely in browser-side JavaScript looks like a great way to make simple applications like puzzle games and specialized calculators.
Is there any downside to this other than exposing your source code? How should you handle data storage for this kind of program?
Edit: yes, Gears and cookies can be used for local storage, but you can't easily get access to files and other objects the user already has around. You also can't save data to a file for a user without having them invoke some browser feature like printing to PDF or saving page as a file.
I've written several application in JS including a spreadsheet.
Upside:
great language
short code-run-review cycle
DOM manipulation is great for UI design
clients on every computer (and phone)
Downside:
differences between browsers (especially IE)
code base scalability (with no intrinsic support for namespaces and classes)
no good debuggers (especially, again, for IE)
performance (even though great progress has been made with FireFox and Safari)
You need to write some server code as well.
Bottom line: Go for it. I did.
Another option for developing simple desktop like applications or games in JavaScript is Adobe AIR. You can build your app code in either HTML + JavaScript or using Flash/Flex or a combination of both. It has the advantage of being cross-platform (actually cross-platform, Linux, OS X, and Windows. Not just Windows and OS X).
Heck, it may be the only time in your career as a developer that you can write a web page and ONLY target ONE browser.
SproutCore is a wholly JavaScript-hosted application framework, borrowing concepts particularly from Cocoa (such as KVO) and Ruby on Rails (such as using a CLI generator for your models, views and controllers). It includes Prototype, but builds plenty of stuff such as sophisticated controls on top of that. Its Photos demo is arguably impressive (especially in Safari 3.1).
Greg already pointed you to Gears; in addition, HTML 5 will come with a standardized means of local storage. Safari 3.1 ships with an implementation where you have a per-site SQLite database with user-settable size maximums, as well as a built-in database browser with SQL querying. Unfortunately, it will be a long time until we can expect broad browser support. Until then, Gears is indeed an alternative (but not for Safari… yet!). For simpler storage, there is of course always cookies.
The downside to this would be that you are at the mercy of them having js enabled. I'm not sure that this is a big deal now. Virtually every browser supports js and has it enabled by default.
Of course the other downside would be performance. You are again at the mercy of the client handling all the intensive work. This also may not be that big of a deal, and would be dependent on the type of app you are building.
I've never used Gears, but it looks like it is worth a shot. The backup plan would be to run some server side script through ajax that dumps your data somewhere.
Not completely client side, but oh well.
Nihilogic (not my site) does a lot of stuff with Javascript. They even have several games that they've made in Javascript.
I've also seen a neat roguelike game made in Javascript. Unfortunately, I can't remember what it was called...
If you want to write a standalone JavaScript application, look at XULrunner. It's what Firefox is built on, but it is also built so that you can distribute it as an application runtime. You will write some of the interface in JavaScript and use JavaScript for your code.
Gears might provide the client-side persistent data storage you need. There isn't a terribly good way of not exposing your source code, though. You could obfuscate it but that only helps somewhat.
I've done simple apps like this for stuff like a Sudoku solver.
You might run into performance issues given that you're completely at the mercy of the client's Javascript interpreter. Gears would be a nice way of data storage, but I don't think it has penetrated the market that much. You could just use cookies if you're not fussy about that kind of thing.
I'm with ScottKoon here, Adobe AIR is great. I've really only made one really nice (imho) widget thus far, but I did so using jQuery and Prototype.js, which floored in such wonderful ways because I didn't have to learn a whole new event model. Adobe AIR is really sweet, the memory foot print isn't too bad, upgrading to a new version is built into AIR so it's almost automatic, and best of all it's cross-platform...they even have an alpha-version for Linux, but it works pretty well already on my Eee.
Standalone games in GWT:
http://gpokr.com/
http://kdice.com/
In regard to saving files from a javascript application:
I am really excited about the possibilities of client-side applications. Flash 10 introduced the ability to create files for save right in the browser. I thought it was super cool, so I built a javascript+flash component to wrap the saving feature. Right now it only works for creating text based files (vcard, ical, xml, html, css, etc.)
Downloadify Home Page
Source Code & Documentation on Github
See It In Use at Starter for jQuery
I am looking to add support for non-text files soon, but this is a start.
My RSS feeds have served me well- I found that Javascript roguelike!
It's called The Tombs of Asciiroth.
Given that you're going to be writing some server code anyway, it makes sense to keep storage on the server for a lot of domains (address books, poker scores, gui configuration, etc.,.) For anything the size of what you'll get in Webkit or Gears, you can probably also keep it on your server.
The advantage of keeping it on your server is two-fold:
You can integrate it fairly simply as a Model layer in a typical MVC framework, and,
Users get a consistent view without being tied to their browser/PC, or in a less-than-ideal environment (Internet Cafés).
The server code for handling this can also be fairly trivial, particularly if it's written with this task in mind, so it's not a huge cognitive burden.
Go with qooxdoo. They recently realsed 1.0, although most users of it say it was ripe for 1.0 at least two versions ago.
I compared qooxdoo with YUI and ext, and I think qooxdoo is the way to go for programmers - YUI isn't that polished as qooxdoo, from a programmer's point of view and ext has a not so friendly licensing model.
A few of the strong points (for me) of qooxdoo are:
extremely clean code
the nicest OO programming model I've seen among Javascript frameworks
an extremely rich UI widget library
It also features a test runner for unit tests, an API doc generator and reader, a logging facility, and several useful features for debugging, grouped under something called Inspector.
The only downside is that there aren't readymade themes (something like skins) for qooxdoo. But creating your own theme is quite easy.

Categories

Resources