Server-Side DOM Manipulation Technology the right option get html responses - javascript

I'm a security guy by day a coder by night and I usually write my tools in java. It was something I was taught in school so really its just the simple fact that I'm comfortable writing in that language.
I have been for a few years now been limited by the inability to take a response from a server and build it up by actually running the JavaScript. This has drastically hurt my ability to do so many things like spider and simply see an actual response as if it was built in a browser. In order to compensate for this Weakness I have tried to use a number of different headless browsers. I tried JX browser, CORBA,SWT, HTML unit, etc. You name it I have tried to use it. As many of you know none of these technologies can even accurately handle some of the most simple JavaScript applications. Therefore, I'm really looking for an alternative.
I've been looking into potentially using something like server-side JavaScript in order to render the HTML and manipulate the DOM. I was looking at things like Node.js,PhantomJS, etc. ideally I would like to be able to keep using my Java code and somehow use these technologies as a supplement for those headless browsers. However, it looks like being able to talk to these technologies will be difficult. It looks like I might be able to leverage the commandline with Java and write to files with those js technologies and read those files with my Java code. This does seem a little clunky so I thought maybe I should just flip over to fully using these to build my tools. I wanted to get others' ideas or opinions on this. Just to give you an idea of what I typically need to do I need to be able to I need to build the send HTTP requests and get responses back,crawl sites, things like that.
I'm really just looking to see if there are other options available or if I'm going about this the right way

Related

Generating modern js-based rich client from server side?

I am working on a intra-net project, which uses javascript for client side effects/widgets, like tree, grid, dragging. We are thinking about to use jquery/backbone plus some jquery ui.
The problem is, with great flexibility of jquery/backbone, our low experienced developers are almost certained to make code MESSY. And I hate that.
To reduce the chance they can make code messy, I am going to build a server-side program to parse xml (those developer will write xml instead of javascript when possible) and generate javascript at runtime. By doing this, I can force restriction on the xml. And xml tends to be less messy.
But before jumping into that, I want to ask is there any similar existing framework? Preferably the javascript generated also make use of model/view/controller pattern.
Or what else can I do to prevent the project from being messy?
ps: I want to educate the developer. But that is a long term job. Not really helpful for current project.
Thank you guys/girls.
Alright, a terrible idea I think. So besides training my developers (for long term), what else can I do? Especially in short term because the project is rolling on?
I like your goal but I think that trying to enforce coding standards in a programmatic way is over-solutionising the problem. The following steps help to steer you in the right direction though:
Use a version control system such as SVN or Git. Not only does this allow you to pinpoint who is messing up your code and educate them, but it gives you the option to roll back changes that completely break your project. It shouldn't be seen as a "blaming" mechanism, but is certainly a reliable way to identify the culprit. You may also find that your developers are less likely to commit code that isn't up to scratch after they realise their work is open to such scrutiny.
Documentation - and I don't necessarily mean having a 300 page wiki on how your project is organised and how functions should be written. But even something as simple as commenting every function will often encourage others to do the same. The important thing is that the developer sees the standard of work they are expected to deliver in your work. Developers with potential will take their cues from you.
Paired programming - get into the habit of sitting with your developers while they code. Not 100% of the time, but often enough that you are aware of their strengths and weaknesses. Teach them the way you want them to code.
If all else fails, resort to tools that help developers with their standards, such as the Resharper plugin for Visual Studio. While this is primarily designed for C# usage, it also provides code hints etc. for Javascript that will help inexperienced developers adopt a set of practices that should improve the readability of their code.
I don't think there is any substitute for a good tutor, and providing any sort of automated solution is likely to cause a lot of headaches for you without ever really improving the skills of those you are hoping to teach.

Same code for client and server

Is it feasible to have parts of the code shared between webapp's client and server? Assuming I use a javascript-based server, I hope I could have at least form verification code and parts of logic shared.
I worry about incompatibilities between javascript engines, though.
If your code runs on Chrome (V8), it will also run under Node.js (V8) which today, is the #1 choice for server side JavaScript.
Of course you need to make sure that you don't use any Chrome specific stuff, but the same hold true for things that only work Firefox (like noSuchMethod) etc.
But I can't think of any particular things when it comes to form validation that would break cross engine compatibility.
http://jaxer.org/ has some really nice looking examples for sharing code between client and server, as well as manipulating the DOM in both locations. However, this project looks like it may have been abandoned by its creator. Just posting the link because the design concepts might be useful to you when trying to share validation code in both locations.
Engine incompatibility should not extend to data-validation, it should be limited to DOM interaction.
If your javascript engines are so weirdly different that they treat strings and integers and whatnot differently -- stop using them.

Where can I get advice on how to build completely ajax web apps?

I am building a completely ajax web app (this is the first web app I have ever created). I am not exactly sure if I am going about it the right way. Any suggestions or places where I can go to find suggestions?
Update:
I currently am using jQuery. I am working on fully learning that. I have designed a UI almost completely. I am struggling in some parts trying to balance a good UX, good design and fitting all the options I want to fit in it.
I have started with the design. I am currently struggling with whether to use absolute positioning or not and if not how do I use float etc. to do the same type of thing. I am trying to make it have a liquid layout (I hate fixed-layout pages) and am trying to figure out what I should use to make it look the same in most screen sizes.
Understand JavaScript. Know what a closure is, how JavaScript's event handling works, how JavaScript interacts with the DOM (beyond simply using jQuery), prototypal inheritance, and other things. It will help you when your code doesn't work and you need to fix it.
Maintain usability. All the AJAX magic you add is useless if users cannot figure out how to use it. Keep things simple, don't overload the user by giving him information he doesn't need to know (hide less important information, allowing the user to click a link to show it), and if possible, test your app with actual users to make sure that the interface is intuitive to them.
Code securely. Do not allow your server to get hacked. There are many different types of security flaws in web apps, including cross-site scripting (XSS), cross-site request forgery (CSRF), and SQL injection. You need to be well aware of these and other pitfalls and how to avoid them.
One starting point is to look at the Javascript Libraries and decide which one to use:
http://code.google.com/apis/libraries/
http://en.wikipedia.org/wiki/Comparison_of_JavaScript_frameworks
You probably don't want to do raw Javascript code without any library. Once you decide on a library to use, then you can look at its documentations online or the books about using them. jQuery does have pretty good documentation.
Define "right way."
There are many "right ways" to code an app.
Things to keep in mind are trying to design a nice interface. The interface can make or break an application and studies show that it can even make it seem faster if you do it right. jQuery is good for this.
Another thing to consider going in is what browsers do you want to support? Firefox is really doing well and Google Chrome's market share is growing so you will want so support those for sure. IE is a tough one as it doesn't have the best support for standards, but if you are selling a product you will really want this.
One of the best articles that I've ever come across about the structure of an ajax web application is this one. A little outdated because it refers to XML as the primary data-interchange format, now JSON. jQuery, a javascript framework, contains excellent functionality for both DOM manipulation and AJAX calls. Both are a must in any AJAX-driven web app.

Questions about capability of Javascript

Many years back, I was told that Javascript was harmful, and I remember being annoyed with endless popup when I right-clicked an image to download it.
Now it seems suddenly that Javascript is great, and you can do a lot of things with it to let users have native-like web application experience.
I admit I have missed 6-7 years of Javascript literature, so I hope to start anew with SO kickstarting me to understand the following:
Is Javascript mainly concerned about user interface i.e. smoothen interaction between application and users and not about logic processing, number crunching or form processing etc.?
Can Javascript write to local hard drive (besides cookies)?
Can Javascript web application run with Javascript capabilities in browsers turned off? (I would think outright no, but an article on Adaptive Path said 'maybe')
Is AJAX illegal to use due to Eolas patent claim? Is it worth it spending effort learning it when the future is not secure? (I know AJAX is not Javascript)
Thanks. Hoping for enlightenment.
Yes. JavaSscript is usually used to enhance the user's experience and make the site easier to use. It is also possible to delegate validation tasks and the like to JavaScript, however (though this should never absolve the server of its responsibility to check input).
No.
That depends on how the application is written. If it's done properly, then the JavaScript will merely enhance the interface, and the application will still work without it; this is called progressive enhancement.
Not at all. AJAX is used extensively on this very site!
One reason for the resurgence of popularity for JavaScript lately is the emergence of several frameworks. These make the process of writing JavaScript much, much easier, allowing tasks that would previously have been horribly complex to be implemented with minimal time and effort. The most popular of these is jQuery, which is a good place to start if you're intending to get in on the action.
Overall, JavaScript is a very powerful tool that allows you to create very rich interfaces. Well worth learning.
Yes, Javascript is all about client side processing, but also about AJAX where it calls back to the server asynchronously so that users do not see pages reloading.
No
No, but there are ways to gracefully degrade the experience for non javascript users. It requires carefult planning however.
No, that lawsuit was just about the browser technology that enables it. As a developer you dont have to worry about that.
Can Javascript write to local hard drive (besides cookies)?
Not really. However, as HTML5 support becomes more widespread you'll be able to use things like Web Storage and Web SQL. You won't be able to write arbitrary files on the user's hard drive, but using those two technologies you'll be able to persistently store and access data.
Can Javascript web application run
with Javascript capabilities in
browsers turned off? (I would think
outright no, but an article on
Adaptive Path said 'maybe')
It really depends on how you define "web application." You can write web apps without using Javascript for anything but UI candy, in which case you can degrade gracefully without without it. However, it's also possible to write web apps that rely heavily (entirely, even) on Javascript, which will utterly fail without it.
Is AJAX illegal to use due to Eolas
patent claim? Is it worth it spending
effort learning it when the future is
not secure?
I'm not a lawyer, but I'd agree with the other answers -- you shouldn't worry about it. I'm certainly still writing AJAX stuff :)
Is Javascript mainly concerned about
user interface i.e. smoothen
interaction between application and
users and not about logic processing,
number crunching or form processing
etc.?
It's about both. And more than that.
Javascript has really come into its own in the past few years. Browsers have gotten a lot faster at executing it quickly, and people have been figuring out new ways to use the language itself to its full potential. You can really start using Javascript like a full-out application programming language, and not just to write little scripts that animate something or validate input.
If you're just getting back into the language and haven't read Crockford yet, I would highly recommend it. It's a great starting point to realizing the full potential of Javascript.
Edit: Some good Crockford Links
Javascript: The World's Most Misunderstood Programming Language
Javascript: The Good Parts (This is a presentation. Crockford also wrote a book by the same name that I haven't read myself, but I hear it's quite excellent.)
It's mainly for UI, but it can be used to save server-time on some operations (for example, Mathoverflow uses it to render LaTeX) and it's becoming popular to do so. But when you do this, you need to be respectful of the end-users time, because JavaScript can hold up some browsers, while it runs. But in general, it's a good and interesting idea.
Not without permission
If it's written correctly, it can. It's called "Graceful degredation" (some other variant terms exist, but the idea is the same). The basic idea is that you have it such that the JavaScript fails 'gracefully', and links that would normally get handled via JavaScript (i.e. to do some inline next-paging) will navigate to a 'backup' page that shows the relevant content.
I don't know about that, but AJAX can be implemented in different ways, XMLHTTPRequest is just one of them :) (And the most common, and suitable). Generally you like a library do this for you anyway (jQuery, or otherwise) but you can do it yourself for fun.
Yes, in my experience JavaScript is generally used to create a streamlined interface and relays information from the client to a server application for processing.
Yes, if the browser is configured to allow this (most aren't by default since this can be very dangerous).
No, JavaScript will not run if the browser is configured to have JavaScript disabled.
I wouldn't forgo learning JavaScript for this reason - as for the legality of the whole thing I wouldn't feel comfortable advising you about this. Still I think JavaScript is worth learning in spite of this situation.
The Eolas patent covers the embedding of objects in a HTML document (see US patent 5,838,906 titled "Distributed hypermedia method for automatically invoking external application providing interaction and display of embedded objects within a hypermedia document") ... this scope would not seem to include AJAX as a suite of technologies (being essentially scripting in a document to load content elements).
Partial answers:
I think all the security vulnerabilities associated with javascript have been fixed? IIRC the problems weren't with javascript, they were with particular browser's implementation of javascript.
I wouldn't worry about any patent claims on the AJAX technology. Patent sueing and counter-sueing is common place in the software world and invariably ends up with the affected parties licencing each other's technology. AJAX is not going anywhere :)

Will server-side JavaScript take off? Which implementation is most stable? [closed]

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
Does anyone see server-side JavaScript taking off? There are a couple of implementations out there, but it all seems to be a bit of a stretch (as in, "doing it BECAUSE WE CAN" type of attitude).
I'm curious to know if anyone actually writes JavaScript for the server-side and what their experiences with it have been to date.
Also, which implementation is generally seen as the most stable?
I like to read Googler Steve Yegge's blog, and recently I came across this article of his where he argues that Mozilla Rhino is a good solution for server-side JS. It's a somewhat sloppy transcript, you might prefer to watch the video of the talk. It also offers a little bit of insight on why he thinks server-side JS is a good idea in the first place (or rather, why he thinks that it's a good idea to use a dynamic language to script Java). I thought the points he makes were convincing, so you might want to check it out.
A while earlier, he also posted something about dynamic languages in general (he's a big fan of them), just in case you were wondering why to use JS at all.
Why would you want to process
something in Javascript when you can
process it in PHP or ASP.NET which are
designed specifically for this task?
Perhaps because JavaScript is a more powerful programming language than those two? For example, it has functions as first-class data types and support for closures.
Steve Yegge has blogged about porting Ruby on Rails to server-side JavaScript as an internal project within Google ("Rhino on Rails"). He did it because he likes Rails but using Ruby isn't allowed within Google.
Before it was acquired by Google, JotSpot used server-side JavaScript to let you query their database and display your pages. They used Rhino to do it. CouchDB uses server-side JavaScript to create views of their database.
As you can see from these examples, a great way to use JavaScript on the server is for plugins. One of the reasons it's used is that you can create a very isolated sandbox for people to run their code in. Also, because of the way that JavaScript as a language works, you can provide a user tooling specifically honed to the tasks your users need to complete. If you do this right, users don't need to learn a new language to complete their tasks, a quick glance at your API and examples is enough to get them on their way. Compare this to many of the other languages and you can see why using server-side JavaScript to provide a plugin architecture is so enticing.
A secondary popular solution, one which can be seen through a project like Jaxer, is that a common problem of web applications that do client-side validation is that, since JavaScript is easily bypassed in the browser, validation has to be run once again on the server. A system like Jaxer allows you to write some validation functionality that is reusable between both server and client.
Support for JS on the server has been getting stronger and the number of frameworks is getting bigger even faster.
Just recently the serversideJS group was founded. They have a lot of smart people that have been working on serverside JS for years (some of them more then 10).
The goal for this project is to create
a standard library that will
ultimately allow web developers to
choose among any number of web
frameworks and tools and run that code
on the platform that makes the most
sense for their application.
to the people who say "why would you choose JS over java or any other language?" - you should read this Re-Introduction by Crockford and forget about the DOM - the DOM is superugly, but that's not JS fault and JS is not the DOM.
I've never even heard of this, but it strikes me as using the wrong tool for the job. Since programming languages are just tools designed to help us solve some problem.
Why would you want to process something in Javascript when you can process it in PHP or ASP.NET which are designed specifically for this task?
Sure you can pound a nail in with a screw driver, but a hammer works much better because it was actually designed for it...
So no, I don't see it taking off.
Well, plain ol' ASP supported JavaScript server-side years ago and everyone onad their dog used VBShiate instead. But I have to agree with the others: JS does not seem to be the right tool here - and I love to do client-side JS :)
I personally did a whole site in server side JavaScript using ASP. I found it quite enjoyable because I was able to have some good code reuse. This included:
validation of parameters
object modeling
object transport
Coupled with a higher-level modeling tool and code gen, I had fun with that project.
I have no numbers on perf unfortunately, since it is used only on an intranet. However, I have to assume performance is on par with VBScript backed ASP sites.
It seems like most of you are put off by this idea because of how unpleasant the various client-side implementations of Javascript have been. I would check out existing solutions before passing judgment, though, because remember that no particular SS/JS solution is tied to the JS implementations currently being used in browsers. Javascript is based on ECMAScript, remember, a spec that is currently in a fairly mature state. I suspect that a SS/JS solution that supports more recent ECMA specs would be no more cumbersome than using other scripting languages for the task. Remember, Ruby wasn't written to be a "web language" originally, either.
Does anyone see Server-side Javascript
taking off?
Try looking at http://www.appjet.com a startup doing hosted JavaScript applications to get a feel for what you can do. I especially like the learning process which gently nudges the user to build things with a minimal overhead ~ http://appjet.com/learn-to-program/lessons/intro
Now it might seem a weird idea at the moment to use JavaScript but think back when PC's started coming out. Every nerd I knew of was typing away at their new Trash-80's, Commodore64's, Apple ]['s typing in games or simple apps in BASIC.
Where is todays basic for the younger hacker?
It is just possible that JavaScript could do for Web based server side apps as BASIC did for the PC.
XChat can run Javascript plugins.
I've some accounting software completely written in Javascript.
There's this interesting IO library for V8: http://tinyclouds.org/node/
CouchDB is a document database with 'queries' written in Javascript (TraceMonkey).
Considering this, i believe, server-side Javascript did take off.
Server-side programming has been around for a lot longer than client side, and has lots of good solutions already.
JavaScript has survived and become popular purely because developers have very little choice in the matter - it's the only language that can interact with a DOM. Its only competition on the client side is from things like Flash and Silverlight which have a very different model.
This is also why JavaScript has received so much effort to smart it up and add modern features. If it were possible for the whole browser market to drop JavaScript and replace it with something designed properly for the task, I'm sure they would. As it stands Javascript has strange prototype-based objects, a few neat functional programming features, limited and quirky collections and very few libraries.
For small scripts it's fine, but it's a horrible language for writing large complicated systems. That things like Firefox and Gmail are (partly) written in it is a heroic accomplishment on their part, not a sign that the language is ready for real application development.
Flash Media Server is scripted by using Server Side Action Script, which is really just javascript (ECMAScript). So, I do it a lot. In fact, most of my day was dealing with SSAS.
And I hate it. Though to be fair, a bunch of that is more related to the (not so great) codebase I inherited than the actual language.
I think server-side Javascript is guarenteed to take off. Its only a matter of time.
Mozilla, Google, and Adobe have so much vested interest for Javascript that it would take a miracle to dislodge it from the browser world. The next logical step is to move this into the server-side.
This is a step towards moving away from the hodge podge of Internet technology that usually includes all of these
HTML
CSS
Javascript
Serverside Language J2EE/ASP/Ruby/Python/PHP
SQL
I haven't heard much about the current state of Javascript Server frameworks, except that they are mostly incomplete.
I see server-side js will offer considerable advantages in future applications. Why? Web apps that can go offline, client-side db store, google gears, etc...
Following this trend, more and more logic are moving into the client-side. Use an ORM that works for client-side, and use another on server-side (be it PHP / Ruby / whatever), write your synchronization logic twice in two different languages, write your business logic twice in two different languages?
How about use js on the client AND the server side and write the code once?
Convincing?
Personaly i've been developing and using my own JavaScript framework for about 4 years
now.
The good thing about JS on serverside is that implemented in ASP Classic you don't need
any other plugin or software installed, besides i'm also using my javascript (client)
framework on my server, that allows me to enjoy of the same functionality and proven
performance of my functions at both environments client and serverside.
Not only for data validation, but also lets say HTML or CSS dynamic constructions
can be done client or serverside, at least with my framework.
So far it works fast, i have nothing to complain or regret except its great usability
and scalability that i have been enjoying during this past 4 years, until the point
that i'm changing my ASP Classic code to javascript code.
You can see it in pratice at http://www.laferia.com.do
Node.js has taken off and proven that server-side JavaScript is here to stay =)
I can't see most developers getting over their distaste for client-side JavaScript programming. I'd rather go to Java for server-side stuff before choosing JavaScript.

Categories

Resources