Related
We recently upgraded to a newer build of a JavaScript minification library.
After a significant amount of quality assurance work by the testing team, it was discovered that the new version of our minifier had an issue that changed the intention and meaning behind a block of code.
(Life lesson: don't upgrade JS minifiers unless you are really convinced you need the new version.)
The minifier is used for client side JavaScript code with a heavy emphasis on DOM related activity, not nearly as much "business logic".
A simplified example of what was broken by the minifier upgrade:
function process(count)
{
var value = "";
value += count; //1. Two consecutive += statements
value += count;
count++; //2. Some other statement
return value; //3. Return
}
Was minified incorrectly to the following:
function process(n){var t="";return t+n+n,n++,t}
While we could write some unit tests to catch some of the issues potentially, given that the JavaScript is heavy on DOM interactions (data input, etc.), it's very difficult to test thoroughly without user testing (non-automated). We'd pondered using a JS to AST library like Esprima, but given the nature of the changes that could be done to the minified code, it would produce far too many false positives.
We also considered trying to write representative tests, but that seems like a never-ending task (and likely to miss cases).
FYI: This is a very sophisticated web application with several hundred thousand lines of JavaScript code.
We're looking for a methodology for testing the minification process short of "just test everything again, thoroughly, and repeat." We'd like to apply a bit more rigor/science to the process.
Ideally, we could try multiple minifiers without fear of each breaking our code in new subtle ways if we had a better scientific method for testing.
Update:
One idea we had was to:
take minification with old version
beautify it
minify with new version,
beautify, and
visually diff.
It did seem like a good idea, however the differences were so common that the diff tool flagged nearly every line as being different.
Have you considered a unit test framework, such as QUnitjs ? It would be quite a bit of work to write the unit tests, but in the end you would have a repeatable test procedure.
Sounds to me like you need to start using automated Unit Tests within your CI (continuous integration environment). QUnit has been thrown around, but really QUnit is a pretty weak testing system, and its assertions are barebones at the minimum (it doesn't even really use a good assertion-based syntax). It only marginally qualifies as TDD and doesn't handle BDD very well either.
Personally I'd recommend Jasmine with JsTestDriver (it can use other UT frameworks, or its own, and is incredibly fast...though it has some stability issues that I really wish they'd fix), and setup unit tests that can check minification processes by multiple comparisons.
Some comparisons would likely need to be:
original code & its functionality behaves as expected
compared to minified code (this is where BDD comes in, expect the same functional performance/results in minified code)
I'd even go a step further (depending on your minification approach), and have a test that then beautifies the minification and does another comparison (this makes your testing more robust and more ensured of validity).
These kinds of tests are why you would probably benefit from a BDD-capable framework like Jasmine, as opposed to just pure TDD (ala the results you found of a visual diff being a mess to do), as you are testing behavior and comparisons and prior/post states of functionality/behavior, not just if a is true and still true after being parsed.
Setting up these Unit Tests could take a while, but its an iterative approach with that large of a codebase...test your initial critical choke points or fragile points fast and early, then extend tests to everything (the way I've always set my teams up is that anything from this point on is not considered complete and RC unless it has Unit Tests...anything old that has no Unit Tests and has to be updated/touched/maintained must have Unit Tests written when they are touched, so that you are constantly improving and shrinking the amount of untested code in a more manageable and logic way, while increasing your code coverage).
Once you have Unit Tests up and running in a CI, you can then tie them into your build process: fail builds that have no unit tests, or when the unit tests fail send out alerts, proactively monitor on each checkin, etc. etc. Auto-generate documentation with JSDoc3, etc. etc.
The issue you are describing is what CI and Unit Tests were built for, and more specifically in your case that approach minimizes the impact of the size of the codebase...the size doesn't make it more complex, just makes the duration to get testing working across the board longer.
Then, combine that with JSDoc3 and you are styling better than 90% of most front end shops. Its incredibly robust and useful to engineers at that point, and it becomes self-perpetuating.
I really could go on and on about this topic, there's a lot of nuance to how you approach it and get a team to rally behind it and make it self-forming and self-perpetuating, and the most important one being writing testable code...but from a concept level...write unit tests and automate them. Always.
For too long frontend devs have been half-assing development, not applying actual engineering rigor and discipline. As frontend has grown more and more powerful and hot, that has to change, and is changing. The concept of well tested, well covered, automated testing and continuous integration for frontend/RIA applications is one of the huge needs of that change.
You could look at something like Selenium Web Driver Which allows you to automate tests for web applications in various environments. There are some cloud hosted VM solutions for doing multi environment testing, so you don't get caught out when it works in Webkit but not in IE.
You should definitely look into using source maps to help with debugging minimized JavaScript. source maps will also work with supersets of JavaScript such as CoffeeScript or my new favorite TypeScript.
I use closure compiler which not only minifies, but also will create the source maps. not to mention it is the most agressive and produces the smallest files. Lastly you kinda have to know what's going on in minification and write compatible code, your example code could use some refactoring.
check out this article on source maps:
http://www.html5rocks.com/en/tutorials/developertools/sourcemaps/
also check out the documentation for closure compiler, it's got suggestions on how to write better code for minification:
https://developers.google.com/closure/compiler/
Not a testing solution, but how about switching to TypeScript to write large JS apps like yours?
I tested it out with TypeScript and its default min engine and it works fine.
Assuming your count arg is an number.
The type script will be:
class ProcessorX {
ProcessX(count: number): string {
var value = '';
value += count.toString();
value += count.toString();
count++;
return value;
}
}
Which produces js like this:
var ProcessorX = (function () {
function ProcessorX() { }
ProcessorX.prototype.ProcessX = function (count) {
var value = '';
value += count.toString();
value += count.toString();
count++;
return value;
};
return ProcessorX;
})();
Then minified to:
var ProcessorX=function(){function n(){}return n.prototype.ProcessX=function(n){var t="";return t+=n.toString(),t+=n.toString(),n++,t},n}()
It's on jsfiddle.
If your count is a string, then this fiddle.
we use closure compiler in advanced mode which both minifies and changes code, as such we compile our unit tests as well, so you could consider minifying your tests alongside your code and running it like that.
I am writing a Media-HUD that runs totally on local notecard files stored within the prim's inventory, with notecards named like index.html, style.css, icon.svg etc.
My hope is to use the LSL HttpServer functions, and the script's URL to create a totally self-contained media based HUD that is easy to edit like editing any web page.
This is completely possible on its own, however there is a limitation in that, the pages must fit into the memory allocated to the LSL script. Under mono this is only 64kb.
I want to remove this limitation, by somehow, perhaps from javascript, reading in each 'file' from a notecard line by line in the users browser itself (thusly, getting around the memory limit by only bringing one notecard line into memory at a time).
Is there a way to do this? generate a entire file in javascript procedurally by loading in the strings making it up line by line, and then serve it as though it were a whole file? I'm not sure how feasible this is.
Any idea's/guidance greatly appreciated!
You could do this through Javascript using XMLHttpRequest. jQuery's wrapper for this is called Ajax. You could request each line individually, which would be slightly slower, or read in a number of lines at a time, at the script's leisure. http_request is not throttled so either works. Note that the loader has to be sent in a single response, because the LSL server has no way of pushing data "piecemeal" like an actual server does.
Notes:
llGetNotecardLine only returns the first 255 bytes per line.
llHTTPResponse must be called within ~20 seconds of the request, so you can't feasibly read more than 20 lines from a notecard at a time.
I'm not sure how this would work for non-DOM filetypes. All files would need to be embed-able in the HTML using the Javascript DOM. To my knowledge, Javascript can't arbitrarily create an external file and serve it to itself. Obviously it would not work for non-text filetypes, but you could certainly load in the rest of the HTML, CSS, and HTML5 SVG. Basically, if it can go in a single HTML file, you could load it in through Javascript.
I have no experience with React but it gives a good example of what is possible on the UI side with loading things in purely through Javascript.
So less than 64 thousand characters in memory at most per script. I will give you some advise that might make your task feasable:
External resources
Minimize the amount of code you have to have in your notecards by sourcing popular libraries from the web Bootstrap, React and such.
You will have to rely on their mirror's availability thought. But it will greatly reduce the amount of memory needed to provide pretty and functional pages.
Minify your code
Use tools like Uglify or Closure Compiler to make your javascript lighter. Though you must be careful, as these tools will fit all your code a single long line by the default, and you can't read lines longer than 255 characters with LSL. Luckily you can customize your options in these tools to limit the amount of characters per line.
Divide and conquer
Since a single script can't handle much memory, make dedicated scripts. One could serve resources (act as a file server, providing your html and js) while the other would receive API request calls, to handle the application's logic.
We recently upgraded to a newer build of a JavaScript minification library.
After a significant amount of quality assurance work by the testing team, it was discovered that the new version of our minifier had an issue that changed the intention and meaning behind a block of code.
(Life lesson: don't upgrade JS minifiers unless you are really convinced you need the new version.)
The minifier is used for client side JavaScript code with a heavy emphasis on DOM related activity, not nearly as much "business logic".
A simplified example of what was broken by the minifier upgrade:
function process(count)
{
var value = "";
value += count; //1. Two consecutive += statements
value += count;
count++; //2. Some other statement
return value; //3. Return
}
Was minified incorrectly to the following:
function process(n){var t="";return t+n+n,n++,t}
While we could write some unit tests to catch some of the issues potentially, given that the JavaScript is heavy on DOM interactions (data input, etc.), it's very difficult to test thoroughly without user testing (non-automated). We'd pondered using a JS to AST library like Esprima, but given the nature of the changes that could be done to the minified code, it would produce far too many false positives.
We also considered trying to write representative tests, but that seems like a never-ending task (and likely to miss cases).
FYI: This is a very sophisticated web application with several hundred thousand lines of JavaScript code.
We're looking for a methodology for testing the minification process short of "just test everything again, thoroughly, and repeat." We'd like to apply a bit more rigor/science to the process.
Ideally, we could try multiple minifiers without fear of each breaking our code in new subtle ways if we had a better scientific method for testing.
Update:
One idea we had was to:
take minification with old version
beautify it
minify with new version,
beautify, and
visually diff.
It did seem like a good idea, however the differences were so common that the diff tool flagged nearly every line as being different.
Have you considered a unit test framework, such as QUnitjs ? It would be quite a bit of work to write the unit tests, but in the end you would have a repeatable test procedure.
Sounds to me like you need to start using automated Unit Tests within your CI (continuous integration environment). QUnit has been thrown around, but really QUnit is a pretty weak testing system, and its assertions are barebones at the minimum (it doesn't even really use a good assertion-based syntax). It only marginally qualifies as TDD and doesn't handle BDD very well either.
Personally I'd recommend Jasmine with JsTestDriver (it can use other UT frameworks, or its own, and is incredibly fast...though it has some stability issues that I really wish they'd fix), and setup unit tests that can check minification processes by multiple comparisons.
Some comparisons would likely need to be:
original code & its functionality behaves as expected
compared to minified code (this is where BDD comes in, expect the same functional performance/results in minified code)
I'd even go a step further (depending on your minification approach), and have a test that then beautifies the minification and does another comparison (this makes your testing more robust and more ensured of validity).
These kinds of tests are why you would probably benefit from a BDD-capable framework like Jasmine, as opposed to just pure TDD (ala the results you found of a visual diff being a mess to do), as you are testing behavior and comparisons and prior/post states of functionality/behavior, not just if a is true and still true after being parsed.
Setting up these Unit Tests could take a while, but its an iterative approach with that large of a codebase...test your initial critical choke points or fragile points fast and early, then extend tests to everything (the way I've always set my teams up is that anything from this point on is not considered complete and RC unless it has Unit Tests...anything old that has no Unit Tests and has to be updated/touched/maintained must have Unit Tests written when they are touched, so that you are constantly improving and shrinking the amount of untested code in a more manageable and logic way, while increasing your code coverage).
Once you have Unit Tests up and running in a CI, you can then tie them into your build process: fail builds that have no unit tests, or when the unit tests fail send out alerts, proactively monitor on each checkin, etc. etc. Auto-generate documentation with JSDoc3, etc. etc.
The issue you are describing is what CI and Unit Tests were built for, and more specifically in your case that approach minimizes the impact of the size of the codebase...the size doesn't make it more complex, just makes the duration to get testing working across the board longer.
Then, combine that with JSDoc3 and you are styling better than 90% of most front end shops. Its incredibly robust and useful to engineers at that point, and it becomes self-perpetuating.
I really could go on and on about this topic, there's a lot of nuance to how you approach it and get a team to rally behind it and make it self-forming and self-perpetuating, and the most important one being writing testable code...but from a concept level...write unit tests and automate them. Always.
For too long frontend devs have been half-assing development, not applying actual engineering rigor and discipline. As frontend has grown more and more powerful and hot, that has to change, and is changing. The concept of well tested, well covered, automated testing and continuous integration for frontend/RIA applications is one of the huge needs of that change.
You could look at something like Selenium Web Driver Which allows you to automate tests for web applications in various environments. There are some cloud hosted VM solutions for doing multi environment testing, so you don't get caught out when it works in Webkit but not in IE.
You should definitely look into using source maps to help with debugging minimized JavaScript. source maps will also work with supersets of JavaScript such as CoffeeScript or my new favorite TypeScript.
I use closure compiler which not only minifies, but also will create the source maps. not to mention it is the most agressive and produces the smallest files. Lastly you kinda have to know what's going on in minification and write compatible code, your example code could use some refactoring.
check out this article on source maps:
http://www.html5rocks.com/en/tutorials/developertools/sourcemaps/
also check out the documentation for closure compiler, it's got suggestions on how to write better code for minification:
https://developers.google.com/closure/compiler/
Not a testing solution, but how about switching to TypeScript to write large JS apps like yours?
I tested it out with TypeScript and its default min engine and it works fine.
Assuming your count arg is an number.
The type script will be:
class ProcessorX {
ProcessX(count: number): string {
var value = '';
value += count.toString();
value += count.toString();
count++;
return value;
}
}
Which produces js like this:
var ProcessorX = (function () {
function ProcessorX() { }
ProcessorX.prototype.ProcessX = function (count) {
var value = '';
value += count.toString();
value += count.toString();
count++;
return value;
};
return ProcessorX;
})();
Then minified to:
var ProcessorX=function(){function n(){}return n.prototype.ProcessX=function(n){var t="";return t+=n.toString(),t+=n.toString(),n++,t},n}()
It's on jsfiddle.
If your count is a string, then this fiddle.
we use closure compiler in advanced mode which both minifies and changes code, as such we compile our unit tests as well, so you could consider minifying your tests alongside your code and running it like that.
I am currently using jQuery to write an online application, that started off with a couple of lines of code, and have quickly now become over a 1000 lines.
My code's structure is simple. I have a window.load which wraps my javascript, and inside it I start adding my click event handlers, and the various functions that makeup my application.
$(window).load(function(){
// Code goes here...
});
My code functions can definitely be grouped into categories; e.g. 5 functions perform animation, 12 are event handlers, etc.
I would like to group the functions in their own js files, and import them individually. I can later use my CMS engine to concatenate and compress the files on the fly.
What is the best way in doing so. I am thinking that maybe I can give some of my functions their own namespace for further clarity; e.g. all animation functions are prefixed with ANIMATION - ANIMATION.moveDiv1(), ANIMATION.moveDiv2, MYEVENT.div1Clicked, etc.
I generally stick all related items into their own file, with a namespace that matches the file for readability sake.
An example file could look like:
Example.js
var Animation = {}; // create the namespace-like object
Animation.moveDiv1 = function() {...};
Animation.moveDiv2 = function() {...};
There's really a lot of ways to do this. Speaking of compression, there are some nice tools that you can use to compress things. Check out YUI Compressor
Modularity is a good goal with Javascript, but I would say the next level would be to actually use some Javascript OO techniques. If your app is simple enough, you can probably do without it though.
Your code files should mirror your classes.
Your classes should follow principles of good OO design.
In terms of load-time within the browser, kekoav and knut have the right idea - just use YUI or another script compressor/minifier (and optionally an obfuscator), combine them into a single file and load them all from a single script include directive.
I'd also have a look at JS the prototype property of your classes - if they're getting large and you're creating multiple instances of them, you'll start to see significant performance gains by putting your public (and optionally, private/privileged) methods into the class prototype.
You should definitely be using fully-qualified namespaces for your classes, either using Microsoft's Type.registerNamespace if you're using their AJAX solution, by declaring your own namespace functions as per kekoav's post, or using a squillion other similar approaches that Google will offer.
Good idea from a standpoint of application management, bad idea from the standpoint of loading time. The browser has to load all those little scripts synchronously, therefore taking more time for each additional script you want to load. That's not including the main jQuery library script, the jQuery UI and whatever else you plan on having in your document. Test both premises: abstracting functions out into their own individual scripts and load them vs. one big script that only requires one call to load. Take it a step further and minify the "one big script", one more step and make sure it's served as a compressed file.
You may split the JavaScript files into classes when you are developing, but you should combine your scripts, and minimize them in a production environment. Please take a look at YUI Compressor for more information.
My project seems to be getting bigger and bigger and some of my classes are thousands of lines long. It's too hard to search through them every time I want to make changes.
I find JavaScript is not as easy to lay out cleanly as some other programming languages. So when the classes get to be a few thousand lines, I have troubles reading it.
I've tried splitting it into multiple files, but then you're breaking classes apart, which doesn't seem right. For example, if every method in a class uses a global variable, you would only be able to find the global variable in one of the files for that class.
Also, if I want to use the JavaScript code from 100 different .js files, I end up with something like this...
<script type="text/javascript" src="Scripts/classes/Node.js"></script>
<script type="text/javascript" src="Scripts/classes/Queue.js"></script>
<script type="text/javascript" src="Scripts/classes/DblyLinkedList.js"></script>
.... 97 more lines like this
Although, I figured there may be something where I can do...
<script type="text/javascript" src="Scripts/.../*.js"></script>
or something similar... is that right?
Anyone have any tips on managing code as it reaches its extremes?
Tips on cleaning up JavaScript code would also be helpful.
Breaking up JS into separate files has some major drawbacks, chiefly that you're forcing web browsers to make a separate request for each file.
Have you taken a look at leaving all of your files separated out, but making a single-file "bundle" for each project containing only the necessary files? This can be automated via a script.
This SitePoint article might help you get started: http://www.sitepoint.com/blogs/2007/04/10/faster-page-loads-bundle-your-css-and-javascript/
(a) keep your classes shorter [even
though that will mean yet more
files],
(b) keep them "full-text
indexed" for speed of search and
operation (not aware of any IDE
specifically supporting Javascript
this way, but strong editors like
Emacs, Vim, Eclipse, or TextMate sure
do),
(c) group them up hierarchically
so your pages can have just a few
<script> tags for "upper layer"
scripts each of which just pulls in
several of the "lower layer" ones.
Oh, and, of course, religiously keep everything under a good change control system (SVN, Mercurial, or the like), otherwise your life will be surely very very miserable:-(.
You might want to group related classes together into packages, where each package is a single file. Check out YSlow for best practices on performance.
Well, a good editor is always usefull, as it will give you shortcuts to all your functions defined in the files.
Second, make sure you're not looking at a wall of code. Indentation, spaces and newlines are a good help.
Be extremely strict in your indentation. 2 spaces is 2 spaces, always(or whatever amount you use)
if you put your { under a declaration, then always put it there, without exception)
Clear rules about how you want your text aligned will help a lot.
And I don't know about that last thing... I'm not sure browsers can work with that kind of wildcard.
<script type="text/javascript" src="Scripts/.../*.js"></script>
will not work, and besides, you end up with an interesting problem when splitting up dependent files, as you can't guarantee they will all be downloaded in the order you expected.
Unfortunately, your best bet is really a good IDE that can produce an outline for easy navigation. I use Eclipse with Spket and/or Aptana, but whatever makes it easier to manage is what you're looking for.
edit: one more quick note about splitting js into multiple files. Avoid it where possible. Each separate file means a separate http request. Reducing the number of requests required can make a massive difference in site performance.
AvatarKava's advice is sound. Work in separate files and concatenate them at build time. However, I would also recommend you take a look at your class structure. A class "thousands of lines long" doesn't sound too healthy. Are you sure you classes aren't taking on too much responsibility? Are there not tasks that can be shipped out to other single responsibility classes? This would help improve the clarity in your code far more than cunning ways of splitting files.
Same advice applies to most languages...
Avoid globals wherever possible. You can often get the desired behavior by wrapping a static variable, or using objects.
if (foo == undefined)
var foo
Use naming conventions where possible so you can track things down just by reading the variable or function names. Either that or get good with grep, or get an IDE with intellisense.
Split things into directories. Having 100 files in a directory called "classes" is not helpful. As an example, you may have a collections directory for queues, lists, trees, etc. If you've got a lot you may even have a tree subdir, or a list subdir, etc.
You can then also create a global include for that directory... simply a file called collections.js that includes all of the files in that directory. Of course you have to be careful about splitting things up well, since you don't want to be including files you'll never use.