I've written a lot of asynchronous unit tests lately, using a combination of Angular's fakeAsync, returning Promises from async test body functions, the Jasmine done callback, etc. Generally I've been able to make everything work in a totally deterministic way.
A few parts of my code interact in deeply-tangled ways with 3rd party libraries that are very complex and difficult to mock out. I can't figure out a way to hook an event or generate a Promise that's guaranteed to resolve after this library is finished doing background work, so at the moment my test is stuck using setTimeout:
class MyService {
public async init() {
// Assume library interaction is a lot more complicated to replace with a mock than this would be
this.libraryObject.onError.addEventListener(err => {
this.bannerService.open("Load failed!" + err);
});
// Makes some network calls, etc, that I have no control over
this.libraryObject.loadData();
}
}
it("shows a banner on network error", async done => {
setupLibraryForFailure();
await instance.init();
setTimeout(() => {
expect(banner.open).toHaveBeenCalled();
done();
}, 500); // 500ms is generally enough... on my machine, probably
});
This makes me nervous, especially the magic number in the setTimeout. It also scales poorly, as I'm sure 500ms is far longer than any of my other tests take to complete.
What I think I'd like to do, is be able to tell Jasmine to poll the banner.open spy until it's called, or until the test timeout elapses and the test fails. Then, the test should notice as soon as the error handler is triggered and complete. Is there a better approach, or is this a good idea? Is it a built-in pattern somewhere that I'm not seeing?
I think you can take advantage of callFake, basically calling another function once this function is called.
Something like this:
it("shows a banner on network error", async done => {
setupLibraryForFailure();
// maybe you have already spied on banner open so you have to assign the previous
// spy to a variable and use that variable for the callFake
spyOn(banner, 'open').and.callFake((arg: string) => {
expect(banner.open).toHaveBeenCalled(); // maybe not needed because we are already doing callFake
done(); // call done to let Jasmine know you're done
});
await instance.init();
});
We are setting a spy on banner.open and when it is called, it will call the callback function using the callFake and we call done inside of this callback letting Jasmine know we are done with our assertions.
I am reading some tutorials on promise tests in mocha. There is a piece of codes:
before(function(done) {
return Promise.resolve(save(article)).then(function() {
done();
});
});
Why done() called in the then() in the before()? What is the difference between the above codes and the following codes:
before(function(done) {
return Promise.resolve(save(article));
});
Thanks
UPDATE
My question is to compare with the following codes:
before(function() {
return Promise.resolve(save(article));
});
Sorry for the typo.
The first code snippet with the before hook returns a promise and calls done. In Mocha 3.x and over, it will result in this error:
Error: Resolution method is overspecified. Specify a callback *or* return a Promise; not both.
It used to be that it did not particularly matter if you used done and returned a promise, but eventually the Mocha devs figured that specifying both done and returning a promise just meant the test designer made a mistake and it was better to have Mocha pitch a fit rather than silently allow it.
In your 2nd snippet, you have the done argument and return a promise but Mocha will still wait for done to be called and will timeout. (It really should detect the argument and raise an error like in the 1st case, but it doesn't...)
Generally, if you are testing an asynchronous operation that produces a promise, it is simpler to return the promise than use done. Here's an example illustrating the problem:
const assert = require("assert");
// This will result in a test timeout rather than give a nice error
// message.
it("something not tested properly", (done) => {
Promise.resolve(1).then((x) => {
assert.equal(x, 2);
done();
});
});
// Same test as before, but fixed to give you a good error message
// about expecting a value of 2. But look at the code you have to
// write to get Mocha to give you a nice error message.
it("something tested properly", (done) => {
Promise.resolve(1).then((x) => {
assert.equal(x, 2);
done();
}).catch(done);
});
// If you just return the promise, you can avoid having to pepper your
// code with catch closes and calls to done.
it("something tested properly but much simpler", () => {
return Promise.resolve(1).then((x) => {
assert.equal(x, 2);
});
});
With regards to the completion of asynchronous operations, it works the same whether you are using it, before, beforeEach, after or afterEach so even though the example I gave is with it, the same applies to all the hooks.
I am not sure if I understood 100% the question, but the tests will not start until done is called.
beforeEach(function(done) {
setTimeout(function() {
value = 0;
done();
}, 1);
});
This test will not start until the done function is called in the call to beforeEach above. And this spec will not complete until its done is called.
it("should support async execution of test preparation and expectations", function(done) {
value++;
expect(value).toBeGreaterThan(0);
done();
});
You don't have to pass done in your example, just:
before(function() {
return Promise.resolve(save(article));
});
If you do pass done the test runner will expect to be called before continue, otherwise it will probably throw a timeout error.
In this particular case there is no functional difference. The callback, often called done, was introduced to handle asynchronous tests when using callbacks. Returning a Promise is sufficient, but note that you cannot define the done callback, because the test suite will wait until it's called. Use done when you can't easily return a Promise. In your case the second test will be infinite, because you define done, which you never actually call.
I am trying to test the postMessage API as there is a slight delay before message are receive i can not run expectation right after sending a message.
In jasmine 1.3 i used to wait() a few milliseconds before running expectation and that worked fine. However with jasmine 2.0 wait() is deprecated and it now seems that everything inside a setTimeout do not get run unless done() is called, witch in my case doesn't cut it as i actually want to wait real time before running my expectation..
Not sure if that all make sense, if it does I'd love some pointers on how I could go about this.
Thanks!
This works for me:
beforeAll(function (done) {
setTimeout(done, 5000);
});
The beforeAll function occurred first, but it will end when you invoke the done callback function. So if you use setTimeout function with 5000, it will wait 5000 milliseconds before continuing.
Rather than waiting some number of milliseconds, jasmine has hooks to wait until a function returns. This page has some good example, and I've copied one here to show a specific way of testing ajax callbacks. Just add a spy as a callback to your function and wait for that callback to be executed.
it("should make a real AJAX request", function () {
var callback = jasmine.createSpy();
makeAjaxCall(callback);
waitsFor(function() {
return callback.callCount > 0;
}, "The Ajax call timed out.", 5000);
runs(function() {
expect(callback).toHaveBeenCalled();
});
});
EDIT:
Since you're testing that your application makes a specific callback, you can just replace that callback with a spy instead of creating a new one like I did.
Jasmine 2.0 added a "done" style callback, so you should be able to do something like: (I haven't tested the syntax of this but hopefully a good start)
it("should make an ajax callback with jasmine 2.0", function(done)) {
// this is the object you are testing - assume it has the ajax method you want to call and the method that gets called when the ajax method is finished
var myObject
spyOn(myObject, "callback").andCallFake(function() {
done();
});
myObject.makeAjaxCall();
}
I use jasmine runs and wait to test asynchronous operations. Everything works fine but I'm not quite sure what goes on behind the scenes.
The jasmine documentation states the following example to which I added three log statement.
describe("Asynchronous specs", function() {
var value, flag;
it("should support async execution of test preparation and exepectations", function() {
runs(function() {
flag = false;
value = 0;
setTimeout(function() {
flag = true;
}, 500);
});
waitsFor(function() {
value++;
if(flag) {
console.log("A");
}
return flag;
}, "The Value should be incremented", 750);
console.log("B");
runs(function() {
console.log("C");
expect(value).toBeGreaterThan(0);
});
});
});
});
The first runs and waitsFor are perfectly clear to me. Runs starts an asynchronous operation and waitsFor waits for a condition.
However I do not understand why the second runs does not start until the waitsFor is finished. The waitsFor is not a blocking call.
My guess is that waitsFor implicitly blocks any following runs call until it is finished. Is this so?
My evidence is that the console.log statements output:
B A C
But if waitsFor would really block it should be
A B C
waitsFor does block until the conditions it's waiting for are met or it times out.
From the jasmine docs: "waitsFor() provides a better interface for pausing your spec until some other work has completed. Jasmine will wait until the provided function returns true before continuing with the next block.".
The linked docs also have a slightly clearer example or waitsFor.
EDIT: Ah I see what you mean now. waitsFor won't block JS that isn't wrapped in runs, waitsFor, ect.
What jasmine is doing is taking the function passed to it via runs or waitsFor and if jasmine is not currently waiting, it executes the function immediately. If it is waiting, it doesn't call it until it's finished waiting.
That doesn't stop the console.log as it's been passed to jasmine so jasmine can't prevent it from being executed straight away.
The solution is in the documentation:
Multiple runs() blocks in a spec will run serially. (Jasmine Documentation)
From the site: http://www.htmlgoodies.com/beyond/javascript/test-asynchronous-methods-using-the-jasmine-runs-and-waitfor-methods.html#fbid=mzNDUVfhFXg
Jasmine will call the runs() and waitsFor() methods in the order you
passed them. As soon as the JS parser gets to a waitsFor() method it
will poll it until it returns true and only then will it continue onto
the next runs() method.
Essentially, the runs() and waitsFor() functions stuff an array with their provided functions. The array is then processed by jamine wherein the functions are invoked sequentially. Those functions registered by runs() are expected to perform actual work while those registered by waitsFor() are expected to be 'latch' functions and will be polled (invoked) every 10ms until they return true or the optional registered timeout period expires. If the timeout period expires an error is reported using the optional registered error message; otherwise, the process continues with the next function in the array. Presumably, expects within the runs() can also trigger a failure report (and perhaps even in the latch functions themselves).
The documentation is particularly obtuse on these asynchronous features.
In Javascript, I have two versions of a recursive function, one that runs synchronously and one that uses simple scheduling to run asynchronously. Given certain inputs, in both cases the function is expected to have an infinite execution path. I need to develop tests for these functions, specifically a test to check that the asynchronous version does not block the main thread.
I already have tests that check the output callback behavior of these functions in non-returning cases, I am only concerned about testing the blocking behavior. I can limit how long the function runs to some long but finite amount of time for testing purposes as well. I am currently using QUnit but can switch to another testing framework.
How can I test that a non-returning, asynchronous function does not block?
Edit, For Clarification
This would be a bare bones example of the function I am working with:
function a()
{
console.log("invoked");
setTimeout(a, 1000);
}
a();
I am intentionally misusing some threading terms in my description because I felt they most clearly expressed the problem. By not blocking the main thread, I mean that invoking the function does not prevent the scheduling and execution of other logic. I expect the function itself will be executed on the main thread but I consider the function running as long as it is scheduled for execution in the future.
Unit Test are based on single-responsability-principle and isolation (separate the subject under test from it's dependencies).
In this case, you expect your function to run asynchronously but this behaviour is not done by your function, is done by the "setTimeout" function, so I think you must isolate your function from "setTimeout" since it's a dependency you don't want to test, the browser guarantees you it will work.
Then, as we trust "setTimeout" will do the asyncrhonous logic, we can only test our function calls to "setTimeout" and we can do this replacing "window.setTimeout" with another function while we must always restore it after the test is complete.
function replaceSetTimeout() {
var originalSetTimeout = window.setTimeout;
var callCount = 0;
window.setTimeout = function() {
callCount++;
};
window.setTimeout.restore = function() {
window.setTimeout = originalSetTimeout;
};
window.setTimeout.getCallCount = function() {
return callCount;
};
}
replaceSetTimeout();
asyncFunction();
assert(setTimeout.getCallCount() === 1);
setTimeout.restore();
I recommend you to use sinon.js as it provides many tools like spies who are functions than will inform you about how many times and with what arguments where called.
var originalSetTimeout = window.setTimeout;
window.setTimeout = sinon.spy();
asyncFunction();
// check called only once
assert(setTimeout.calledOnce);
// check the first argument was asyncFunction
assert(setTimeout.calledWith(asyncFunction));
Sinon also provides fake timers who does the setTimeout substitution but with so much more features, like the .tick(x) method who will simulate "x" milliseconds but in this case I think it doesn't help you.
Update to answer question edit:
1 - Your function executes infinitely so you cannot test it without interrupting it's execution, so you must overwrite "setTimeout" somewhere.
2 - You want your function to execute recursively allowing other code to be executed between iterations? great! but understand than your function can not do this your function only can call setTimeout or setInterval and hope this function work as expected. You should test what your function does.
3 - You want to test from Javascript (a sandboxed environment) than another Javascript code uses and releases the only one execution thread (the same you are using to test). Do you really think this is an easy test?
4 - but the most important one - I don't like white box because it couples the test with the dependency, if you change your dependency or how it's called in the future you will have to change the test. This problem doesn't exist with DOM function, DOM functions will keep the same interface for years, and for now, you have no other way to do what you want than calling one of those two functions, so I don't think in this case "white box testing" is a bad idea.
I told you this because I had the same problem testing a Promise pattern implementation than had to be always asynchronous, even if the promise is already fulfilled, and I've tested it using test-engine asynchronous-test way (using callbacks and stuff) and it was a mess, test failing randomly, so much slow test execution. Then I asked a TDD expert how can test be so hard and he answered than I was not following Single Responsability Principle since I was trying to test my promise implementation AND the setTimeout behaviour.
If you think about it from a Behaviour Driven Testing perspective then 'Does my function block?' is not a useful question. It will definitely block, a better question might be 'does it return in no more than 50ms'.
You could do this with something like :
test( "speed test", function() {
var start = new Date();
a();
ok(new Date() - start < 50, "Passed!" );
});
The issue with this is that if someone does do something silly that makes your function block indefinitely the test won't fail, it will hang.
Because JavaScript is single threaded there is no way around this. If I come along and change your function to :
function a() {
while(true) {
console.log("invoked")
}
}
The test will hang.
You can make breaking things this way harder by refactoring things a little. There are 2 separate things being done. Your chunk of work and the scheduling. Separate these and you'll end up with something like the following functions :
function a() {
// doWork
var stopRunning = true;
return stopRunning;
}
function doAsync(workFunc, scheduleFunc, timeout) {
if (!workFunc()) {
scheduleFunc(doAsync, [workFunc, scheduleFunc, timeout], timeout);
}
}
function schedule(func, args, timeout) {
setTimeout(function() {func.apply(window, args);}, timeout);
}
Now you're free to test everything in isolation. You can supply a mock workFunc and scheduleFunc to a test for doAsync to verify it behaves as expected and you can test your function a() without worrying about how it is scheduled.
It's still possible for a dunce programmer to put an infinite loop into the function a(), but because they don't have to consider how to run further units of work it should be less likely.
To test or prove an infinitely executing execution path will never block is next to impossible, so you have to split your problem up into parts.
Your path is basically foo(foo(foo(foo(...etc...)))), nevermind that SetTimeout actually removes recursion. So all you have to do is test or prove that your foo does not block (I tell you now that testing will be "a bit" easier than proving, more below)
So, does function foo block?
Talking a bit maths, if you want to know whether f(f(...f(x)...)) always has a value, you actually only have to prove that f(x) always has a value for any x that f can return. It does not matter how many recursions you have, if you can make sure their return values are fine.
What that means for your foo is that you only have to prove that foo does not block for any possible input value. Keep in mind that in this case, all global variables and closures are input values too. This means you have to sanity-check every single value you are using on every call.
To test, of course you will have to replace SetTimeout, but that is trivial, and if you replace it with an empty function (function(){}) it is easy to prove that this function does not block or otherwise alter your execution. You will then
Making things easier
Taking in what I wrote above, this also means that you would have to make sure no global function or variable that you are ever using will ever be changed to a point that your function breaks to a point it breaks. This actually is quite hard, but you can still make things easier for you by making sure you always use the same functions and values and that other functions can not touch them by using closures.
function foo(n, setTimeout)
{
var x = global_var;
// sanity check n here
function f()
{
setTimeout(f, n)
}
return f();
}
This way, you only have to test those values on the first execution. It's nice to be able to assume Math.Pi is actually Pi and not a string value containing "noodles". Really nice.
Do not use global mutable objects
Call those you can not circumvent using setTimeout to ensure they can not block
If you need return values, things will get really tricky, but possible, consider this:
function() {
var x = 0;
setTimeout(function(){x = insecure();}, 1);
}
All you have to do is
Use x next iteration
Sanity check value of x first!
Does SetTimeout block?
Of course this depends on whether setTimeout blocks. This is quite hard to prove, but a bit easier to test. You can't actually prove it since it's implementation is up to the interpreter.
Personally I would assume that setTimeout behaves like an empty function when it's return value is discarded.
Performing this asynchronous testing is actually possible in QUnit but is handled better in another JavaScript testing framework, Jasmine JS. I'll provide examples in both.
In QUnit you need to first call the stop() function to signal that the test is expected to run asynchronously, you should then call setTimeout with a function that includes your expectations as well as a call to the start() function to complete the block. Here's an example:
test( "a test", function() {
stop();
asyncOp();
setTimeout(function() {
equals( asyncOp.result, "someExpectedValue" );
start();
}, 150 );
});
Edit: Apparently there's also a whole asyncTest construct that you can use that simplifies this process. Take a look: http://api.qunitjs.com/asyncTest/
In Jasmine (http://pivotal.github.com/jasmine/), a Behavior Driven Development (BDD) testing framework, there are built-in methods for writing asynchronous tests. Here's an example of an asynchronous test in Jasmine:
describe('Some module', function() {
it('should run asynchronously', function() {
var isDone = false;
runs(function() {
// The first call to runs should trigger some async operation
// that has a side-effect that can be tested for. In this case,
// lets say that the doSomethingAsyncWithCallback function
// does something asynchronously and then calls the passed callback
doSomethingAsyncWithCallback(function() { isDone = true; });
});
waitsFor(function() {
// The call to waits for is a polling function that will get called
// periodically until either a condition is met (the function should return
// a boolean testing for this condition) or the timeout expires.
// The optional text is what error to display if the test fails.
return isDone === true;
}, "Should set isDone to true", 500);
runs(function() {
// The second call to runs should contain any assertions you need to make
// after the async call is complete.
expect(isDone).toBe(true);
});
});
});
Edit: Also, Jasmine has several built-in methods of faking out the setTimeout and setInterval functions of the browser without hosing any other tests in your suite that may depend on that. I would take a look at using those rather than manually overriding the setTimeout/setInterval functions.
Basically, JavaScript is single-threaded, so it will block the main thread. But :
I assume you're using setTimesout to schedule your function, so it won't be noticeable to the user if calls to that function don't take too much time (say, less than 200 or 300ms).
If you're doing DOM manipulation during that function (including Canvas or WebGL), then you're screwed. But if not, you can look into Web Workers, which can spawn separate threads that are guaranteed not to block the UI.
But anyway, JavaScript and the main loop, that's a tricky issue that's been bugging me a lot these past months, so you're not alone!
As soon as your function returns (after having set the timeout for it's next run), javascript will look at the next thing that requires running and run that.
As far as I can tell, the 'main thread' in javascript is just a loop that is responding to events (such as onload for a script tag, which runs the contents of that tag).
Based on the above two conditions, the calling thread is always going to run to completion despite any setTimeouts, and those timeouts will begin after the calling thread has nothing left to run.
The way I tested this was to run the following function right after the call to a()
function looper(name,duration) {
var start = (new Date()).getTime();
var elapsed = 0;
while (elapsed < duration) {
elapsed = (new Date()).getTime() - start;
console.log(name + ": " + elapsed);
}
}
Duration should be set to some period of time longer than the setTimeout duration in a(). The expected output would be the output of 'looper', followed by the output of repeated calls to a().
The next thing to test would be whether other script tags are able to run while a() and its child calls are executing.
You can do this like so:
<script>
a();
</script>
<script>
looper('delay',500); // ie; less than the 1000 timeout in a();
</script>
<script>
console.log('OK');
</script>
You would expect 'OK' to appear in the log despite the fact that a() and its children are still executing. You can also test variations of this, such as window.onload(), etc.
Finally, you'd want to ensure that other timer events work fine as well. Simply delaying 2 calls by half a second and checking that they interleave should show that works fine:
function b()
{
console.log("invoked b")
setTimeout(b, 1000);
}
a();
looper('wait',500);
b();
Should produce output like
invoked
invoked b
invoked
invoked b
invoked
invoked b
Hope that's what you were looking for!
EDIT in case you need some technical details on how to do it in Qunit:
If Qunit can't capture console.log output (i'm not sure), just push those strings into an array or a string and check that after it's run. You could override console.log in the test module() setup and restore it at teardown. I'm not sure how Qunit works but 'this' might have to be removed and globals used to store the old_console_log and test_output
// in the setup
this.old_console_log = console.log;
this.test_output = [];
var self = this;
console.log = function(text) { self.test_output.push(text); }
// in the teardown
console.log = this.old_console_log;
Finally, you can utilize stop() and start() so that Qunit knows to wait for all the events in the test to finish running.
stop();
kickoff_async_test();
setTimeout(function(){
// assertions
start();
},<expected duration of run>);
Based on all the answers, I came up with this solution that works for my case:
testAsync("Doesn't hang", function(){
expect(1);
var ranToLong = false;
var last = new Date();
var sched = setInterval(function(){
var now = new Date();
ranToLong = ranToLong || (now - last) >= 50;
last = now;
}, 0);
// In this case, asyncRecursiveFunction runs for a long time and
// returns a single value in callback
asyncRecursiveFunction(function callback(v){
clearInterval(sched);
var now = new Date();
ranToLong = ranToLong || (now - last) >= 50;
assert.equal(ranToLong, false);
start();
});
});
It tests that 'asyncRecursiveFunction' does not hang while processing by looking at the time between another scheduled function calls.
This is really ugly and not be applicable to every case but it seems to work for me because I can restrict my function to some large set of async recursive calls so it runs for a long but not infinite time. As I mentioned in the question, I am happy proving that such cases do not block.
BTW, the actual code in question is found in gen.js. The main problem was an async reduce generator. It correctly returned a value asynchronously, but in previous versions would stall because of synchronous internal implementation.