There's a lot of interesting and compelling features in the experimental "Behavior Driven" extentions to the Scriptaculous Unit Testing Framework. I'd like to see some of them incorporated into Crosscheck, but others... well, not so much.
When I contrast Crosscheck and the Scriptactulous Unit Testing framework, or SUT from here on out, the comparison is only skin deep. The SUT consists only of a testing API, so if your intent is to test code's interaction with a browser, then you have to run it inside a browser. Crosscheck on the other hand provides both a testing API and a suite of simulated browser environments, and so it cannot be run in an actual browser. Where and how you would use them are completely different; each with its pros and cons, but I won't discuss that here. I'm just talking about the differences in the style of the testing API itself
The Good
Let's start with an example of the SUT in action:
Test.context("Behavior Driven Assertions",{
'objects know how to compare other objects against themselves': function() {
'one'.shouldEqual('one');
'one'.shouldNotEqual('two');
}
})
The analogous crosscheck code, which uses the more classical assertion style:
crosscheck.addSuite("Classic Assertions", {
test_assertions_in_classic_form: function() {
assertEquals('one', 'one')
assertNotEquals('one', 'two')
}
})
For starters, I find the SUT style of naming tests with a free-form string to be very attractive. Notice how the test name sets itself apart from the rest of the code. There aren't any of those irritating underscores to interfere with the test cases name. It's such a small thing, but it does alot to convey the meaning of the code. Though it hadn't occured to me to use it as our convention, a similar effect can be achieved in Crosscheck.
crosscheck.addSuite("Alternate Test Naming Style", {
'test assertions in classic form': function() {
assertEqual('one', 'one')
assertNotEqual('two', 'two')
}
})
This works because all crosscheck does is look for functions whose names start with the string "test." The rest of the string isn't important, so this works without changing anything. This does, however, highlight another subtle difference between the two: SUT doesn't even have the restriction of beginning your test functions with "test"! Once again, I find this style very attractive. Those who know me know that I loath repetition (example: I despise using semi-colons in javascript except where needed), and so the opportunity to eliminate the repetition of needlessly prepending "test" in front of every test method is tempting.
As much as I like it, this one isn't without it's drawbacks. The problem is, if you allow any string to indicate a test, then how do you determine which methods are test methods, and which are plain vanilla methods of the testcase itself? In our tests, we frequently use instance methods to provide helper functions or custom assertions to the specific testcase
Example:crosscheck.addSuite("Differentiating Test Cases", {
'any string can be a test name': function() {
this.assertValidTestName(this.name) //-> in crosscheck all test cases have 'name' property
},
assertValidTestName: function(name) {
assertEquals('string', typeof name)
}
})
The above has an obvious flaw. How can the framework determine that "any string can be a test name" is a test method, but that "assertValidTestName" is not. It could check for function properties that have interlaced whitespace, but then what about test cases that are just a single word? The heuristics for what's a testcase and what isn't soon become complex and altogether too sketchy for my taste. The only solution I see is to wrap the testcases in their own scope so that they are completely disambiguated from the rest of the test class:
crosscheck.addSuite("Disambiguated Test Space", {
tests: {
'any string can be a test name': function() {
this.assertValidTestName(this.name)
}
},
assertValidTestName: function(name) {
assertEquals('string', typeof name)
}
})
I have mixed feelings about this one. While the free-form naming of tests make them stand out, it comes at the cost of another level of indentation and "curly clutter" which cuts into the gains to be had therein. Unless I'm missing something, to seperate the name spaces, SUT must employ some strategy like this, or else forbid any functions on a testcase other than test definitions (ick!).
But it isn't just how you collect tests. That's thinking from the framework's perspective. The whole point of what they're trying to do with these SUT extensions(I think) is to make the tests read like an expectation of how the code under test should behave
Let's look at that first SUT snippet again:Test.context("Behavior Driven Assertions",{
'objects know how to compare other objects against themselves': function() {
'one'.shouldEqual('one');
'one'.shouldNotEqual('two');
}
})
The first time I looked at the SUT, I thought Test.context()?!? That's a weird, vague, and overused term. Everybody has a context, and right now my current context is "What the hell are you talking about?" but after subsequent reading, and if I correctly divine the SUT authors' intent, they chose context, because they want the testcase to read as a statement to the effect that
In the context of Behavior Driven Assertions, objects know how to compare other objects against themselves.
The compelling principle here is that the mechanics of test assembly are subordinate to expressing the semantic significance which the test implies. By comparison, crosscheck.addSuite() seems to tell you more about how the tests are constructed and organized than what they mean.
Perhaps we can apply this principle to the problem of how to differentiate test functions from plain-vanilla function by choosing an "expressive" namespace
Test.context("Behavior Driven Assertions",{
'such that': {
'objects know how to compare other objects against themselves': function() {
'one'.shouldEqual('one');
'one'.shouldNotEqual('two');
}
},
'assuming': function() { //-> this becomes the setup function
//create the test fixture
},
//normal methods go here.
})
Perhaps the choice of words could be better, but you get the picture. Which leads us to the question: does anyone besides test framework authors such as myself and those of the SUT give half a rat's ass about the form that they're tests take? Unless they want to try and document their code via the tests (a laudable goal, but one which I've yet to see realized effectively) people are, in all probability, more interested in the mechanics of test assembly rather than the semantic implications of the tests themselves. I.e. "huh? so what functions actually gets run when I run the test, and when?" That said, I see no reason that both styles can't peacefully coexist inside the same framework.
The Ugly
While I found much of value in the behavior driven SUT style, I did find one aspect of it extremely distasteful: the addition of assertion methods to the base objects themselves. I concede that it has a slight edge on relaying intent, but the negative consequences far outweigh the minimal gains in readability. Extending the builtin prototypes in application code is a dubious practice at best and considered an abuse by many, but for a testing framework it is indefensible. It is simply beyond its charter to bemerd the core objects with its own functions. Period. It's hard enough as it is to get different javascript libraries to live together peacefully in the ring without your testing framework leaping in and swinging wildly.
Let's be real here.
'one'.shouldEqual('two')
assertEquals('one', 'two')
Those two ways of saying the same thing just aren't different enough to warrant the potential nightmares for framework users. Of course, it could be argued that seeing the assertions written in that style was key to reorienting my view of the tests from how they're constructed to what they mean, but if that's your vantage point from the get-go, I don't see those type of assertions as a major requirement.