Friday, November 30, 2007

JSFUnit Beta 1 is Released !!!

As I write this I look at the title and wonder how many exclamation points I should have used. What I thought would be a tiny little framework has grown into a full-fledged project with a real beta release. We even have live demos.

JBoss JSFUnit is the first open source community dedicated to testing JSF applications. With this release, it is not one, but three different tools:
  • In-container Testing framework This is for testing both client and server side JSF artifacts. Test scope ranges from fine-grained tests of individual classes to full-fledged integration testing of a running JSF application. This is done with no mock objects, so you know you are getting reliable real-world behavior in your tests.
  • Framework for JSF Static Analysis testing This allows you to test your JSF configuration so that you find config problems early.
  • JSFTimer for performance testing of the JSF lifecycle This shows you how long each phase of the JSF lifecycle took for a given request. Of course, it also gives you total JSF lifecycle time. The JSFTimer allows you to write threshold tests that fail whenever a use case doesn't meet your performance standards.
If you want more info, read the other posts in this blog and then take a look at

But now a few more words about the release itself. First, it's called Beta 1. To me, that implies a few things. The first implication is that it's useful. Even before a downloadable release was ready, many developers built the project from subversion and started using it. So I know it meets my main criteria for a beta, which is that it fills a need. It's useful, so by all means, feel free to use it.

The second thing about being a beta release is that it's not done. That is, we know some features are missing. Mainly, JSFUnit needs support for more Ajax components and it needs more deployment tools.

The last thing I want to say about the release is "Thank You". First, thanks goes to Dennis Byrne of ThoughtWorks and Matt Wringe of JBoss. These guys contributed code to the project and continue to be active. We need more help as demand for bug fixes and new features has ballooned even before we ever released. So if you want to help out just send me an email or post on this blog and let me know how you'd like to get involved.

Second, I want to thank Julien Viet, JBoss Portal lead, and Patrick McDonald, my manager at JBoss/Red Hat. These guys have given the high level support that made it possible for me to work on the project in the first place.

Lastly, thanks to all who have been the early, early, early adopters. Your feedback on the JSFUnit forums and through email have been invaluable.

So try it out. Let me know what you think.

So long, and thanks for all the fish,

Stan Silvert

Monday, November 5, 2007

JSFUnit and Grey-box testing

An anonymous viewer posted a question about the nature of JSFUnit. I was going to reply directly to the comment, but I felt that the answer needed its own top-level post since it is so fundamental to the nature of JSFUnit.

Anonymous asks,
Why do you want to implement another testing framework? Why do you not consider WebTest ?
The answer is that frameworks like WebTest, HttpUnit, HTMLUnit, etc. only test the client side. These are pure black-box frameworks for web apps.

With JSFUnit, you get both client side and server side testing in one test. Some call this grey-box testing.

Plus, the JSFUnit API is written specifically for ease of testing everything you would want to know about a JSF application.

For instance, with JSFUnit I can not only submit data on a form, I can also make sure that managed beans were properly updated (using EL expressions).

I can make sure that my component tree on the server side is correct using the actual server-side objects created by JSF. Checking the server-side component tree has three distinct advantages that you don't get with just checking the HTML:
  1. It's much easier as I can make simple JSF API calls instead of checking raw text or the contents of the client-side DOM.
  2. I don't have to rewrite my test when the HTML changes.
  3. I can validate components where rendered=false. In this case, the component is not rendered on the client side, but still has state on the server side.
With access to the FacesContext, I can test navigation from the server side with a simple check of the viewID. If you are only testing the client side, you have to infer the viewID and that inference may become wrong as the application evolves. Again, the test breaks simply because you changed the HTML, not because you changed the application logic.

I could go on and on about server side testing. There is a lot more to be validated such as database state, security state, and even memory usage.

If I want to do the client side (black box) testing, JSFUnit gives full access to the HttpUnit API and the client-side DOM. So that is covered as well. And again, I can even do both black box and white box testing in the same test if I want.

Lastly, JSFUnit includes static analysis tests that do validation of your configuration beyond what JSF itself provides. This lets you know about problems specific to JSF that you can catch long before they show up in some obscure use case at run-time.

Good question Anonymous. So long, and thanks for all the fish,


Tuesday, October 30, 2007

Introducing JSFUnit

A few months ago, Julien Viet, creator of JBoss Portal, sends me an email. He says, "Hey JSF guy, why are there no good testing tools for JSF? I need a tool for this. Maybe you can help?"

I took a look and he was (mostly) right. There are a couple of mock object frameworks for JSF as part of Shale and Seam but they are there primarily for testing artifacts you might create for those frameworks. They are really not general testing tools for JSF applications. And they don't help with integration testing at all.

So I started thinking up what a general JSF testing tool might look like. Here is what I came up with:
  • No mock objects!!! I really hate mocks. I hate writing mocks. I hate maintaining mocks. And I hate knowing that my mocks don't simulate the environment well enough. But most of all, I hate it when I have to add stupid init() methods to my application code - just for the mocks. And sooner or later I know my tests will fail because of the mocks! BTW, here is a saner discussion of Mock vs. In-Container testing.
  • Leverage the JSF API JSF provides tons of information about a running application. If I want to assert that my JSF application state is correct, I should be able to just ask JSF.
  • Use plain old JUnit If I want anybody else to use the tool, they shouldn't need to learn how to write tests all over again. JUnit has stood the test of time. It works well and everybody knows how to use it.
  • Allow navigation testing A big part of validating JSF is validating that a given user input will navigate to the proper page view.
  • Test a JSF App as it is really deployed and run I suspect that this is why nobody has built a full JSF testing framework yet. A real JSF request might start with an AJAX component that fires an HTTP request, then pass through servlet filters, hit the FacesServlet, the JSF lifecycle, a myriad of PhaseListeners, do magic Seam stuff at every turn, and then all get rendered with Facelets. How can you possibly test all that stuff together with a mock framework?
So it has to be an in-container thing like Apache Cactus. But Cactus would have me instantiate a FacesServlet and call it directly. That won't quite cut it. What I really need is something that runs in the browser to do real HTTP requests and some kind of JUnit test on the server that lets me do asserts when the request is done. Why not a JUnit test that does both at the same time in the same JVM? Then I can assert that the server side state is correct and the client state is correct - all in the same test.

JSFUnit is done
I just combine Cactus and HttpUnit and I'm done, right? It took six classes to tie it all together. Cactus lets me run JUnit tests in-container. I can make real HTTP calls with HttpUnit, then test the HTML that comes back. And with a few FacesContext tricks, I have access to all the real server-side JSF objects for navigation, managed beans, EL, the component tree, etc. So I can do real JSF requests, test the real HTML that comes back, and test the real state of the real server-side objects.

For real. All in one JUnit test.

Only six classes. All that and only six little classes! Julien will really like it.

Julien hates it
"Stan, this sucks." OK, he didn't exactly say that. At JBoss, they don't just come out and tell you your code sucks. They use a lot more cuss words and throw in an SMD or two.

He told me it needed a simpler API and he was right. I wrote some Facade classes and got a Hello World test down to about five or six lines of code:

JSFClientSession client = new JSFClientSession("/hello.jsf");
JSFServerSession server = new JSFServerSession(client);
client.setParameter("name", "Stan");
assertEquals("Stan", server.getManagedBeanValue("#{}"));
assertTrue(client.getWebResponse().getText().contains("Hello, Stan"));

JSFUnit is just getting started
So now I'm done? Actually, that was just the beginning. It turns out that AJAX components need special handling so I've added support for most RichFaces components. Matt Wringe of the JBoss Portal team added Ant integration. Then Dennis Byrne from ThoughtWorks and MyFaces came on board to work on JSF static analysis tools. This is getting big and broad real fast.

There is still a lot to be done. We need to finish the support for the last few RichFaces components and then move on to other popular component libraries. We need better Maven integration so that it is on par with the Ant tool (JSFUnit tests itself with JSFUnit and Maven. So you can run your tests from Maven, but it needs to be easier). Dennis has lots of good ideas for improving the JSF static analysis. Who knows, at some point maybe we'll even add mocks (maybe not).

JSFUnit Beta 1
No matter what, we are doing our first Beta release on November 30. Right now you have to download JSFUnit source and build it yourself with Maven. Even so, we've got a lot of folks out there who have done that and are already finding JSFUnit to be a useful tool. So to me, as long as it passes the usefulness test, it's time for Beta. Stay tuned for details.

In the mean time, check out

So long, and thanks for all the fish,

Stan Silvert