Monthly Archives: December 2012

Using Pex as whitebox testing tool

Using PEX as white box testing tools

Problems

Unit testing is nothing new, it has been there from the decades and it has significant history regarding the notion of unit testing. Various software development process were developed like TDD (Test Driven Development), BDD (Behavior Driven Development) around the unit testing. Unit test has many pro and cons and it is beyond the scope of this article. But, basic understanding of Unit testing is required for making remain article to make sense.

There, I am going to discuss about the problem maintaining the Unit test case. When project is in the developmental stage, zest among the development team are high, they read specs, create test cases and write code in the TDD notions. All, good! Team produce thousands of unit test cases. These piles up in Unit test suite silos. Team has zeal to set up continuous integration were this piles of test cases are executed on checked-in or fortnightly, and test report are send via automated email to relevant stockholders in the team.

Till now, all good! Now, as the dead line approaches, tension run high in the team, there are lots of fixes in code in the test cycles, but developer start to cut corner and fails to update test case for code changes he has been making. Many unit test starts to fail, code coverage goes down and so does the code quality. Every new line developer is writing in gust, has potential to introduce new bug in the system. These bugs are related to boundary conditions, unhandled exception and architecture violation to pattern.

At the end of delivery, unit test suites contain many fail tests, reason remain best unknown, code which was shiny in the beginning of project, now is dilapidated, in disrepair and breaking at various places now and then, when accost with un predicted condition.

As the result of this, maintenance support increases, integration havocs are created and hereafter cost shoot ups, which in turn put frown on the client face.

Now, we understand the problem, there is quest of automation tool, which would find hidden bugs in the code, sneaks into each and every line of code, traverses to branches and unfold the areas where code could break, and act as savior in midst of code mayhem which last minute pressure and cut corner has created.

Basically, in technical language, we are looking for white box testing tool, which reads out lines of code and tell us where we have gone wrong. It also addresses the staling test suites issues, where newly written line of code is automatically explored and check for bugs in them without writing additional test case to cover it.

PEX

To save the day, let me introduce PEX. PEX is White box testing tool which is still in beta (at the time, I was writing this article) and was developed by Microsoft Research and development team in Redmond, CA.

I am not gone to write a lot about introduction of PEX, as lot has been written about it. You can find more information at http://research.microsoft.com/en-us/projects/pex/ . I am going to focus of the methodology using PEX unit test cases to solve the problem of we describe above.

Goal

  1. To explore our code looking for boundary conditions, crash test and unhandled exception in autonomous fashion without much effort need in writing test case.
  2. Keep unit test suites for getting stale over the period of time, as new lines of code are been written.

Solution

To demonstrate the PEX, I have create the over simplified project as show below. This project with has 3 layers, (Entity, DAL and Insurer). Now, layer architecture is as given below,

Insure and DAL both refer entity, and Insure refer Entity.

Here, we are going to test two methods in the Insurer class.

First one is IsPersonEligible, which validates the Person is eligible or not for insurance base on the business rules as defined in code

public
virtual
bool IsPersonEligible(Person p)

{


Contract.Requires(p != null);


List<string> SelectedAreasZipCodes = new
List<string> { “400001”, “400002” };


if (p.Age > 17 && p.Age < 45 & SelectedAreasZipCodes.Contains(p.pAddress.ZipCode))

{

dal.saveInsurePerson(p);


return
true;

}


else

{


return
false;

}

}

DAL is injected into Insurer class via constructor using IOC principles.


public
class
Insurer

{


public Insurer(DAL.IDAL dal)

{


this.dal = dal;

}

}

Now, let concentrate on testing of first method

public
virtual
bool IsPersonEligible(Person p)

In order to write manual unit test, you need to stub person object to particular state. More, information of stub, mock refers http://martinfowler.com/articles/mocksArentStubs.html.

Three test cases have to be written in order to cover 100% for this test method.

Unit Test one:
Pass person as null and expect the contract null exception

Unit Test two:
Stub person and set person object to some state, which doesn’t fulfill business rules and expect method to return false

Unit Test Three:
Stub person and set person object to some state, which fulfill business rules and expect method to return true

Now, when developer writes all three test cases, it assumes code coverage of this method is 100% and code is fit to be used. But, do you think, these test case are sufficient? Does it cover boundary condition, unhandled exception and hidden bugs?

Ten-ta-ten, there come PEX to rescue. For quick run, just right click anywhere inside the method body, and select “Run Pex”

And see the magic happen. Pex will explorer your method and find tries to find out all the bug arises due to boundary condition, unhandled exception and hidden bugs.

It will on its own, will explorer the code under test, create the stubs on person is such manner, that it traverse to each branch in code in order to get 100% coverage of your method .

Above is the result, when you run PEX on the code.

But, life is never so easy, we do not write simple objects, our systems are full of complex objects and interaction between them. When you unit test code which involves creation of complex box, PEX will initially try hard to figure itself about the object creation, but our modern day object like controller which deals with httpcontext, viewbag, and various other dependency, PEX fail to create such object.

In that case, you need to create Pex Factories methods which will help PEX in able to create object in particular state, which can be used to test your subject under test. Here, I have underline particular state, because, when you create factory method, you instantiate the object in particular state, which is needed by the code under state. Example, when you instantiate controller, controller hold httpsession state, and method you are testing, expects, certain value to be store in session bag before this method is called. Hence, in factory, you add the particular value in the session when setting httpcontext to controller.

Also, it’s not always possible to set state in the factory. Consider this scenario, that you are sharing the factory across many methods. You will set state which is generic to all methods, but specific once, you need to set in special unit test method which PEX will use to explorer. These special methods are called Pex Methods.

Enough of abstract talking, lets point back to example.

public
virtual
bool IsPersonEligible(Person p)

Now, order to test this method, we need to first instantiate Insurer class, which contain this method. So, what the problem? If you check the constructor of insurer, there is just one public constructor which take IDAL (Data access layer dependency) as parameter.

PEX is in problem, because PEX doesn’t know, how to mock and pass IDAL object to Insurer class. PEX factory methods came to rescue!

We need to create factory method for Insurer class, which PEX will use to instantiate Insurer class, whenever it needs to initiate it.

///
<summary>A factory for SampleApp.Insurer instances</summary>

[PexFactoryMethod(typeof(Insurer))]


public
static
Insurer Create()

{

DAL.Moles.SIDAL Mdal = new DAL.Moles.SIDAL();


Insurer insurer = new
Insurer(Mdal);


return insurer;

}

Here, in factory, I have use Moles for mocking IDAL and pass it Insurer object. I won’t discuss Moles there, as it will require separate set of post to describe it. But, you can find more about Moles at http://research.microsoft.com/en-us/projects/moles/

Also, we would create PEX method for IsPersonEligible. Just, click under the method and select create parameterized Unit test, follow the wizard, your favorite test framework, than ‘pufs!’, it done. Pex has create boiler plate PUT (Parameterize Unit test) for you and you could either use as it is or customize to suit you need for testing.

///
<summary>Test stub for IsPersonEligible(Person)</summary>

[PexMethod]


public
bool IsPersonEligible([PexAssumeUnderTest]Insurer target, Person p)

{


bool result = target.IsPersonEligible(p);


return result;


// TODO: add assertions to method InsurerPexTest.IsPersonEligible(Insurer, Person)

}

To explore your method, just right click inside PexMethod and click explore.

Up till now, I have covered two concepts, Pex Factories and Pex Method. Pex Method which nothing but PUT. Now, let see how, these two concept help to solve our problem.

Once you written you Pex factories, and Pex Method, you can setup Pex to run in continuous integrated environment. I.e. whenever a developer check in the code or you can set it fortnightly if team is big (10+ developers), Pex would run its exploration using Pex Factories and Pex Method and emailed you the report with exceptions if any of your code when ka-put! Aai-la!

To see the point here, no new test case were written, no effort taken for code review, still boundary condition, unhandled exception, crash values are detected. Hence, it improves the bug detection induce in code, and fixing those bug improves code quality.

I want you to see my point in action, consider method under test below

public
virtual
bool IsPersonEligible(Person p)

{


Contract.Requires(p != null);


List<string> SelectedAreasZipCodes = new
List<string> { “400001”, “400002” };


if (p.Age > 17 && p.Age < 45 & SelectedAreasZipCodes.Contains(p.pAddress.ZipCode))

{


int zipcode = Convert.ToInt32(p.OtherAddresses[0].ZipCode);

dal.associateZipCodewithPersonIndex(p);

dal.saveInsurePerson(p);


return
true;

}


else

{


return
false;

}

}


Yellow highlighted code is added by zombie developer, which will break the elegant. What if there no address in person other addresses collection, index 0 will throw Index out of bound exception, what if zipcode in otherAddress is null or not numeric, this will throw null exception or parse exception. What if zipcode is larger than int max value, this will throw over flow exception. Developer didn’t think about the boundary conditions and exception handling, and our PEX will find out that.

This exceptions maybe be caught with the manual written test case too, but, in manual test case, you need to manually create person stub. If stub contain otheraddress, and has well behave value set, boundary condition exception won’t be caught.

In PEX exploration, PEX generates the person stub and hence, will uncover hidden exception.

In order to improve code, developer needs to put check for the boundary conditions and implement proper exception handling.

In war, knight needs helpers in form of soldiers. This are parameterize mocks and partial stubs. To understand this, let take another method example

Second method in Insurer class is GetInsureAmount, which retrieves the insured amount of the person from DAL repository after checking if person is eligible as per business rules.

Here, DAL is data access repository which is not implemented as it beyond the scope of the unit testing for the Insurer class.

public
float GetInsureAmount(Person p)

{


float rtnAmount = 0.0f;


if (this.IsPersonEligible(p))

{


Policy ply = dal.getInsursePersonPolicy(p);


if (ply.InsureAmount > 0)

rtnAmount = ply.InsureAmount;


else

rtnAmount = 0.1f;

}


else


throw
new
NotEligibleException(“Person “ + p.Name + “not eligible for insurance”);


return rtnAmount;

}

If we study the method, this method has two nested branching. In order to pass the first branch (if-else) for positive flow, we need stub person object in such state, that, it pass the condition. IsPersonEligible would return true with that stub.

In order to pass through the second branch, we need to return policy from dal.getInsursePersonPolicy such that it has InsureAmount greater than zero. Wait, do you see the problem here, dal is mock layer which we are injecting Insurer constructor in Pex factory for insurer. Mock layer doesn’t have body and then, will not return any policy when explore with Pex.

Solution? We need to fake the call dal.getInsursePersonPolicy such way, that it returns policy with InsureAmount greater than zero.

How do we do it? In Pex Factory for Insurer,

[PexFactoryMethod(typeof(Insurer))]


public
static
Insurer Create()

{

DAL.Moles.SIDAL Mdal = new DAL.Moles.SIDAL();


Mdal.GetInsursePersonPolicyPerson = ((p) =>

{


Policy ply = new
Policy();

ply.PolicyNumber = 123;

ply.InsureAmount = 100;


return ply;

});


Insurer insurer = new
Insurer(Mdal);


return insurer;

}

Yellow highlighted code represents the faking of call to dal.getInsursePersonPolicy leveraging Mole framework delegate capabilities.

Just run the Pex exploration with this method, and see the outcome. What, Pex was not able to cover 100% of code in method under test. Why? Because, we stub Policy which return InsureAmount as 100 always, and hence, Pex was never able to explorer else condition on second branch.

Solution? We need to tell Pex somehow, that one time put InsureAmount as 0 for one test case and InsureAmount as 100 for another test case. This, way, we would be able to cover both the condition of second branch. But how?

Using PexChoose. PexChoose is the static method in the Microsoft.Pex.Framework which set the value which is required in the symbolic execution of the Pex Exploration. I won’t do in deep details on how Pex explore method, use constraint solver Z3 engine, and how PexChoose will take part in Constraint solving equation generate by PEX, you can get all these details at http://research.microsoft.com/en-us/projects/pex/

But in lay man language, it tell PEX to use 0 for one test case, and any value greater than 0 for another test it execute to cover method branch 100%.

There is another advantage of PexChoose. If tomorrow, business rule changes and there is new condition is as shown below


if (ply.InsureAmount > 10)

rtnAmount = ply.InsureAmount;


else

rtnAmount = 0.1f;

We don’t need to change your Pex Factories and Pex Method, pex exploration will figure out the new condition and will generate new Policy stubs base on PexChoose and make sure that it would cover both the branches of code.

The new Pex Factory method, with Parameterize Mock is show below

[PexFactoryMethod(typeof(Insurer))]


public
static
Insurer Create()

{

DAL.Moles.SIDAL Mdal = new DAL.Moles.SIDAL();

Mdal.GetInsursePersonPolicyPerson = ((p) =>

{


Policy ply = new
Policy();

ply.PolicyNumber = PexChoose.Value<int>(“PolicyNumber”);

ply.InsureAmount = PexChoose.Value<float>(“InsureAmount”);


PexObserve.Value<Policy>(“PolicyReturn”, ply);


return ply;

});


Insurer insurer = new
Insurer(Mdal);


return insurer;

}

Here, you have learn the new concept Parameterize Mock and seen how it helps in the Pex Exploration. Now, I have cover Pex Factories, Pex Method and Parameterize Mock, there remain one concept which I need to touch is partial stub.

Now, let re-visit the second method GetInsureAmount.

public
float GetInsureAmount(Person p)

{


float rtnAmount = 0.0f;


if (this.IsPersonEligible(p))

{


Policy ply = dal.getInsursePersonPolicy(p);


if (ply.InsureAmount > 0)

rtnAmount = ply.InsureAmount;


else

rtnAmount = 0.1f;

}


else


throw
new
NotEligibleException(“Person “ + p.Name + “not eligible for insurance”);


return rtnAmount;

}

Here, you have seen IsPersonEligible is test again with when running test on GetInsureAmount. Thru, it’s ok and simple in our example, consider a other a scenario where IsPersonEligible
is the complex method which in turn calling another third party dll. Under the context of unit testing of GetInsureAmount, we don’t want to test IsPersonEligible. Remember, we are doing Unit test, and not integration testing. IsPersonEligible testing would be done when doing unit testing on that method.

Here, there a problem, IsPersonEligible is public method of the same class Insurer, and I need some notion where I could fake IsPersonEligible call in test case, but not GetInsureAmount call.

Once solution would be decoupled the interaction, where I put IsPersonEligible is other class, and do dependency injection in the Insurer class, where I could fake call to IsPersonEligible. But this is not always possible. There are cases, when you can’t change class blueprints.

Here comes the concept of Partial Stubs, where I would mock the Insurer class in such a way, that I could fake IsPersonEligible call, but, let the GetInsureAmount go to real object.

public
static
Insurer Create01()

{

DAL.Moles.SIDAL Mdal = new DAL.Moles.SIDAL();

Mdal.GetInsursePersonPolicyPerson = ((p) =>


PexChoose.Value<Policy>(“Policy”));


SInsurer insurer = new
SInsurer(Mdal) { CallBase = true };

insurer.IsPersonEligiblePerson = ((p) => PexChoose.Value<bool>(“IsEligible”));


return insurer;

}

Yellow highlighted in above code stubs the Insurer class, where CallBase is set to true. I.e. whenever, any call to any of the method is called on the SInsurer stub, it will redirect its call to actual implementation of Insurer Class.

Grey color line is where, I have override the IsPersonEligible method, and set to parameterize value using PexChoose which would either return true or false base on the symbolic constraint condition needed by Pex.

Now, let run the Pex exploration with new partial stub, and see the result

Pex has found the bug in our code,

Policy ply = dal.getInsursePersonPolicy(p);


if (ply.InsureAmount > 0)

Following above lines, I do not check if the ply return from dal.getInsursePersonPolicy is null or not and directly access ply. InsureAmount property, then NullReferenceException is thrown!

Summary

Wow, I am reaching the end on the article and here the summary

  • Pex is great candidate for white box testing tool, which help to find boundary conditions, crash test and unhandled exception in our code
  • In order to use Pex on complex object and interaction, we need to use Pex Factories and Pex Method
  • In order to improve Pex Factories or Pex Method to take part in symbolic execution of the Pex, we can use Parameterize Mock in them
  • If there is hard dependency between the method on the same class, and we need to unit test it. We can leverage the power of Moles mocking framework from same Microsoft Research team and could use Partial Stubs

Where do we go now?

References