Packaging functional tests using TestPackage

TestPackage provides a clean way to package your functional tests and run them as a jar file with sharding and package level run options. All that we need to do is created a maven/gradle subproject for functional tests and have the testpackage’s dependency and manifest file entry to get an executable jar package.

Once the jar is created all we need to do is run ‘java -jar the-test-package.jar package.that.contain.tests’. This helps a lot in continuous delivery where the functional test artifact can be created at the build stage and promoted to run at the functional test stage without having to access the source code and rebuild later.

Safety nets in application development

Two major categories of safety nets in agile application development.

• Process side
• Engineering side

Process Side

The process side safety nets ensure that sufficient communication between the stakeholders happen on time and strive to get together all the stakeholders on to the same page. The emphasis is not letting information lost in translation when moving from one stakeholder to the other. Some of the process level safety nets used in agile development are

Acceptance Criteria Definition

Defining acceptance criteria for stories help to define the scope of a requirement in crisp steps. The acceptance criteria will be used as a basis for automated unit/integration/functional test cases during development.

Story Review

Story review is done with analysts or developers once it is written and added to the backlog for the sprint. This will provide additional perspective to the requirements and help identify gaps in the requirement. Since the size of a story is typically small, it will be easy to distribute the stories to multiple people for review.

Story Kickoff

Every developer when starting to work on a story will be debriefed by an analyst on the scope and boundaries of the story. This ensures that the ambiguities related to the story are cleared and in case any thing new identified could be handled immediately.

Dev Box Verification

When every story is considered to be complete, the developer gets the application developed verified by the analyst if it is according to the story and it matches the acceptance criteria.

Showcase/Demo

At the end of every iteration all the stakeholders come together to view the demo of the application developed so far. This provides a common forum for all the stakeholders to agree or disagree on what they see. This ensures that the conflict of interests are resolved immediately and also provides a view of progress in terms of how the application is coming up. It provides the right forum to feed data to make course corrections.

Retrospective

A time set aside every iteration for the team to retrospect on what happened and take necessary actions to mitigate risks and resolve issues. This helps to iron out inefficiencies in the team and also a break from regular work to concentrate on improving effectiveness of the team overall.

Engineering side

The engineering side safety net ensure program verification. The correctness of the application being developed is ensured through multiple levels of automated test cases derived from the requirements. The greater the level of the test coverage the greater is the confidence on the application being developed.

Code Quality

Having strict adherence to code quality standards helps to keep the code readable, which in turn helps to onboard a new person with ease. Following effective code development practices will tend to make the code as a document for itself eliminating the need for documentation. Code quality also could be improved by pair programming and peer reviews where the review comments are incorporated then and there instead of a formal log, track and fix cycle.

Unit Tests

Having unit tests as part of the development is mandatory as it deals with the smallest unit of code. Unit tests ensure that the software is developed as per specification and since it is automated and multiple times faster than manual testing, it is one of the most effective safety nets one can have during application development. Test driving the unit test also forces good design in the code and helps to refactor with high confidence. Since the coverage area of unit tests will be so high, making last minute changes to the code will be a breezy change.

Integration Tests

Integration tests complement the unit tests, it helps to catch integration issues as early as in the coding phase. These tests will be used along with the unit tests and together will cover large areas of the developed code making changes to the code any time a welcoming one.

Automated Functional Tests

Automating the functional UI tests provides ample time for testers to concentrate on exploring the least touched areas of the application than repeating the same use cases for verifying the sanity of the application.

Exploratory Testing

This has no solid test case or plan written upfront and is the last line of defence in the safety nets, it is completely based on keeping the end user in mind and use intuition and heuristics to try to break the system. Once an issue has been found during exploratory testing, a safety net has to be in place for the bug in the of unit, integration or automated functional test

Ratio of Unit, Integration, UI and Exploratory testing

Mike Cohn mentioned about test pyramid in his book “Succeeding with Agile”. It gives the ratio between the different tests in the form of a pyramid.

The safety nets are not self sufficient by themselves and will not be able to catch issues if we concentrate a lot on one area of the safety net as the law of averages comes into effect (The effort increases exponentially but the result increases linearly). Also not all engineering test scenarios need to be automated, the permutations of different scenarios will far exceed the sane number of automated tests written for them in a given amount of time; so a careful selection of scenarios is necessary for automation. This will help in spending the optimum amount of effort for each layer of automation.

Continuous Integration

Continuous integration is an integral part in keeping the safety nets functional. Every time a developer commits code the continuous integration server picks up the changes compiles, integrates, runs all the tests and provides the output in a report. This can act as show stopper if the safety net is broken.  The immediate feedback provided encourages developers to commit code frequently and keep it integrated always.

Having all the above safety net ensures that anything crosses the baseline of development is of very high quality and conforms to all the requirements. The lifestyle of development changes from reactive to proactive as there will be enough room left for identifying the improvement areas and fixing them while the repetitive engineering tasks are automated. This also reduces the business/user acceptance phase to be a complete demo of the product instead of one more testing round.