You have just spent months designing, building, and testing your application. System test is complete and your testing team has declared you are bug free. You deliver the product to your client and set them loose on User Acceptance Test, confident it will go very smoothly leading to a product that is delivered on-time, within budget, with a high degree of quality. You and the testing team declare the Quality Assurance phase a success. Then the call comes. Your client is irate, claiming the product is filled with errors and not at all working as the requirements intended. After the call, you realize production could be delayed weeks, maybe months, costing thousands of dollars. Worst of all, your client now seriously doubts your ability to handle Quality Assurance on any future project. You scratch your head and ask, “Where did all these bugs come from?”

You begin the investigation into what went wrong and start with the Unit Test lead, who shrugs his shoulders and says “All of our code compiled without errors. We were error free.” What about the unit test scripts you ask and are told there are none. The lead says unit testing does not require scripts, if each unit of code compiles, the unit testing phase is complete. Of course, without scripts, there are no errors! Compiled code simply means there are no syntax errors, not that the logic is correct!

You move onto the System Test lead who shows you the entire suite of test scripts, each showing 100% pass rate on the third phase of testing. You ask who wrote the scripts and how they were written and learn from the testers they wrote them based off of the successfully compiled code delivered from the unit testers. Of course, the scripts passed…rather than being written from documented requirements and designs they were written off what assumed to be error free code!

These examples may seem extreme, but I cannot tell you how many projects I have been on where I have seen these exact scenarios occur. It happens for a multitude of reasons. Unit testers may feel they do not need to write scripts because they are writing the code directly from the requirements or technical designs, so if it compiles, they must have it right. Your system testers may not be experienced or confident enough to write their scripts directly from documentation, so they take “sneak peaks” into the application “just to make sure they have it right”, often leading to several errors being written into the scripts. Most commonly these scenarios result from a lack of documented testing methodology to guide projects through the Quality Assurance phase.

Your Testing Methodology should include your overall test plan outlining each testing phase required for the project; a detailed test plan for each testing phase; templates and instructions for creating test cases, conditions, and scripts; a documented process for capturing and reviewing bugs, and standard metrics to actively track how the testing phases are progressing. Every project is unique, and the Testing Methodology will help you determine the testing phases that will ensure a successful Quality Assurance phase. Although each project is unique there are certain testing phases and certain guiding principles that should be followed within each project.

• Unit Testing - Unit testing is much more than simply a successful compiling of code. The objective of Unit testing is to validate that a module’s code reflects the appropriate detailed logic set forth in the design specification, test every unique path through the module’s logic, and encompass all logical branches, limits, etc. Every condition of every If Else, Case, or Do While statement should be tested and not just the first branch of that statement.
• System/Integration Testing - System test scripts should be written directly from your design documentation without any assistance from your application. Ideally, they should be completed before development of your code is even complete. System/Integration Testing ensures that all components that have been successfully unit tested function properly when assembled into process groups and verifies that the interaction of related components function properly when integrated.
• UAT Testing - If all other phases of your testing have been done correctly, User Acceptance testing should be quick and painless. These scripts should be written from the high-level requirements documentation. The ultimate end users of the application should write the scripts and perform the testing as they ultimately know how they want the system to work.
• Errors - There is often an inherent fear within testing teams that errors found in testing phases are bad and a sign that the product is of low quality. This often leads to testers masking errors, not logging them, or having developers log them without confirming if it was an actual error. In actuality, errors are extremely useful in improving the overall quality of the project. Errors come from several sources, code, test scripts, designs, and even requirements. Using a tool to log and classify all defects and discussing errors with all appropriate stakeholders will lead to the most appropriate resolution to errors. That can be a code fix, a change in design, a clarification in requirements, or an enhancement. Documenting and openly discussing errors will ultimately provide a higher quality end product.

The number one thing your clients expect from you is quality. If your client was smart enough to budget for appropriate testing resources, be smart enough to execute it correctly. Without the appropriate rigor and focus, you will miss the mark every time.