Our testing model in based V model. It is extension of traditional water fall model.
Left side of model represent software development life cycle and right side means testing model.
In left side, deliverables are static things such as document or source code.
To verify these static deliverables, it has be done by human.
Verification means “review & inspect artifacts by human”. For example pair-review, code inspection etc.
From right side, we have a actual software system that really works. In this stage we validate real software with automated testing tools.
The test is concentrated on this validation process.
Validation phase is separated into four steps.
Level | Description | Testing Type | Responsibility |
|
Unit Test | Verify software component | White Box | Dev Team |
|
Integration Test | Verify integration between components. Verify software flow, interfaces and interactions. | White Box | Dev Team |
|
System Test | Test system over production environment to verify system itself and production environment. Include Functional & Non functional Test. ( Availability, Stability, Extendibility, Performance etc). | Black box | QA Team |
|
Acceptance Test | Verify customer’s requirement, legal issue, maintenance issue etc. | Black box | QA Team or Customer |
|
Each of test step has a test cycle.
Ø Pre Test
Pre Test phase is preparation phase of testing.
In thin phase prepare target testing system, test scenario and mock up system for testing etc.
Ø Main Test
Execute test. Failed test is logged in test case management system.
Ø Conformation Test
The test that is failed during main test and then fixed later is tested in this phase.
Ø Regression Test
Regression test is for check up the impact from change. This test is done every time when it has a new change.
To execute regression test, all of test case should be automated. Every time execute regression test without automation is a lot of works.
This section describes Unit Test and Integration Test. As I mentioned earlier this test is done by development team by themselves.
1.1.1. Development and Testing Environment
In the test environment perspective Unit test and integration test are almost same. Only difference is scope. Unit test concentrate on individual component itself. Integration test’s scope is functionalities that are made by composition of individual component.
1.1.2. Unit Test
The goal of acceptance test is get
Unit test is testing for verifying software component works well. In J2EE world software component is Java class,EJB component etc.
This test is done by developer itself.
Every time developer develops software component. It is tested by developer itself before committing the code into SCM.
There are many types of software component, so there are many type of unit test framework as well.
Here is functional unit test framework.
Ø Basic Testing framework
Basic Testing framework. It is broadly used to test software component.
Junit is used for basic testing framework.
Ø In container Testing framework
One of extension of basic testing framework. J2EE application needs a application server like weblogic etc. When we tests software component we need a application server as well.
For example, getting session information or getting servlet context can not be run our of application server.
In container testing framework helps test case to run on container.
Ø DB Testing framework
Extension of basic testing framework. It is specialized for Data testing.
For example when we try to test DAO (Data Access Object) ,
1. we have to initiate DBMS
2. Run a logic
3. Compare result with expected data
4. Roll back the database to previous state
To simplify this process we use DB Testing framework. ,
1. Dump data from existing data base or write a data to external file such as excel or xml
2. Initiate DBMS with the file
3. Execute test case.
4. Compare data in DBMS with data (XML file,Excel or file dumped from DBMS)
5. Recover data with the file.
Ø UI Testing framework
Web application test needs a UI test.
For example querying profile application test scenario is
1. Input a name to text box
2. Press submit button
3. Check the result that has a expected name and phone number etc.
Automating UI Testing is very hard to framework. UI Testing framework execute these kinds of web navigation process automatically.
In addition some of component should be tested with performance. To support this we have Performance unit test framework
Ø Performance unit test framework
Most important factors of transformation or document parsing application is performance. Some component’s main factor is performance. To validate this performance testing framework can give a stress(heavy load) to component and can check the response time etc.
There are many issue to generate load for component like threading issue etc.
Performance unit test framework helps developer to execute this performance unit test without changing or making a new code for performance test. This framework just reuse existing unit test case and just add a load.
Even if developer tested software component, we cannot know what components are tested or not. This is testing for test. There are many methodology to validate test itself. But “Test coverage” strategy is mainly used.
Test coverage is validated how much test target is tested during the test.
For example how many source code lines are tested. How may requirements are tested. How may classes are tested etc.
In Unit test, we uses “Code coverage”. Code coverage means ‘How many source codes are tested in unit test”
By using this result we can estimate confidence of unit test.
Ø Code coverage analysis framework
Mechanism of code coverage tool is incrementing software code so gathering coverage data in runtime.
There is two kinds of incrementing mechanism.
1. Static instrumentation
Static instrumentation works in compile time. When compiling source code , coverage tool insert their code to compiled binary. In this case after deployer have to manage two kinds of achieve file. One is for test coverage, the other is for runtime.
2. Dynamic instrumentation
Dynamic instrument works in runtime. It hooks JMV itself so it put instrument code when class is loaded into memory. It doesn’t need to make different version for runtime and coverage.
But it has a runtime overheard for incrementing
In addition Enterprise application is composed of huge lines of code, so it is very hard to achieve 100% code coverage.
So we will employ risk-based test strategy, which was introduced by Eric Van.
We define Risk as follows:
Risk = likelihood * impact
Likelihood can have factors like “Need high skills to implement”, “ Complex functionalities” , “Not validated framework” etc.
Impact can also have factors like “Can lose revenue”, “Lost a user royalty” etc. It is mainly related to business.
Then, make a table for each test item with both likelihood factor and impact factor. A test item could be a class, module or package.
| Likelihood | Impact |
factor | factor | factor | factor |
Test item |
|
|
|
|
Test item |
|
|
|
|
Figure 1 Risk Analysis Table
Calculate risk value with likelihood and impact. Then, we can rank a test item as below:
Figure 2 Risk Analysis Matrix
STA area has a high probability of occurrence with big biz impact. This should be tested.
ITA area also has a high probability of occurrence yet with small biz impact. In general, ITA area is related to technical issues so that it should be handled in development phase.
SSTA area has a low probability of occurrence with big biz impact. Therefore, preferably it should be handled in system testing and acceptance testing.
By using this technique, we can verify the items that should be tested in development phase (Unit Test and Integration Test). And the test items should meet a coverage rate of around 80~90%.
After we define test items, they should be managed in terms of coverage. In this regard, we need to measure code coverage.
Ø Mock Test Framework
Mock Test framework is used for making a mock up object of component.
Unit testing tests component in isolation. But most units do not work alone. To test unit in isolation. We have to simulate the collaborators in the test.
Mock Test framework creates mock up object based on component interface (eg. Java interface). Record a action and replay a action during running unit test.
If unit test tests software runtime. Static test tests source code in compile time.
Code inspection by human is very effective but not efficient. There a huge line of code and there are many change as well. So it is unable to inspect all of code by human. Static test tool helps this.
Ø Static Test
Based on pre-defined defect pattern, the tool validate the source code and find a defect. Even if the tool cannot find all of the defect. A lot of risky defect can be found by this tool. It is very efficient.
There is many static test technique. Defect pattern matching, naming convention check, code complexity analysis (by using cyclomatic complexity analysis.). Feature is depending on Static Test tool.
Here is composition of recommended unit testing framework product that I mentioned above.
1.1.3. Integration Test
Integration Testing tests interface between components, interactions with different parts of a system. For example Operating system, JVM, Middle ware, inter component etc.
In our process there is no additional integration test step. Integration is occurred every day by automated build and regression test of unit test.
Compare to unit test. Unit testing test fine grained test. Integration testing tests functionalities. But in test case perspective, functionalities testing is just big component testing followed by function.
Integration Testing is done in staging system. The system has very similar environment compare to production system. Same middle ware, same os, same jvm etc.
During the integration testing, we can find a defect from integrating component.
System Test and Acceptance test is executed by QA Team.
Here is solution architecture of System & Acceptance Test environment
1.2.1. System Test
System testing tests whole system as defined by the scope of development.
Testing environment of system test are same as production environment as much as possible to minimize the risk of environment specific defect.
System test scenario is based on risk on requirement spec, business impact , technical hardness.
In system test we execute below four testing.
Ø Functional Testing
Functional Testing tests system functionalities only over production environment.
It validate functionalities from system requirement. Even if functional testing is done in integration test by development team, this system test is done by QA team with different view.
Ø Performance Testing
Performance testing test system performance such as throuput, response time etc.
To execute performance test, test scenario is very important. For example if we have 50 performance test scenario. The load for each scenarios are different. This is from “work load model”
The model is prediction that is based on previous service log or estimated usage pattern.
Ø Availability Testing
Availability testing tests fail over (Clustering failover or HA) , recovery thing etc.
Under heavy load, down the system suddenly. Validate fail over functionalities.
Restart the server again and check the recovery and transaction consistency etc.
Ø Scalability Testing
Scalability has two types.
Vertical scalability is about resources like CPU, Memory etc. By adding hardware resources in a box. Horizontal scalability tests scalability by adding new instance(process).
This test result is used for basis of capacity planning.
1.2.2. Acceptance Test
The goal of acceptance test is get a confidence of system. This testing is done by customer or end user of system. Acceptance test contains validation of deliverable , legal issue and company policy as well.
Acceptance test has
Ø User Acceptance Test (UAT)
Test system functionalities based on requirement. This test can be regression test with automated UAT tool
Ø Maintenance Test
This test is done by customer’s maintenance team. They check up deliverables for OAM and tests a OAM functionalities of the system.
Ø Contract and regulation Test
Validate legal issue and company policy.
Ø Beta Test
Beta Testing is done by system end user.
1.3. Test Case Management
Followed by V-Model, there are many test level and many test cases.
This test cases are very hard to manage. QA needs a systemized tools for handling these test cases.
Functionalities of Test case management are
Ø Manage Test case
Create and manage test case (not actual program, logical scenario, plan etc)
Categorize test case and register test case runner.
Priories test case based on priority and critical level.
Ø Run Test case
Run a test thro test case management tool.
Ø Record Test result
Record a test result in system and report it.
Ø Manage relationship with requirement and defect.
Test case is from requirement. Requirement management system manage requirement. Or we can use excel sheet either. Test case management system provides traceability from requirement to test case.
During a testing if new defect has been found, the defect is registered in defect management system. Test case management system also handles this relationship.