A test methodology for an effective regression testing

A test methodology for an effective regression testing  

1. What is regression testing?Regression testing is selective retesting of the system with an objective to ensure the bug fixes work and those bug fixes have not caused any un-intended effects in the system.

2. Types of regression testingThere are two types of regression testing that are proposed here even though it is not being practiced or popular. A “final regression testing” is being done to validate the gold master builds and “Regression testing” being done to validate the product & failed test cases between system test cycles.The final regression test cycle is conducted on an “unchanged build for a period of x days” or for a period, which was agreed as the “cook-time” for release. The product is continuously exercised for the complete duration of this cook-time. Some of the test cases are even repeated to find out whether there are failures in the final product that will reach the customer. All the bug fixes for the release should have been completed for the build used for the final regression test cycle. The final regression test cycle is more critical than any other type or phase of testing, as this is the only testing, which ensures “the same build of the product that was tested reaches the customer“. A normal regression testing can use the builds for a period that is exactly needed for the test cases to be executed. However unchanged build is highly recommended for each cycle of regression testing.  

3. How to select test cases for regression testing?It was found that some of the defects reported by customers in the past were due to last minute bug fixes creating side effects and hence selecting the test case for regression testing is really an art and not that easy.The selection of test cases for regression testing

1. Requires knowledge on the bug fixes and how it affect the system
2. Includes the area of frequent defects
3. Includes the area which has undergone many/recent code changes
4. Includes the area which is highly visible to the users
5. Includes the core features of the product which are mandatory requirements of the customer

Selection of test cases for regression testing depends more on the criticality of bug fixes than the criticality of the defect itself. A minor defect can result in major side effect and a bug fix for an Extreme defect can have no or a minor side effect. So the test engineer needs to balance these aspects for selecting the test cases for regression testing.When selecting the test cases we should not select more test cases, which are bound to fail and has no or less relevance to the bug fixes. You need to select more positive test cases than negative test cases for final regression test cycle as this may create some confusion and unexpected heat. It is also recommended that the regular test cycles before regression testing should have right mix of both positive and negative test cases. Negative test cases here I mean those test cases which are introduced newly with an intent to break the system.It is noticed that several companies have “constant test cases set” for regression testing and they are executed irrespective of the number and type of bug fixes. Sometimes this approach may not find all side effects in the system and in sometimes it may be observed that the effort spend on executing test cases for regression testing can be minimized if some analysis is done to find out what test cases are relevant and what are not.It is a good approach to plan and act for regression testing from the beginning of project before the test cycles. One of the ideas is to classify the test cases into various Priorities based on importance and customer usage. Here I am suggesting the test cases be classified into three categories; 

• Priority-0 – Sanity test cases which checks basic functionality and are run for pre-system acceptance and when product goes thru major change. These test cases deliver a very high project value to both engineering dept and to customers.

• Priority-1 – Uses the basic and normal setup and these test cases deliver high project value to both engineering and to customers.

• Priority-2 – These test cases deliver moderate project value. Executed part of ST cycle and selected for regression testing on need basis.

There could be several right approaches to regression testing which needs to be decided on “case to case” basis; 

• Case 1: If the criticality and impact of the bug fixes are LOW, then it is enough a test engineer selects a few test cases from TCDB and executes them. These test cases can fall under any Priority (0, 1 or 2).

• Case 2: If the criticality and the impact of the bug fixes are Medium, then we need to execute all Priority-0 and Priority-1 test cases. If bug fixes need additional test cases from Priority-2, then those test cases can also selected and used for regression testing. Selecting Priority-2 test cases in this case is desirable but not a must.

• Case 3: If the criticality and impact of the bug fixes are High, then we need to execute all Priority-0, Priority-1 and carefully selected Priority-2 test cases.

• Case 4: One can also go thru the complete log of changes happened (can be obtained from CM engineer) because of bug fixes and select the test cases to conduct regression testing. This is an elaborate process but can give very good results.

4. Resetting the test cases for regression testingIn a big product release involving several rounds of testing, it is very important to note down what test cases were executed with what build and related information. This is called test case result history. In many organizations not all types of testing and all test cases were repeated for each cycle. In such cases resetting the test cases become very critical for the success of regression testing. Resetting a test case is nothing but setting a flag called NOTRUN or EXECUTE AGAIN with zero base thinking.RESET of test case, are not expected to be done often. Resetting of the test cases needs to be done with following considerations;a.      When there is a major change in the productb.      When there is a change in the build procedure which affect the productc.      Large release cycle where some test cases were not executed for a long timed.      You are in the final regression test cycle with a few selected test casese.      Where there is a situation the expected results of the test cases could be quite different from previous cyclesWhen the above guidelines are not met, you may want to RERUN the test cases rather than resetting the results of the test cases. There are only few differences between RERUN and RESET states in test cases, either way the test cases are executed but in case of RESET one has to think zero base and expect different result than what was obtained in earlier cycles and therefore those test cases affect the completion rate of testing. In case of RERUN the management need not worry about completion rate as those test cases can be considered complete except for a formality check and are expected to give same results.To give you an example, if there is a change in Installation of a product, which does not affect the product functionality, then the change can be tested independently by rerunning some test cases and we don’t have to RESET the test cases. RESET is also decided based on how stable the functionalities are. If you are in Priority-1 and have reached a stage of comfort level on Priority-0 (say for example more than 95% pass rate) then you don’t RESET Priority-0 test cases unless there is a major change. This is true with Priority-1 test cases when you are in Priority-2 test phase.

 4.1 Pre-system test cycle phaseFor pre-system acceptance only Priority-0 test cases are used. For each build that is entering the system test, the build number is selected and all test cases in Priority-0 are reset to NOT RUN. The system test cycle starts only if all pre-system test cases (Priority-0) pass. Test manager or CCB, can decide exceptions if any. 

4.2 System test cycle – Priority-1 testing phase After pre-system acceptance is over, Priority-1 test cases are executed. Priority-1 testing can use multiple builds. In this phase the test cases are RESET only if the criticality and impact of the bug fixes and feature additions are high. A RESET procedure during this phase may affect all Priority-0 and Priority-1 test cases and these test cases are reset to NOTRUN in TCDB. 

4.3 System test cycle – Priority-2 testing phase Priority-2 testing starts after all test cases in Priority-1 are executed with an acceptable pass % as defined in test plan. In this phase several builds are used. In this phase the test cases are RESET only if the criticality and impact of the bug fixes and feature additions are very high. A RESET procedure during this phase may affect Priority-0, Priority-1 and Priority-2 test cases.  

 4.4 In what way regression testing is related to above three phases? Regression testing is normally done after Priority-2 testing or for the next release involving only few changes. Resetting test cases during the above phases are not called as regression testing as in my assumption regression comes into picture only after the product is stable. A testing for a release can be decided either by saying a regression testing is sufficient or we can do all phases of testing starting from Priority-0 to Priority-2.  A regression testing for a release can use test cases from all priorities (as mentioned before). A regression testing involving multiple priorities of test cases also needs the test cases executed in strict order i.e. Priority-0 test cases are executed first, Priority-1 next and Priority-2 test cases.  

4.5 Why we need to RESET the test cases? Regression testing uses good number of test cases, which would have been executed already and associated with some results and assumptions on the result. A RESET procedure makes them to NOTRUN so that it gives a clear picture about how much of testing is still remaining, and reflect the results of the regression testing on Zero base.  If test cases are not RESET, then the test engineers tend to report a completion rate and other results based on previous builds. This is because of the basic assumption that multiple builds can be used in each phase of the testing and a gut-feel that if something passed in the past builds, it will pass in future builds also. Regression testing doesn’t go with an assumption that “Future is an extension of the past”. 

 5. How to conclude the results of a regression testing? Regression testing uses only one build for testing (if not it is strongly recommended). It is expected that all 100% of those test cases pass using the same build. In situations where the pass % is not 100, the test manager can look at the previous results of the test case to conclude the expected result;a.      If the result of a particular test case was PASS using the previous builds and FAIL in the current build, then regression failed. We need to get a new build and start the testing from scratch after resetting the test cases.b.      If the result of a particular test case was a FAIL using the previous builds and a PASS in the current build, then it is easy to assume the bug fixes worked. c.      If the result of a particular test case was a FAIL using the previous builds and a FAIL in the current build and if there are no bug fixes for this particular test case, it may mean that the result of this test case shouldn’t be considered for the pass %. This may also mean that such test cases shouldn’t be selected for regression. d.      If the result of a particular test case is FAIL using the previous builds but works with a documented workaround and a.      if you are satisfied with the workaround then it should considered as PASS for both system test cycle and regression test cycleb.      If you are not satisfied with the workaround then it should be considered as FAIL for a system test cycle but can be considered as PASS for regression test cycle. 

6. Can we apply the regression test guidelines for patch/upgrade releases?. The regression guidelines are applicable for both cases wherea.      You are doing a major release of a product, executed all system test cycles and planning a regression test cycle for bug fixes.b.      You are doing a minor release of a product (CSPs, patches …etc) having only bug fixes, and you are planning for regression test cycles to take care of those bug fixes. There can be multiple cycles of regression testing that can be planned for each release, if bug fixes come in phases or to take care of some bug fixes not working with specific build. 

 7. How do I find out which test case to be executed for a particular defect fix? When failing a test case it is a good practice to enter the defect number(s) along so that you will know what test cases to be executed when a bug fix arrives. Please note that there can be multiple defects that can come out of a particular test case and a particular defect can affect more than one test case.    Even though, it is easy to do the mapping between test cases and defects using these mechanisms, the test cases that are to be executed for taking care of side effects of bug fixes, may remain as a manual process as this requires knowledge and several other perspectives discussed earlier in this doc.

Testing a WEB Applications

Testing Strategy for WEB Applications

    Business Requirements

     Web Application Testing

The usage of the World Wide Web for transaction processing is a recent phenomenon. To that extent web application testing is a relatively new area.

Although web application testing does have a lot in common with client/server applications, there are unique considerations that affect the focus of testing strategy.

    What to Test

All components of a Web application on both the client and server side should be completely tested. This may not be always possible given the varied target audience and platforms that a web applications addresses.

In this case, the best approach is to examine the project’s requirements, set priorities based on risk analysis, and then determine where to focus testing efforts within budget and schedule constraints.

   Requirements

Web applications are typically more graphics oriented than standard software applications, and this, along with multimedia functionality, will need to be considered in terms of requirements specifications for page layouts, appearance, site-specific standards etc. Database –driven applications will merit particular focus on performance requirements such as response times at various load levels and rate of error-free interactions per unit time.

   Risk Analysis

Database driven web applications can involve complex interactions among Web browsers, operating systems, plug-in applications, communications protocols, Web servers, databases, applications server, security enhancements, and firewalls. Such complexity makes it impossible to test every possible dependency and everything that could go wrong with this site. Since the typical web application development project also will be on an aggressive schedule, the best testing approach will employ risk analysis to determine where to focus testing efforts. Risk analysis should include considerations of how closely the test environment will match the real production environment. Will the test scenarios be able to closely mimic communications, hardware, clients, loads, data and so on? Other considerations in risk analysis include:·     

         Which functionality in the Web application is most critical to its purpose?·      

       Which is function is most likely to be the most frequently used or the sequence of steps to be most frequently performed?·      

        What type of areas could possible cause the most complaints or bad publicity?·     

         What parts of the application pose the maximum security risk?

  Types of Testing

The following are the different types of testing that are used for various Web applications. These aspects should be considered while planning for Unit, integration and system testing wherever applicable. Existing testing related checklists/templates in QSD could be enhanced to address these additional requirements. 

S.No	Type of Testing	Description
1.	Validation or Functional Testing	This is typically the core aspect of testing to determine that the Web application functions correctly as per the requirement specifications.
2.	Unit and Integration Testing	Unit testing of code modules, objects or discreet applications functions is a standard part of testing any distributed application; integration testing may be needed to determine if various modules, interfaces and other applications work together properly.
3.	HTML Validation	The need for this type of testing will be determined by the intended audience, the type of browser(s), use of standards such as JavaScripts, whether your site delivers pages based on browser type or target a common denominator, and how strictly you want to adhere to HTML standards.
4.	Link Testing	This type of testing determines if your site’s links to internal and external Web pages are working. A Web site with many links to outside sites will need regularly scheduled link testing, because Web sites come and go and URL’s change. Sites with many internal links (such as enterprise wide Intranet, which may have thousands of internal links) may also require frequent link testing.
5.	Usability Testing	Is your intended audience the general Internet public? In–house Intranet users? Computer experts? The intended audience will determine the “usability “ testing needs of the Web application. Additionally, such testing should take into account the current state of the Web and Web culture, because these will influence user expectations (for example, Web site navigation is expected to be extremely intuitive –Web users do not expect to read manuals or help files).
6.	Load Testing	Will there be a large number of interactions per unit time on the Web site? If so you may want to perform testing under a range of loads to determine at what point your system’s response time degrades or fails. The hardware and software chosen for the application and the design could have an impact on the same.
7.	Stress Testing	This refers to testing system functionality while the system is under unusually heavy or peak load; its similar to the validation testing mentioned previously but is carried out in a “high stress” environment. This requires that you make some predictions based on the expected load levels of the Web application.
8.	Reliability and Recovery Testing	If 24 X 7 uninterrupted availability is required, and depending on how critical the Web site is to the business, simulation of different “emergency” situations may be required to ensure that the production system can handle it successfully.
9.	Security Testing	If the site requires firewalls, encryption, user authentication, financial transactions or access to databases with sensitive data, there may be a need to test these and also test the site’s overall protection against unauthorized internal or external access.
10.	Regression Testing	If the iterative approach is adopted for development, there is a continuous need to retest the application that was initially developed and the code that was reworked to accommodate changes and bug fixes.
11.	Server Log / Report Testing	Web sites that use advertising and track site usage for marketing needs may need extensive testing to ensure the accuracy of the logging and reporting capabilities.

 Testing Requirements

Components of a Web Application   Unit Testing and Integration Testing  A unit is one component of a system. Integration testing refers to integrating different components together.  If different developers are developing the different components such as front end, middle tier and back-end, each should be considered a unit and tested and integration testing would involve testing the interfaces between these components.  

	Front END	Middle Tier	Backend
FUNCTIONAL TESTING	Required in all cases, could also include random testing other than unit and integration testing	Required in all cases, could also include random testing other than unit and integration testing	Required in all cases, could also include random testing other than unit and integration testing
HTML VALIDATION	It might not be possible to carry out all possible validations of the client environments. Risk analysis should be done and based on this, what to test should be decided and tested	NA	NA
LINK TESTING	This is very essential on large web sites  that encounter a lot of changes; to ensure that links are not outdated or inaccessible	NA	NA
USABILITY TESTING	This is required if any special set of users is targeted. Otherwise these inputs will occur as part of the first deployed iteration	NA	NA
LOAD TESTING	NA	Very essential when expecting  number of hits  > 200 per hour	Essential when database is suspect of handling large number of requests
STRESS TESTING	NA	Essential when reliability is important	Essential when reliability is important
RELIABILITY & RECOVERY TESTING	NA	Very essential when 24X7 up time is envisaged	NA
SECURITY TESTING	Can be used to check user authentication and authorization	Used to check firewall scenarios	Used to check access to database in a secure environment
REGRESSION TESTING	Necessary after the first iteration to ensure that the older functionality has not broken in the process of making changes or bug fixes	Necessary after the first iteration to ensure that the older functionality has not broken in the process of making changes or bug fixes	Necessary after the first iteration to ensure that the older functionality has not broken in the process of making changes or bug fixes

  Required Testing Tools

Mercury Interactive has come out with a suite of tools for testing Web applications, called Astra SiteTest. But this is still in its first release and not quite stable as yet. (Refer http://www.merc-int.com). The other tools that could be used are Mercury Interactive’s LoadRunner, SunTest Suite of testing tools (Refer http://www.suntest.com.), SilkTest from Segue (Refer http://www.segue.com.). Details of some other tools like parallel testing tool, Purify (for memory leakages) or RELACS are available in Tools repository on QD homepage. 

 Testing Process

For every application that is to be tested carry out the following steps.

   Step 1: Test Planning

Prepare Test Plan·        Define test objectives  ·        Identify environmental needs·        Define the test tools, if any to be used·        Identify test team members  ·        Assess risks  ·        Define critical success factors  ·        Write test cases·        Determine sequence/integration procedure·        Define test stop and resume criteria, example if 20 errors are found or when all the test cases have been run through or if more than 3 show stopper defects are found etc.·        Define the number of test cycles to be carried out, example 2 complete test cycles must be carried out to certify the system.·        Review test plan

Step 2: Test Execution

Setup test environment·        Identify Test Server (Dedicated server would be good)·        Setup restorable Test Data·        Setup applicable test tools Set up Configuration Mgmt process for test environmental assets such as ·        Web applications·        Test data·        Test scripts·        System configuration files Ensure that test environment accurately represents user platforms such as ·        Browsers·        Operating systems·        Hardware Do test execution & include the following:·        Unit Testing·        Integration testing·        System testing·        User acceptance testing

Step 3: Test Evaluation

Review test resultsDefine how defects are to be classified.Setup criteria in place to prioritize /evaluate defects

  

SDLC Phase	Test Plan Produced	Test Plan Executed
Analysis	System Test Plan
Analysis	Regression Test Plan
System Design	Integration Test Plan
Detailed Design	Unit Test Plan
Construction		Unit Test Plan
Construction		Integration Test Plan
Test		System Test Plan
Production/Maintenance		Regression test Plan

 

DEFECT LIFE CYCLE

Bug Life Cycle & Guidelines

In this tutorial you will learn about Bug Life Cycle & Guidelines, Introduction, Bug Life Cycle, The different states of a bug, Description of Various Stages, Guidelines on deciding the Severity of Bug, A sample guideline for assignment of Priority Levels during the product test phase and Guidelines on writing Bug Description

Introduction:

Bug can be defined as the abnormal behavior of the software. No software exists without a bug. The elimination of bugs from the software depends upon the efficiency of testing done on the software. A bug is a specific concern about the quality of the Application under Test (AUT).

Bug Life Cycle:

In software development process, the bug has a life cycle. The bug should go through the life cycle to be closed. A specific life cycle ensures that the process is standardized. The bug attains different states in the life cycle. The life cycle of the bug can be shown diagrammatically as follows:

The different states of a bug can be summarized as follows: 1. New
2. Open
3. Assign
4. Test
5. Verified
6. Deferred
7. Reopened
8. Duplicate
9. Rejected and
10. Closed

Description of Various Stages:

1. New: When the bug is posted for the first time, its state will be “NEW”. This means that the bug is not yet approved. 2. Open: After a tester has posted a bug, the lead of the tester approves that the bug is genuine and he changes the state as “OPEN”. 3. Assign: Once the lead changes the state as “OPEN”, he assigns the bug to corresponding developer or developer team. The state of the bug now is changed to “ASSIGN”. 4. Test: Once the developer fixes the bug, he has to assign the bug to the testing team for next round of testing. Before he releases the software with bug fixed, he changes the state of bug to “TEST”. It specifies that the bug has been fixed and is released to testing team. 5. Deferred: The bug, changed to deferred state means the bug is expected to be fixed in next releases. The reasons for changing the bug to this state have many factors. Some of them are priority of the bug may be low, lack of time for the release or the bug may not have major effect on the software. 6. Rejected: If the developer feels that the bug is not genuine, he rejects the bug. Then the state of the bug is changed to “REJECTED”. 7. Duplicate: If the bug is repeated twice or the two bugs mention the same concept of the bug, then one bug status is changed to “DUPLICATE”. 8. Verified: Once the bug is fixed and the status is changed to “TEST”, the tester tests the bug. If the bug is not present in the software, he approves that the bug is fixed and changes the status to “VERIFIED”. 9. Reopened: If the bug still exists even after the bug is fixed by the developer, the tester changes the status to “REOPENED”. The bug traverses the life cycle once again. 10. Closed: Once the bug is fixed, it is tested by the tester. If the tester feels that the bug no longer exists in the software, he changes the status of the bug to “CLOSED”. This state means that the bug is fixed, tested and approved. While defect prevention is much more effective and efficient in reducing the number of defects, most organization conducts defect discovery and removal. Discovering and removing defects is an expensive and inefficient process. It is much more efficient for an organization to conduct activities that prevent defects.

Guidelines on deciding the Severity of Bug:

Indicate the impact each defect has on testing efforts or users and administrators of the application under test. This information is used by developers and management as the basis for assigning priority of work on defects. A sample guideline for assignment of Priority Levels during the product test phase includes: 1.      Critical / Show Stopper — An item that prevents further testing of the product or function under test can be classified as Critical Bug. No workaround is possible for such bugs. Examples of this include a missing menu option or security permission required to access a function under test.
. 2.      Major / High — A defect that does not function as expected/designed or cause other functionality to fail to meet requirements can be classified as Major Bug. The workaround can be provided for such bugs. Examples of this include inaccurate calculations; the wrong field being updated, etc.
. 3.      Average / Medium — The defects which do not conform to standards and conventions can be classified as Medium Bugs. Easy workarounds exists to achieve functionality objectives. Examples include matching visual and text links which lead to different end points.
. 4.      Minor / Low — Cosmetic defects which does not affect the functionality of the system can be classified as Minor Bugs.

Guidelines on writing Bug Description:

Bug can be expressed as “Result followed by the action”. That means, the unexpected behavior occurring when a particular action takes place can be given as bug description. 1.      Be specific. State the expected behavior which did not occur – such as after pop-up did not appear and the behavior which occurred instead. 2.      Use present tense. 3.      Don’t use unnecessary words. 4.      Don’t add exclamation points. End sentences with a period. 5.      DON’T USE ALL CAPS. Format words in upper and lower case (mixed case). Mention steps to reproduce the bug compulsorily.