Test management approach

Introduction

First things first. Before elaborating on the Tmap subject it is important to have some background information about the origin of this article. The article is written as a part of the course Method Engineering found at the Method Engineering Encyclopedia, taught in the master study Business Informatics at the University of Utrecht. In this course several important methods in the field of software engineering and product software development projects are researched. The emphasis in the course is not so much in giving a summary of the technique or method but more on how the method is structured and used in practice. The reason behind this is that when a method is broken down into fragments, these fragments can be used to create a new method to suit a specific situation.

An array of methods and techniques is assigned in the course. Each student tackled one method. Picture a situation where a new software system has to be developed. Numerous methods and techniques are available to handle several aspects of the software development process. The thing is that often parts of methods suffice. This means that parts of methods can be left out. The issue here is that for these specific situations so called situational methods can be formed from the parts of existing methods. An overview of all the methods can be found at the Method Engineering entry.

Another key aspect of the course is the technique of meta-modelling. Per method several processes or products needed to be meta-modelled in order to describe the rules in using a process or creating a product. Therefore, also for Tmap several aspects are meta-modelled. The meta-models will be portrayed and explained throughout the article when the situation requires it.

An insight in testing

Before venturing in the domain of the TMap method, it is wise to have a look at the scope in which the method lies and have a feel about what the purpose is of the method. To fully grasp the testing process some definitions need to be given to explain in what context some given terms lie. Of course everyone knows what testing means, but how can it be used in such a way that its results can be used to make important decisions?

What is testing?

The main activity of a testing process is making a comparison. To make this comparison, an object to be tested and a point of reference or a so called test base are needed. The testing process is then needed to give information about the object and the differences between its current and required state. An ISO definition about testing is as follows:

“Activities performed to identify one or more characteristics of a product, process or service using a prescribed procedure” [ISO/IEC Guide 2, 1991]

So testing is about identifying the difference between the actual and required state. This of course has to do with the quality of the test object. Though quality can be perceived as “meeting the demands”, the testing process is about assigning value to the quality of the test object. Low quality meaning required state has not yet been matched by far, high quality meaning close proximity to the required state.

When assigning a value to the quality factor of a test object, management in an organisation can make decisions based on this value. It can give insight in the risks that are taken when accepting a product with less quality than planned. This is what the testing process is all about. It is a tool for decisions on a higher level to give insight in the level of quality of a product that the organisation wishes to put on the market. Is the software product good enough to distribute? Are the requirements that we want to deal with in this version all implemented the way they were meant to? Is our system as fault tolerant as we planned it to be? These are all questions that are asked on a higher level that can be answered with correct testing of the test object. So in a sense, testing is a tool on a high level, but a method on a lower level. On a high level a question is put in the black box of testing, and the answer comes out. On a low level a question comes in, and a set of prescribed actions, techniques, activities and steps help to provide an answer to the question.

Testing is not to be mistaken for checking. Checking is about testing intermediate products and development processes. In the context of software development, it answers the question: “Is the building process properly executed?”. Testing on the other hand has a larger scope. It is about inspecting finished products or finished intermediate products. It answers the question: “Does the product meet the specified demands?”. The line between these two concepts is thin, but has mostly to do with scope. Formally stated testing is about finding errors, but not as a goal but as a means. The goal is to assign a value to quality. The lack of quality is measured by the fault sensitivity of the test object. This leads to another, more suitable definition of testing:

“Testing is a process of planning, preparing, executing and evaluating used to identify the characteristics of an information system and pointing out the difference between actual and required state” [Testen volgens TMap 2e druk, 2000]

Why test?

A question that can be asked is “why test?”. It is nice that you can assign a value to the quality of a software product, but what is the use of this value, what can we do with it? The use of IT is widespread in the organisational world. It is often used to support various business processes within companies. The market poses high demands on these companies with trends such as globalisation for example. A high level of flexibility is therefore desired in combination with shorter planning periods for releasing products quicker.

To cope with these demands management is becoming more and more dependant on the quality of the information systems they sell. Before launching a new information system or distributing it a software company should ask itself the question whether it should do this or not. As ISO formulates: Is it certain that the product possesses the characteristics to meet the given or self-evident demand? Especially the self-evident demand is important. It means that the product should have certain characteristics that go without saying, i.e. have characteristics that would seem normal to have under the given conditions. Have all the errors been dealt with, and has this not yielded new errors? Does the system provide the reliable solution for the situation for which it was designed? Can it be launched within the borders of the present risk factors? For not having to answer these questions in the final stages of the development process a structured testing process is required. It can answer these questions in an earlier stage so fitting solutions can be effectively implemented in time.

Another question that can be asked is: “Where does the process of testing stand in the scope of organisations nowadays?”. Unfortunately, the answer is that in many organisations the process of testing is underdeveloped. They tend to postpone the introduction of a testing structure to structurally improve quality of their product. Lack of organisational know-how to set up such a branch is often the cause of this. The scope and result of testing is also perceived in a wrong way. The result of testing is thought of as perfect. The common thought is that it is able to identify all errors within a product. This is unfortunately not the way this works in practice.

Use of testing in practice

The development and maintenance of information systems demands special attention for quality. In this case, quality is directly linked to the demands the customer or intended user has for the system. In many cases a software product is tailor-made or is a specialised product serving a specified process in the customers business. It is not overreacting when it is stated that not many branches within the economy suffer from an image of having poor quality as the IT industry does.
To obtain the demanded level of quality more and more activities are planned. Some say activities to prevent errors suffice. This would mean that with the right preparation no errors would occur at any time. Due to the fact that errors are dependant on numerous incontrollable factors this is a perfect situation that does not match with the situation in practice.

The goal is to prevent errors from occurring by identifying the cause of the present errors. In fact it is trying to cure a disease rather than trying to fight the symptoms caused by it. This does not mean that all errors can be prevented as stated earlier. It is a combination of prevention and trying to optimize the process of finding errors. The testing process becomes more efficient when the time between the creation and detection of an error is decreased. In an ideal situation, developers of software are more motivated to produce less faulty code when they are supported by a test team that indicates in what area they are most likely to make mistakes. The test process as a whole would then yield not only a direct, but also an indirect positive result on the quality of a product.

As stated in multiple related articles, quality of a product should be built, not tested. The improvement of quality simply by the process of testing is extremely cost intensive. Using this principle, a test indicating a poor system should not be tested over and over again, but be completely redesigned to prevent the occurrence of similar defects in the future. Software companies rarely come to the conclusion that it is better to completely redesign a product. Maybe the thought is there, but other options tend to be more plausible than a complete redesign though it would be the sensible thing to do when the scope is extended over the overall project. Redesign in an early stage prevents the production of a severely quality lacking product that turns out to be an expensive fiasco.

The testing process is not just about executing a test. It should be looked at in a broader scope. There are more activities involved in the process than just the actual test. The planning and preparation are equally if not more important than the actual test. Testing as a concept is often looked at as just performing the tests but actually in most cases the actual execution of a test contributes to just forty percent of the total time spent on the process. The remaining sixty percent is comprised of planning, preparation, specification and completion of the test.

Organisations tend to give less attention to these concepts whereas they are the most important aspect of the process. If, for example, the execution of a test is not properly prepared, how can the results of the test be reliable enough to base decisions on them? There is even a trend that the balance between total test process time and test execution time is shifting. Less hours of the total amount of time available is spent on the actual execution, and more time is assigned to the supporting processes.

Testing is not about acceptance. It is about supporting the decision making process. It gives advice about the quality of a product. In the scope of a development process, for example within the DSDM development method, it is not a separate phase. It is a series of prescribed activities that have to be performed from the start of the overall development. So from the view of just the testing, it is a separate process, but it cannot be detached from a development process. After all, there is nothing to test if nothing is built. From the moment the functional demands or requirements are documented, the test process can be planned. Though the test and building process tend to collide both have to be given equal attention to. The building cannot continue without proper testing, and the testing cannot be performed if nothing is built. Testing is an expensive undertaking. It can form up to fifty percent of total development costs. This may seem much, but research has shown that costs increase exponentially when the time between an error and its detection increases. So the relation between time and cost is exponentially [Boehm, 1981].

What is quality and how does it relate to testing?

To close the discussion about what testing is, one aspect still has to be looked at. The concept of quality is a vague concept. What is quality? How can quality be measured? How can we assign a value to the concept of quality? This remains to this date an important issue in the IT world. Because testing is only a part of the puzzle in trying to achieve a certain level of quality, other aspects have to be involved. Testing in itself cannot guarantee quality. It can only detect if a certain level of quality is reached. The ISO gives the following definition of quality:

“Quality is the sum of characteristics of a product or service that play a part in meeting specified and self-evident needs” [ISO 8402, 1994]

The definition is still pretty vague. Especially the part that states that self-evident needs have to be incorporated in the quality. What are the self-evident needs? In an ideal situation there are no self-evident needs because all of them have been recorded in the specified needs. When dealing with the concept of quality it is evident that it is difficult to use and be specific at the same time. The concept should be specified further for it to be used in a sense that decisions can be based on it.
The way to do this is to break up the concept of quality into sub characteristics.
Per sub characteristic, some form of measurement should be specified so a value can be assigned to it. In 1977, McCall proposed to divide the concept of quality into so called quality attributes. ISO defined a standard for these quality attributes [ISO 9126-1, 1999].

Six characteristics have been defined, each with corresponding sub-characteristics. The idea is that all these characteristics together tackle the entire field of quality present in the field of software development. The attributes are as follows:

• Functionality
• * Suitability
• * Accuracy
• * Interoperability
• * Security
• Reliability
• * maturity
• * fault tolerance
• * recoverability
• Usability
• * understandability
• * learnability
• * operability
• * attractiveness
• Efficiency
• * Time behaviour
• * Resource utilization
• Maintainability
• * analysability
• * changeability
• * stability
• * testability
• Portability
• * Adaptability
• * Installability
• * Co-existence
• * Replaceability

Summarizing the different aspects of testing discussed it can be said that testing is more difficult than it would seem at first glance. It is not just about detecting errors, it is about assigning value to the quality of a system so decisions can be based upon it. It is an underestimated field in the software industry that has only recently gained attention with the development of structured testing methods such as TMap. It provides management with tools to assess its main product and enables it to make decisions based on rational results rather than “gut feelings”. To do this the concept of quality is broken down in measurable quality attributes on which tests can be specified. How the TMap method deals with these attributes will be explained in the next part.

Tmap fundamentals

The Tmap method approaches the testing process in a structured way. In order to define what basic demands should be met in order to successfully implement a structured testing process in an organisation several characteristics or bottlenecks in businesses or projects should be reviewed. The process of testing is not simply conducting an investigation indicating that there are errors in a system; it is a process that should be professionally dealt with and supported in all layers of an organisation. There are several key factors in a project or organisation that can negatively influence the way testing is done or is to be planned.

Bottlenecks

Time pressure

Phasing of test activities:
In projects a structured approach for testing is often omitted. Basic questions like “what test activities should be performed and by whom?” are not always answered. The lack of planning and phasing of test activities causes the test process to be unmanageable and therefore unreliable.

Techniques and tools

Requirement specification

Coordination of test activities

Recommendations

The factors and examples given shed a light on the difficulty and importance of a structured approach of testing a software product. There are so many factors involved that it is easy to negatively influence the quality of the whole testing process. It is easy to see that these factors have to be controlled in such a way that the quality of the performed tests is as high as possible. To do this, a testing method is required to structure the way tests are prepared and performed while keeping the scope of the overall development process in mind. Some recommendations to such a method are given.

Reduction of time pressure

The availability and adequacy of personnel is also important. If the personnel is properly trained, tasks and activities are faster and more thoroughly performed. Another important issue is making agreements prior to starting the testing process. With skilled personnel, well worked out planning, solid agreements and progress monitoring time pressure can be heavily reduced.

Project based

Phasing of test activities parallel to the software development process

Structured and suited use of tools

Teambuilding and reduction of conflicting interests

Setup of a test helpdesk

• Prescriptions for methods, techniques and tools
• Maintenance of these prescriptions
• overall planning of concurring test activities
• creation and maintenance of the test infrastructure
• Education and coaching of test personnel
  

  Method structure

• Phasing of test activities, related to the development cycle of the software product (P)
• Embedding in organisation (O)
• Matching infrastructure (I)
• Usable Techniques for executing the activities (T)
  

  Phasing

In the TMap method three main activities can be distinguished: planning, preparation and execution. These activities are devided into five phases: Planning and Control, Preparation, Specification, Execution and Completion. The completion phase was added to finish the test process properly and to carry the responsibility to hand over the results to the management. The TMap phasing model is a generic model which means that it is applicable to a wide selection of tests. Per phase different activities can be distinguished. It is up to the manager to select which of the available activities in the TMap method are assigned to which phase. This selection has to be done because it is often not required to execute all the available tests. It all depends on which quality attributes matter the most for the product or subproduct.

To accurately describe the phases and provide a view of the relation of Tmap to the overall software development process two models will be given. The first is a process model that shows how the different phases relate to each other. The second is a metamodel of the TMap method in relation to the software development process. The reason behind this is that such models can provide some additional information that plain text cannot.

The first model, as show above, illustrates how the different phases relate to one another. Though it may seem logical, there is something awkward about it. Though the Planning and Control phase passes on to the Preparation phase, it still continues in itself. This is because it contains activities that have to be performed throughout the process which cannot be put in other phases. This has to do with the control aspect of the phase which will be elaborated on later. The remaining phases seem to have a logical order. To view these phases in relation to a software development process, let’s have look at a metamodel to describe this.

In the given metamodel several entities are conceptualized. Let’s first look at the project entity. The project is the conceptualization of a software development process because these are often captured within the boundaries of a project. In many cases, a project is devided into stages with each stage producing a subpart of the product. When you add up all these different subparts you can create the overall product. A project can consist of multiple stages, but has to have at least one. Each stage corresponds with one project because it cannot be in multiple projects. Then let’s have a look at the method, which in this case represents the Tmap method. A method is comprised of different phases with each phase containing different activities or techniques.

The interesting part is the connection with the project. This model states that each method phase can correspond with one or more project stages and vice versa. It can even be so that a method can correspond with a project stage, or that a project can correspond with a method phase, but this is out of the scope of the Tmap method. The reason for the association is the notion that one Tmap phase can overlap several development stages, but could also be linked to just one. It was earlier stated that it is wise to plan the phases in relation to the project stages, this is how the model explains this.

The final part of the model is the use of techniques in phases. As with the Tmap method, each phase makes use of certain techniques. It is not necessary, which explains the zero, but multiple techniques can correspond with a phase. A technique in itself is not obliged to be connected to a, or one phase. It can also be used in multiple phases. A technique in itself results in the creation of a deliverable, but again, this is not an obligation which explains the zero. A technique can also be supported by a tool, but this is not very important in explaining the Tmap method. In general, the whole idea is to perform certain prescribed techniques on objects in order to create deliverables. These deliverables is what the techniques are designed for. It can be the results of a test, or even a document which gives advice on which decisions are based. Either way, it is important that the outcome of the technique is not discussable and reliable. For this to be achieved the technique has to be executed exactly the way it is prescribed to be executed.

Planning and Control

Now the scope and the phasing as a whole have been looked at it is time to take a closer look to the individual phases and explain what phase is intended for what purpose. Also some activities and techniques will be elaborated on per phase. The first phase is the Planning and Control phase. It starts in the specification stage of a software development project. It is the most important phase, and is underestimated in most situations. In this early phase difficult aspects of the testing process have to be anticipated such as the assigning of personnel and resources and the planning of certain tests in relation to milestones of the software project.

One of the key aspects in this phase is the creation of a so called strategy matrix. It gives details about the different quality attributes important to the system. Only in a perfect world can a system be tested on all quality attributes with a hundred percent attention. Together with the client it has to be decided what quality attributes are important for what subpart of the system and how much attention each attribute should have. This could result in the following examplary strategymatrix and its meta datamodel:

Also given is a meta datamodel of this strategy matrix because it is important for the future phases that this is done correctly. Within the matrix there can be an arbitrary number of subsystems per row, but the quality attribute and a relative importance have to be present. Also important is that the percentages of the importance column add up to a hundred percent. This is logical, because you cannot give more than a hundred percent attention to the attributes. This strategy matrix is used in later phases to base tests on and calculations for manhours.

Also during the Planning and Control phase the design for the testinfrastructure is made. The infrastructure is important because it reflects the situation the product is designed for. It makes sense to test the product in an environment that it was designed for, for the results of the tests to be reliable. The first part of the Planning and Control phase can be captured in the development of a master testplan. This mastertestplan contains all the activities needed to set up a test process and creating the supporting concepts.

De remaining activities have to do with the control aspect of the Planning and Control phase. These activities aim at monitoring the progress of the tests and their relationship with the use of time and resources. During the creation of the mastertestplan a planning was set up to dictate how much time could be spent on what test. The monitoring aspect of the phase can indicate if the testprocess is running smoothly or not, whether correcting actions are necessary or not. The most important product of the testprocess is the advice it gives about the quality of the product. This can also be the quality of subproducts. So during the phases the monitoring can provide an indication about where the system stands and what its quality is in relation to the specified strategy matrix. The process should be able to answer questions the decisionmaking asks of it. So not just a simple “no, the system is not ready”, but an indication of what quality attribute is lacking in what subsystem and how much time is necessary to fix it. Decisions could then be to go into production after all, or to postpone the release untill the quality constraint is met.

Preparation

After the mastertestplan has been created the Preparation phase is initiated. Input for the Preparation phase is the testbase. This testbase is the object on which the tests will be executed and the mastertesplan created in the earlier phase. In conjunction with the builders of the product the testbase will be devided into subparts. In most cases these subparts of the testbase correspond with the subparts of the overall product as specified in the strategy matrix. For each of the subparts testtechniques will be assigned and a planning will be made for the tests to be executed.

Specification

In the Specification phase the testcases are specified and the corresponding testinfrastructure defined in an earlier phase is realised. The testcases are created in two steps, the logical and physical testcase. From the moment the testbase is available it is possible to define testcases for testing. For a given testcase this could be specifying the starting situation, the process within and predicting the outcome of the result. This is a logical representation of the testcase. When the testbase is actually realised more technical aspects of the testcases can be specified such as testscripts. To facilitate these testcases the corresponding infrastructure is created. As stated earlier, this infrastructure has to match the environmental factors of the situation in practice.

Execution

In the Execution phase the actual testing takes place. It starts the moment the first testable components of a system are available. The component forms the testbase on which the tests are performed. The part is first checked for completeness and then installed in the corresponding test infrastructure. The first test is if the component actually works in its test environment. If this is the case the testdata is prepared for the execution of the test. This is an important process that is often executed with actual system functions. If all these demands are met, the so called pre-tests are executed first to check the main functions of the testobject. They indicate whether the testobject has a sufficient level of quality for it to be tested thouroughly. If the pre-tests indicate that all is clear, the actual testscripts can be run on the testobject and the test executed. During the testing intermediate results can be handed over to management for decisionmaking, for example tot stop the test if necessary. This has to do with the Planning and Control phase which monitors the overall testing process.

Completion

After all the tests have ben completed there are some remaining activities to be done. These activities tend to be forgotten because of the time pressure involved in the actual testing. This would be a waste of resources because the testresults are the reason the tests were conducted and they have to be documented properly for further and future use. A selection is made for certain testcases and testdate to be preserved for future cases. In a later case this data does not have to be recreated and the testcases not reinvented. Also an evaluation takes place. Have the tests been conducted properly? Is the result what was exptected from it? Should some of the processes be changed or planned diffently in the future to provide better outcome? These are all evaluating questions that seem redundant but can provide significant gain of time in future test processes. Another aspect is the setup of a gain / loss evaluation. It can be calculated how much time and money the testing process cost, but also what it gained. If there was any doubt within the organisation to spend a lot of time on testing, a nice gain/loss ratio will surely fix that.

Organisation

Infrastructure

Techniques

Algorithm Test

One of the available techniques available in the Tmap method is an algorithm test. It is just one of the many available techniques but it is a good example of demonstrating how Tmap techniques work. To do this an example is given as wel as a meta process model. An algorithm test is designed to test the structure of an algorithm and is therefore also known as a structure test. An algorithm can be comprised of several decisionpoints for which several options are possible. If there are five decisionpoints with each five options you get the product of these as possible paths, namely five times five. For each of these paths the outcome has to be tested in order to verify that the algorithm is functioning properly. The process model and meta process model to structure this are as follows:

The technique as explained vaguely is elaborated on in the process model on the left. The first action when viewing the algorithm is to decide how many decisionpoints there are, and where they are in relation to eachother. When you have these, you can devise all possible testpaths from them. A testpath in this case is a walk through the entire algorithm and having selected one of the possible options at each decisionpoint. The next step is to specify a testcase for each path. What data should be present an can be used to test if the path functions like it is supposed to? This brings us to the next point which is gathering the testdata. When all these steps have been completed a testscript can be set up to facilitate the whole testingprocess.

The metamodel is relatively simpel. An overall process, in this case the setup of an algorithm test, consists of one or multiple process steps. Each step can be a beginning stap and can have a next step. This explains the relation it has with itself identifying that it has a next step. From this meta processmodel any processmodel can be created. Steps can be added and left out as pleased.

Test Point Analysis (TPA)

Another important technique in the Tmap arsenal is the Test Point Analysis or TPA. This is one of the most, if not the most important technique available because it deals with the calculation of the manhours needed for the overall test process. This gives information about what the testprocess costs and in what timespan it can be achieved which reflects in the planning.

External links

• Official TMap website
• Overview of some basic testing techniques
• Sogeti, One of the founders of the method
• Improveqs, One of the founders of the method
• Testing background information

  References