ISO/IEC 14598-6:2001

INTERNATIONAL ISO/IEC
STANDARD 14598-6
First edition
2001-06-01
Software engineering — Product
evaluation —
Part 6:
Documentation of evaluation modules
Ingénierie du logiciel — Évaluation du produit —
Partie 6: Documentation des modules d'évaluation
Reference number
©
ISO/IEC 2001
PDF disclaimer
This PDF file may contain embedded typefaces. In accordance with Adobe's licensing policy, this file may be printed or viewed but shall not
be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In downloading this
file, parties accept therein the responsibility of not infringing Adobe's licensing policy. The ISO Central Secretariat accepts no liability in this
area.
Adobe is a trademark of Adobe Systems Incorporated.
Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation parameters
were optimized for printing. Every care has been taken to ensure that the file is suitable for use by ISO member bodies. In the unlikely event
that a problem relating to it is found, please inform the Central Secretariat at the address given below.
© ISO/IEC 2001
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means, electronic
or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or ISO's member body
in the country of the requester.
ISO copyright office
Case postale 56 � CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.ch
Web www.iso.ch
Printed in Switzerland
ii © ISO/IEC 2001 – All rights reserved

Contents Page
1 Scope .1
2 Conformance.1
3 Normative references .1
4 Terms and definitions .2
5 The evaluation module concept.2
6 Format for documentation of an evaluation module.3
6.1 EM0 : Foreword and introduction .3
6.1.1 Foreword.3
6.1.2 Introduction.3
6.2 EM1 : Scope.3
6.2.1 Characteristics.3
6.2.2 Level of evaluation .3
6.2.3 Techniques.4
6.2.4 Applicability.4
6.3 EM2 : References.4
6.4 EM3 : Terms and definitions.4
6.5 EM4 : Inputs and metrics .4
6.5.1 Input for the evaluation .4
6.5.2 Data elements.5
6.5.3 Metrics and measures .5
6.6 EM5 : Interpretation of results.5
6.6.1 Mapping of measures.5
6.6.2 Reporting.5
6.7 EMA : Application procedure .5
6.7.1 Definition of Technical Terms Used.5
6.7.2 Resources Required.5
6.7.3 Evaluation instructions .6
6.7.4 Documentation.6
Annex A (informative) Development of evaluation modules .7
Annex B (informative) Example of an evaluation module – Fault density.8
Annex C (informative) Example of an evaluation module – Functionality .12
Annex D (informative) Example of an evaluation module – Usability and Quality in Use .25
Bibliography.31
© ISO/IEC 2001 – All rights reserved iii

Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical Commission)
form the specialized system for worldwide standardization. National bodies that are members of ISO or IEC
participate in the development of International Standards through technical committees established by the
respective organization to deal with particular fields of technical activity. ISO and IEC technical committees
collaborate in fields of mutual interest. Other international organizations, governmental and non-governmental, in
liaison with ISO and IEC, also take part in the work.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 3.
In the field of information technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1.
Draft International Standards adopted by the joint technical committee are circulated to national bodies for voting.
Publication as an International Standard requires approval by at least 75 % of the national bodies casting a vote.
Attention is drawn to the possibility that some of the elements of this part of ISO/IEC 14598 may be the subject of
patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.
International Standard ISO/IEC 14598-6 was prepared by Joint Technical Committee ISO/IEC JTC 1, Information
technology, Subcommittee SC 7, Software engineering.
ISO/IEC 14598-6 is intended for use in conjunction with ISO/IEC 9126-1 (in preparation) which will replace
ISO/IEC 9126:1991.
ISO/IEC 14598 consists of the following parts, under the general title Software engineering — Product evaluation:
� Part 1: General overview
� Part 2: Planning and management
� Part 3: Process for developers
� Part 4: Process for acquirers
� Part 5: Process for evaluators
� Part 6: Documentation of evaluation modules
Annexes A, B, C and D of this part of ISO/IEC 14598 are for information only.
iv © ISO/IEC 2001 – All rights reserved

Introduction
Software product evaluation depends on a set of evaluation techniques and metrics that provide information about
the quality characteristics of the software. Many metrics and associated methods for using the measurement
results can be used for specific software product evaluation. ISO/IEC 9126-2 and ISO/IEC 9126-3 provide example
metrics that correspond to one sub-characteristic. It is difficult to use these metrics consistently in an organisation.
It may be necessary to develop new metrics for specific use. Therefore, it is necessary that a supporting function
(see 14598-2) in the organisation specifies each metric for correct and consistent use within the organisation. The
format for documenting a metric and associated methods, as well as guides for their use, should be standardised.
The concept of an evaluation module provides a solution to this need.
An evaluation module specifies the evaluation methods applicable for evaluating a quality characteristic and
identifies the evidence it needs. It also defines the elementary evaluation procedure and the format for reporting the
measurements resulting from the application of the techniques.
A consistent way of documenting evaluation modules has a number of advantages:
� It provides a common reference in the description of the theoretical basis of evaluation modules.
� It identifies a minimal set of requirements for documenting and developing evaluation modules.
� It provides a necessary tool in collecting and cataloguing the large number of evaluation modules anticipated.
Evaluation modules provide a flexible and structured approach to making metrics applicable for evaluating
intermediate and completed products. The use of evaluation modules produced according to this part of ISO/IEC
14598 helps to ensure that software product evaluations can be repeatable, reproducible and objective.
The format for documenting an evaluation module takes into account the following:
� It will be applied within the context of the evaluation of software products.
� The format supports the need for developing new metrics with respect to state of the art.
� The format provides a precise definition of metrics and their application.
� It provides the information needed for those who will use them for an evaluation.
Annex A provides guidance for the development process for new evaluation modules.
Annexes B, C and D are examples of evaluation modules.
© ISO/IEC 2001 – All rights reserved v

INTERNATIONAL STANDARD ISO/IEC 14598-6:2001(E)
Software engineering — Product evaluation —
Part 6:
Documentation of evaluation modules
1 Scope
This part of ISO/IEC 14598 defines the structure and content of the documentation to be used to describe an
Evaluation Module. Evaluation modules are intended to be used within the context of the ISO/IEC 9126 and the
ISO/IEC 14598 multipart standards.
This part of ISO/IEC 14598 is intended to be used by experts in evaluation technology such as testing laboratories,
research institutes and others when producing new evaluation modules.
2 Conformance
The documentation of an evaluation module conforms to this part of ISO/IEC 14598 if it meets the requirements of
clause 6 (format for documentation of an evaluation module).
3 Normative references
The following normative documents contain provisions which, through reference in this text, constitute provisions of
this part of ISO/IEC 14598. For dated references, subsequent amendments to, or revisions of, any of these
publications do not apply. However, parties to agreements based on this part of ISO/IEC 14598 are encouraged to
investigate the possibility of applying the most recent editions of the normative documents indicated below. For
undated references, the latest edition of the normative document referred to applies. Members of ISO and IEC
maintain registers of currently valid International Standards.
ISO/IEC 9126-1, Software engineering — Product quality — Part 1: Quality model.
ISO/IEC 12207, Information technology — Software life cycle processes.
ISO/IEC 14598-1, Information technology — Software product evaluation — Part 1: General overview.
ISO/IEC 14598-2, Product evaluation — Part 2: Planning and management.
ISO/IEC 14598-3, Product evaluation — Part 3: Process for developers.
ISO/IEC 14598-4, Software engineering — Product evaluation — Part 4: Process for acquirers.
ISO/IEC 14598-5, Information technology — Software product evaluation — Part 5: Process for evaluators.
© ISO/IEC 2001 – All rights reserved 1

4 Terms and definitions
For the purposes of this part of ISO/IEC 14598, the following terms and definitions apply.
4.1
evaluation module
a package of evaluation technology for measuring software quality characteristics, sub-characteristics or attributes
NOTE The package includes.
� evaluation methods and techniques
� inputs to the evaluation
� data to be measured and collected
� supporting procedures and tools
4.2
evaluation technology (technology used for evaluation)
the techniques, tools, metrics, measures and other technical information used for evaluation [ISO/IEC 14598-2]
5 The evaluation module concept
Evaluation of software products can be a comprehensive task. Different aspects of quality characteristics and sub-
characteristics may require different evaluation techniques to be applied and different data to be collected. In order
to manage this complexity, an evaluation should be structured into manageable units. Each of these units can
cover one or more quality aspects. However, each unit should be focused on the evaluati
...