ISO/IEC 9646-2:1991

ISO/IEC
I NTER NATIONAL
STANDARD 9646-2
First odi t ion
1991 -07- 1 5
Information technology - Qpen Systems
Interconnection - Conformance testing
methodology and framework -
Part 2:
Abstract test sui te specification
Technologies de l'information - Interconnexion de systèmes oiiverts -
Cadre général et méthodologie des tests de conformité OS1 -
Partie 2: Spécification des suites de tests abstraites
Reference number
ISO/IEC 9646-2 : 1991 (E)
Contents Page
Foreword . v
....................................................... vi
Introduction
1 Scope . 1
2 Normative references . 1
3 Definitions . 1
4 Abbreviations . 1
5 Compliance . 2
6 Conformance requirements in OS1 International Standards and CCITT
Recommendations . 2
.................................................. 2
6.1 Introduction
6.2 General requirements . 2
6.3 Conformance clauses . 2
7 Requirements on PICS proforrnas . 2
8 Abstract Test Suite production process leading to conformance testing
standards . 3
9 Conformance requirements and PICS proforma . 3
10 Test Suite Structure and Test Purposes . 3
10.1Basic requirements . 3
10.2Çpecification of the test suite structure . 3
10.3Specification of the test purposes . 4
IOACoverage . 5
10.5TSÇ&TP compliance clause . 6
11 Specification of generic test suites . 6
I2 Abstract Test Methods . 6
12.1lntroduction . 6
12.2General principles . 6
1 2.2.1 Lower Testers . 6
12.2.2 Upper Testers . 7
12.3GeneraJ specification of the ATMs for end-system IUTs . 7
12.3.1 Introduction . 7
12.3.2 The Local test method . 7
12.3.3 The Distributed test method . 7
12.3.4 The Coordinated test method . 7
12.3.5 The Remote test method . 9
12.3.6 Singlelayer and embedded variants . 9
12.4Variants for single-protocol IUTs . 9
O ISO/IEC 1991
All rights reserved . Unless otherwise specified. no part of this publication may be reproduced or
utilized in any form or by any means. electronic or mechanical. including photocopying and
microfilm. without permission in writing from the publisher .
ISO/IEC Copyright Office Case postale 56 CH-I211 Genève 20 Switzerland
Printed in Switzerland
ii
ISO/IEC 9646-2 : 1991 (E)
12.4.1 Introduction . 9
12.4.2 The LS test method . 9
12.4.3 The DS test method . 10
12.4.4 The CS test method . 10
12.4.5 The RS test method . 10
12.5Variants for multi-protocol lUTs . 10
12.5.1 Introduction . 10
12.5.2 The LSE test method . 11
12.5.3 The DSE test method . 11
12.5.4 The CSE test method . 11
12.5.5 The RSE test method . 12
12.6ATMs for open relay-systems . 12
12.6.1 Introduction . 12
12.6.2 The YL test method . 12
12.6.3 The YT test method . 12
..............................................
12.7Choice of ATM 12
...........................................
12.7.1 Introduction 12
12.7.2 Comprehensive testing service . 13
12.7.3 IUT environments .
12.7.4 Applicability of the ATMs . 13
12.8Test coordination procedures . 13
............................................
13 Specification of ATSs 13
13. 1General . 13
13.2Testcases . 14
13.3ATS conformance clause . 15
13.4 Consistency with protocol . 15
...........................................
14 Specification of a TMP
.......................................
15 Use of an ATS specification
16ATSmaintenance . 16
Annexes
A (normative) Requirements and guidance for a PICS proforma . 17
A.l Introduction . 17
A.2 Relationship between PICS proformas and conformance
requirements . 17
A.3 Generallayout . 18
A.4 Copyright . 18
A.5 First section - Identification of the implementation . 18
A.6 Second section - Identification of the protocol . 19
A.7 Global statement of conformance . 19
A.8 Other sections - Capabilities . 19
...........................................
A.8.1 Introduction 19
A.8.2 Initiator/responder capability . 19
A.8.3 Major capabilities . 19
A.8.4 Timers and protocol parameters . 19
A.8.5 PDUs . I9
A.8.6 PDU parameters . 19
A.8.7 Negotiation capabilities . 20
A.8.8 Protocol error handling . 20
A.8.9 Multi-layer dependencies . 20
A.8.lOOther conditions . 20
A.9 Formats of the tables . 20
A.9.1 Structure of the tables . 20
A.9.2 Symbols and conventions . 22
A.9.3 Instructions for completing a PICS proforma . 23
B (informative) Guidance for protocol specifiers to facilitate conformance
testing . 24
B.l Introduction . 24
8.2 Guidanceonscope . 24
8.3 Guidance on normative references . 25
8.4 Guidance on requirements and options . 25
iii
ISO/IEC 9646-2 : 1991 (E)
I
........................................... 25
I 8.5 Guidanceon PDUs
8.6 Guidanceon states . 26
8.7 Guidance on FDTs . 26
8.8 Miscellaneous guidance . 26
C (informative) Incomplete static conformance requirements . 27
D (informative) Guidance on generic test cases . 28
D.l Introduction . 28
0.2 Description of generic test cases . 28
D.3 Relation of genetic to abstract test cases . 28
D.4 Derivation of abstract test cases from generic test cases . 28

ISO/IEC 9646-2 : 1991 (E)
Foreword
IS0 (the International Organization for Standardization) and IEC (the
International Electrotechnical Commission) form the specialized system
for worldwide standardization. National bodies that are members of IS0
or IEC participate in the development of International Standards through
technical committees established by the respective organization to deal
with particular fields of technical activity. IS0 and IEC technical com-
mittees collaborate in fields of mutual interest Other international or-
ganizations, governmental and non-governmental, in liaison with IS0
and IEC, also take part in the work
In the field of information technology, IS0 and IEC have established a
joint technical committee, ISO/IEC JTC 1. Draft International Standards
adopted by the joint technical committee are circulated to national bod-
ies for voting. Publication as an International Standard requires ap-
proval by at least 75 % of the national bodies casting a voie.
International Standard ISO/IEC 9646-2 was prepared by Joint Technical
Committee ISO/IEC JTC 1, lnformation technology.
ISOAEC 9646 consists of the following parts, under the general title In-
formation technology - Open Systems Interconnection - Conformance
testing methodology and framework:
-- Part 1: General concepts
- Part 2: Abstract test suite specilicaiion
- Part 3: The Tree and Tabular Combined Notation (TTCN)
- Part 4: Test realization
-- Part 5: Requirements on lest laboraiories and clients for the con-
formance assessment process
Annex A forms an integral part of this pait of ISO!IEC 9646. Apnexes W,
C and D are for information only.
V
ISO/IEC 9646-2 : 1991 (E)
Introduction
This part of ISO/IEC 9646 provides a common approach to the specificatic of OS1
conformance test suites at a level which is independent of the means of executing
those test suites (hereafter called “abstract test suites”). This level of abstraction is
suitable for standardization and facilitates the comparison of results produced by dif-
ferent organizations which run the corresponding executable test suites.
6 and 7 recall that there are requirements on OS1 protocol specifiers which
Clauses
have to be fulfilled before there can be an objective basis for the process of develop-
ing an abstract test suite. The need is expressed for consistent conformance clauses
and for PICS proformas in International Standards or CCITT Recommendations
which specify OS1 protocol standards.
Clauses 8 to 16 describe the process of developing an abstract test suite, including
the design criteria to be used and guidance on its structure and coverage. The pos-
sible abstract test methods are defined and guidance is given to help the test suite
specifier to decide which test method(s) to use in the production of a particular test
suite. Requirements and guidance are given on the specification of abstract test
cases. These include the subdivision of test cases into test steps and the assignment
of test verdicts to test outcomes.
The test suite specifier is also required to provide information to the test realizers
(e.g. limitations governing test case selection).
Finally, guidance and requirements are given on test suite maintenance.
This pari of ISO/IEC 9646 is also to be published by CClTT as Recommendation
X.291 (1991).
vi
~
INTERNATIONAL STANDARD
ISO/IEC 9646-2 : 1991 (E)
Information technology - Open Systems Interconnection -
Conformance testing methodology and framework -
Part 2:
Abstract test suite specification
and IS0 maintain registers of currently valid International
1 Scope
Standards.
1.1 This part of ISO/IEC 9646 specifies the requirements
IS0 7498 : 1984, Information processing systems - Open
and gives guidance for the production of system-independent
Systems Interconnection - Basic Reference Model.
conformance test suites for one or more OS1 International
(See also CClTT Recommendation X.200 (1984))
Standards or CClTT Recommendations. In particular, it is
applicable to the production of all conformance testing stand-
ISOKR 8509 : 1987, Information processing systems - Open
OS1 and ISDN two-party protocols, including all drafî
ards for Systems Interconnection - Service conventions.
versions of such conformance testing standards.
(See also CClTT Recommendation X.21 O (1988))
1.2 This part of ISO/IEC 9646 is applicable to the production IS0 8825 : 1990, Information technology - Open Systems
Interconnection - Specification of Basic Encoding Rules for
of conformance test cases which check the conformance of
Abstract Syntax Notation One (ASN. 1).
an implementation to the relevant static and/or dynamic con-
(See also CClTT Recommendation X.$$O9 (1988))
formance requirements by controlling and observing protocol La
behaviour. The Abstract Test Methods included in this part of
ISO/IEC 9646-1 : 1991, Information technology - Open Sys-
ISOIIEC 9646 are, in fact, capable of being used to specify
tems Interconnection - Conformance testing methodology
any test case which can be expressed abstractly in terms of
and framework - Part 1: General concepts.
control and observation of Protocol Data Units and Abstract
(See also CClTT Recommendation X.290 (1991))
Service Primitives. Nevertheless, for some protocols, test
ISO/IEC 9646-3 : -'I, Information technology - Open Sys-
cases may be needed which cannot be expressed in these
tems Interconnection - Conformance testing methodology
terms. The specification of such test cases is outside the
and framework - Part 3: The tree and tabular combinednota-
scope of this pari of ISOllEC 9646, although the test cases
tion.
may themselves need to be included in a conformance test-
ing standard.
NOTE - For example, some static conformance requirements related 3 Definitions
to an Application service element may require testing techniques
For the purposes of this part of ISO/IEC 9646, all the defini-
which are specific to that particular Application.
tions given in ISOAEC 9646-1 apply.
1.3 The following are outside the scope of this part of IS01
IEC 9646:
4 Abbreviations
a) the relationship between Abstract Test Suite
For the purposes of this part of ISO/IEC 9646 the abbrevia-
specification and Formal Description Techniques;
tions given in ISOllEC 9646-1 apply. The following abbrevia-
b) testing by means of test methods which are specific to
tions also apply to this part of ISOAEC 9646.
particular applications, protocols or systems, including the
CS : coordinated single-layer (test method)
testing of non-protocol conformance requirements;
CSE : coordinated single-layer embedded (test method)
c) test methods that involve more than two end-systems
communicating together.
DS : distributed single-layer (test method)
NOTE - This part of ISOAEC 9646 applies fully to some but not all
DSE : distributed single-layer embedded (test method)
Physical layer protocols. Nevertheless, many of the concepts apply
FDT : formal description technique
to all protocols.
LS : local single-layer (test method)
2 Normative references LSE : local single-layer embedded (test method)
The following standards contain provisions which, through RS : remote single-layer (test method)
reference in this text, constitute provisions of this part of lSO/
RSE : remote single-layer embedded (test method)
IEC 9646. At the time of publication, the editions indicated
TSS&TP : test suite structure and test purposes
were valid. All standards are subject to revision, and parties
to agreements based on this part of ISO/IEC 9646 are
YL : loop-back (test method)
encouraged to investigate the possibility of applying the most
Yi : transverse (test method)
recent editions of the standards listed below. Members of IEC
1) To be published.
ISOAEC 9646-2 : 1991 (E)
For example, it should be possible to look at a record of PDU
5 Compliance
activity and decide whether or not it is valid with respect to the
5.1 An International Standard or CCITT Recommendation
relevant International Standard or CClïT Recommendation.
which specifies a protocol in compliance with this part of BO/
IEC 9646 shall satisfy all the requirements stated in clauses
6.3 Conformance clauses
6 and 7 plus annex A.
6.3.1 Each International Standard or CClTT Recommenda-
NOTE -Such compliance is a precondlion for the protocol specific&
OS1 protocol or transfer syntax shall
tion which specifies an
tion to be an effective basii for conformance testing of implement&
include a conformance clause, which shall be expressed
tions.
clearly and unambiguously.
5.2 An ATS specification which complies with this part of
6.3.2 Conformance clauses shall distinguish between the fol-
I SO/I EC 9646
lowing categories of information:
a) shall be a conformance test suite;
a) references to clauses which state dynamic
conformance requirements;
b) shall be specified in a test notation standardized by
ISO/IEC or CCITT;
b) static conformance requirements concerning the proto-
c) shall satisfy all the requirements stated in clauses 9 to col implementation;
14 inclusive;
c) static conformance requirements concerning multi-
d) shall be an International Standard or CClTT Recom- layer dependencies.
mendation or, in the absence of such an International
6.3.3 The requirement to produce a PICS, in compliance with
Standard or CClTT Recommendation, shall be a publicly
the PICS proforma, shall be stated separately from the
available document which is in the process of being stand-
requirements on the protocol implementation itself.
ardized within ISOAEC or CCITT, which has the highest
6.3.4 The conformance clause should also include
standardization status available, and which has the status
of at least a Committee Draft, Draft Recommendation or
a) the requirement to be able to accept all correct
equivalent.
sequences of PDUs received from peers, and respond with
correct PDU sequences;
NOTE - ATSs outside the standardization process need to be submit-
ted for international standardization before they can fully comply with
b) the requirement to be able to respond correctly to all
this part of ISO/IEC 9646, in order to ensure that they are subject to
incorrect sequences of PDUs received;
public scrutiny, correction and acceptance, internationally.
c) in connection oriented protocols, the option to support
5.3 It is recommended that the test notation used be TTCN.
either the initiation of a connection or the acceptance of a
If TTCN is used, the ATS shall comply with ISO/IEC 96463.
connection, or both;
NOTE - X.290 (I 988) is considered to be obsolete for this purpose.
d) in connectionless protocols, the option to support the
transmission of a PDU, the receipt of a PDU, or both.
6 Conformance requirements In OS1 Interna-
tional Standards and CCln Recommenda-
7 Requirements on PICS proformas
tions
7.1 The specific requirements to be met by suppliers in
respect of each PICS they provide shall be stated in the rel-
6.1 Introduction
evant International Standard or CClTT Recommendation
The meaning of conformance in OS1 Is discussed in ISO/IEC
which specifies the protocol. The specification of these
9646-1. It is necessary that there be an unambiguous and
requirements shall include a PICS proforma. The PICS pro-
objective understanding of the conformance requirements of
forma shall be found in a separate part of the International
an OS1 protocol or transfer syntax specification, as a prereq-
Standard or CCITT Recommendation which specifies the
uisite to the production of an ATS for that specification.
protocol.
Clauses 6 and 7 state the requirements on protocol specifiers
7.2 The PICS proforma shall be in the form of a question-
to ensure that there is such an understanding of the conform-
naire or checklist to be completed by the supplier or imple-
ance requirements.
mentor of an implementation of the relevant OS1 protocol.
Additional guidance is given in annex B.
7.3 The PICS proforma shall cover all optional and condi-
6.2 General requirements
tional functions, elements of procedure, parameters, options,
PDUs, timers, multi-layer dependencies and other capabili-
6.2.1 A clear distinction shall be made between static and
ties identified in the protocol specification.
dynamic conformance requirements. To avoid ambiguity,
they should be stated separately from one another.
There shall be a well-defined mapping (by references) from
the PICS proforma to the static conformance requirements.
6.2.2 It shall be clear what conformance to the International
Standard or CClTT Recornmendation means, in the sense of
Annex A provides requirements and guidance on the con-
what shall be done, what is permitted but not mandatory, and
struction of PICS proformas.
what shall not be implemented, in order to conform to it.
6.2.3 It shall always be decidable whether an instance of
communication conforms dynamically or not.
ISO/IEC 9646-2 : 1991(E)
d) the specification of a TMP (if applicable) (see clause
8 Abstract Test Suite production process
14).
leading to conformance testing standards
8.4 Clauses 9 to 16 provide requirements and guidance
8.1 In order to present the requirements and general guid-
which relate to each step in the above process.
ance for ATS specification, it is useful to assume a normal
form of the process of ATS production leading to a conform-
ance testing standard. This clause describes the process in
9 Conformance requirements and PICS
just such a normal form. ATS specifiers are not required to
proforma
follow this normal form exactly, however they are recom-
9.1 Before an ATS can be specified, the ATS specifier shall
mended to use a similar process involving the same steps,
first determine what the conformance requirements are for
possibly in a different order.
the relevant protocol and/or transfer syntax specifications
8.2 For the purposes of this part of ISOAEC 9646, the ATS
and what is stated in the PICS proforma concerning the
to be as follows:
production process is assumed
implementation of those specification(s).
a) study the relevant specifications and PICS proformas to
9.2 Clauses 6 and 7 specify the requirements to be met by
determine what the conformance requirements (including
protocol specifiers as a prerequisite to the production of an
to be tested (see clause 9);
options) are which need
ATS for a particular protocol.
b) decide which test groups will be needed to achieve the
9.3 If the static conformance requirements are not properly
appropriate coverage of the conformance requirements
specified, the ATS specifier should contribute to the produc-
0 (see 10.2);
tion of an amendment to or revision of the relevant Interna-
c) optionally develop test group objectives: the common
tional Standard or CCITT Recommendation to clarify the con-
testing objectives of the elements of each test group (see
formance requirements. Pending resolution of the problem,
10.3);
additional guidance for the ATS specifier is given in annex C.
d) develop test purposes which reflect the test group
objectives (if any) of the test groups in which they are con-
10 Test Suite Structure and Test Purposes
tained, and which provide adequate coverage of the con-
formance requirements to be tested (see 10.3 and 10.4);
10.1 Basic requirements
e) optionally, specify generic test cases for each test pur-
10.1.1 The test suite structure and set of test purposes appli-
pose, using some appropriate test notation (seek I);
cable to all ATSs to be specified for the same OS1 protocol
.CIWb
shall be specified in the relevant conformance testing stand-
f) choose the test method(s) for which the complete
abstract test cases need to be specified, and decide what ard, preferably in a separate part.
restrictions need to be placed on the capabilities of the
10.1.2 Each ATS shall comprise a number of test cases,
lower tester and (if appropriate to the chosen test meth-
each of which is designed to achieve one of the specified test
od(s)) the upper tester and test coordination procedures
purposes. The test cases may be grouped into test groups, if
(see clause 12);
necessary nested. The structure shall be hierarchical; thus,
an item at a lower level shall be completely contained within
g) apply a standardized test notation to specify the set of
a higher level item. The structure need not, however, be
abstract test cases, including the test step structure to be
strictly hierarchical: thus, any one test case may occur in
used (see clause 13);
more than one test suite or test group. Similar test groups
h) specify the interrelationships
may occur in more than one higher level test group or test
1) among the test cases,
suite.
2) between the test cases and the PICS, and
10.1.3 The ATS specifier shall ensure that a subset of the test
purposes of each ATS is concerned with capability testing,
3) as far as possible, between the test cases and the
and another subset is concerned with behaviour testing. This
PIXIT,
need not lead to distinct test cases for behaviour and capa-
in order to determine the restrictions on the selection and
bility testing because it may be possible to use the same test
parameterization of test cases for execution, and the
steps for both a behaviour test purpose and for a capability
restrictions, if any, on the orders in which they can be exe-
test purpose. The ATS specifier shall provide an explanation
cuted (see clause 15);
of how the test purposes are derived from or relate to the pro-
i) consider the procedures for maintaining the ATS (see
tocol specification. The ATS specifier shall also provide a
clause 16).
summary of the coverage achieved by the ATS.
8.3 It is also assumed that during the ATS production proc-
10.2 Specification of the test suite structure
ess an overall structure for the conformance testing stand-
10.2.1 In order to ensure that the resulting ATS provides ade-
ard(s) will be developed, with appropriate parts for
quate coverage of the relevant conformance requirements,
a) the Test Suite Structure and Test Purposes (TSS&TP)
the test suite specifier is advised to design the test suite
(see clause 1 O);
structure in terms of nested test groups in a top down man-
b) optionally, a generic test suite (see clause 11);
ner.
c) one or more ATSs (see clause 13) for one or more
12);
Abstract Test Methods (see clause
ISO/IEC 9646-2 : 1991 (E)
There are many ways of structuring the same test suite into - substructured as B.l
test groups; no one way is necessarily right and the best
approach for one test suite may not be appropriate for
another test suite. Nevertheless, the test suite specifier shall
ensure that the test suite includes test cases for whichever of
emanti-
the following categories is relevant:
a) capability tests (for static conformance requirements);
b) behaviour tests of valid behaviour;
c) behaviour tests that investigate the reaction of the IUT
to invalid test events; these may be subdivided into those
concerned with syntactically invalid test events, semanti-
cally invalid test events, and inopportune test events, as
relevant to the protocol concerned;
d) tests focusing on PDUs sent to the IUT;
e) tests focusing on PDUs received from the IUT;
9 tests focusing on interactions between PDUs sent and
PDUs received;
g) tests related to each mandatory capability;
h) tests related to each optional capability;
i) tests related to each protocol phase;
j) variations in the test event occurring in a particular state;
k) timing and timer variations;
I) PDU encoding variations;
m) variations in values of individual parameters;
n) variations in combinations of parameter values.
This list is not exhaustive; additional categories might be
needed to ensure adequate coverage of the relevant con-
formance requirements for a specific test suite. Furthermore,
these categories overlap one another and it is the task of the
test suite specifier to arrange them into an appropriate hierar-
chical structure.
10.2.2 The following structure is an example of a single-layer
test suite, provided for guidance:
IUT
A. Capability tests
- substructured as D.l.l
A.l Mandatory capabilities
0.2 Data transfer phase
A.2 Optional capabilities
B. Behaviour tests: response to valid behaviour by peer
implementation
B. 1 Connection establishment phase (if relevant)
B.l.l Focus on what is sent to the IUT
8.1.1.1 Test event variation in each state
8.1.1.2 Timinghimer variation
B.1.1.3 Encoding variation
B.l .I .4 Individual parameter value variation
B.1.1.5 Combination of parameter values
B.1.2 Focus on what is received from the IUT
- substructured as B.l.l
8.1.3 Focus on interactions
- substructured as 8.1.1
B.2 Data transfer phase
ISOAEC 9646-2 : 1991 (E)
group objectives in this way, a structured set of test purposes NOTE - Test purposes for untestable requirements serve to inform
protd specifiirs which Conformance requirements are untestable,
may be produced.
by inditing gaps in the standardized ATSs.
The test purposes could be produced directly from studying
those clauses in the relevant specification(s) which are
10.4 Coverage
appropriate to the test group concerned. For some test
It is possible to give guidance on the meaning of “adequate”
groups, the test purposes might be derivable directly from the
coverage with reference to the test suite structure example in
protocol state table; for others, they might be derivable from
10.2. In order to express this, a shorthand notation will be
the PDU encoding definitions or the descriptions of particular
used: the letter “x” will represent all appropriate values for the
parameters, or from text which specifies the relevant con-
first digit in the test group identifier, and similarly “y“ for the
formance requirements.
second digit, so that B.x.y.1 stands for 8.1.1.1, 8.1.2.1,
This orderly construction technique helps to ensure the ade-
8.1.3.1, 8.2.1.1, 8.2.2.1, 6.2.3.1, 8.3.1.1, 6.3.2.1 and
quate coverage of the conformance requirements to be
8.3.3.1.
It also avoids unnecessary duplication of text in the
tested.
With this notation, a minimum ”adequate” coverage for the
test purposes, because the full description of each test pur-
as follows:
example given in 10.2 is considered to be
pose does not have to be written explicitly, but can be assem-
a) for capability test groups (A.l, A.2)
bled by tracing a path down through the nested structure of
objectives.
1) at least one test purpose per relevant capability,
NOTE - If the test suite specifier employs a formal description of the
2) at least one test purpose per relevant PDU type and
protocol(s) concerned, test purposes may be derived from that by
each major variation of each such type, using “normal”
means of some automated method. If an automated method is used,
or default values for each parameter;
the same requirements apply. However, methods based on FDTs are
outside the scope of this pari of IÇOIIEC 9646. Nevertheless, if an b) for test groups concerned with test event variation in
FDT is to be used for this purpose, it is preferred that it be a stand-
each state (B.x.y.1, C.x.1.1, D.x.y.l), at least one test pur-
ardized one.
pose per relevant state/event combination;
10.3.2 In order to increase the efficiency of testing individual
c) for test groups concerned with timers and timing
parameters on a single PDU, combined test purposes may be
(B.x.y.2, D.x.y.2), at least one test purpose concerned with
specified for a single abstract test case. Test purposes for
the expiry of each defined timer;
invalid parameter values shall not be combined with other
d) for test groups concerned with encoding variations
test purposes of valid or invalid values.
(B.x.y.3, C.x.1.2, D.x.y.3), at least one test purpose for
10.3.3 As part of the process of designing the TSS&TP, it is
each relevant kind of encoding variation per relevant PDU
suggested that test purposes be identified initially for each
type ;
specific parameter that is to be tested. As a second stage,
e) for test groups concerned with valid individual parame-
combinations of individual parameter test purposes related to
ter values (B.x.y.4, D.x.y.4)
the same PDU may be specified. If this is done
1) for each relevant integer parameter, test purposes
a) a combined test purpose shall be written, combining
concerned with the boundary values and one randomly
and referencing individual test purposes;
selected mid-range value,
b) an indication shall be given that one abstract test case
2) for each relevant bitwise parameter, test purposes for
is to be produced for that combined test purpose, rather
as many values as practical, but not less than all the
than a distinct test case for each of the individual test pur-
“normal” or common values,
poses that have been combined;
3) for other relevant parameters, at least one test pur-
c) the individual test purposes shall remain in the set of
pose concerned with a value different from what is con-
test purposes, but shall reference the appropriate com-
sidered “normal” or default in other test groups;
bined test purpose.
NOTE - Tests for valid parameter values should focus on the rel-
10.3.4 The result of defining and then combining particular
evant claims made in the PICS.
test purposes is a specification of a test suite structure and a
9 for test groups concerned with syntactically or semanti-
list of names of the test cases that shall apply to both the test
cally invalid individual parameter values (C.x.l.3, C.x.2.1)
purposes and to any ATS produced for those test purposes.
1) for each relevant integer parameter, test purposes
10.3.5 Whatever method is used to derive the test purposes,
concerned with invalid values adjacent to the allowed
the test suite specifier should ensure, as far as possible, that
boundary values defined in the protocol specification,
they provide an adequate coverage of the conformance
plus one other randomly selected invalid value,
requirements of the relevant specification(s). There shall be
at least one test purpose related to each distinct conformance
2) for each relevant bitwise parameter, test purposes for
requirement.
as many invalid values as practical,
10.3.6 Test purposes should be specified not only for clearly
3) for all other relevant types of parameter, at least one
testable conformance requirements, but also for conform-
test purpose per parameter;
ance requirements that may be untestable using the test
NOTE -Tests for invalid parameter values should focus on values
methods defined in this part of ISOAEC 9646.
outside the range defined in the relevant protocol specifmtion,
rather than valid values outside the range claimed in the PICS.
ISOAEC 9646-2 : 199l(E)
g) for test groups concerned with combinations of parame-
I I Specification of generic test suites
ter values (B.x.y.5, C.x.1.4, C.x.2.2, D.x.y.5)
A conformance testing standard may include a generic test
1) at least one test purpose for each important combina-
suite as a separate part, particularly if there is an intention to
tion of specific values (e.g. boundary values),
produce more than one ATS.
2) at least one test purpose per set of interrelated
A generic test suite shall consist of one generic test case for
parameters to test a random combination of relevant val-
each test purpose (or specified combination of test pur-
ues.
poses), except for any which are not testable using the test
methods defined in this part of ISOIIEC 9646.
The test suite specifier shall not assume that the test realizer
or test laboratory will perform any checking of test events
NOTE -Thus, a generic test suite has the coverage of the set of all
against the values specified in the PICS other than that possible ATSs for the relevant protd(s). The test purposes exclud-
ed from a generic test suite will be testaMe only by using conform
checking which is specified in the abstract test cases. There-
ance resolution tests, which are not standardized.
fore, the test purposes and abstract test cases shall make
explicit use of values given in the PICS whenever checking of
Each generic test case adds detail to the test purpose. It
valid parameter values is specified. The test suite shall
should specify the main sequences of events of the test body
include test cases to check for the support of parameter val-
and the verdicts to be assigned to the corresponding test out-
ues that are allowed by the protocol specification and are
comes.
within the ranges stated in the PICS. Such test cases shall
Each generic test case specified shall be used as a common
make use of test suite parameters that contain the relevant
root of corresponding abstract test cases for different test
PICS values. The test suite shall also include test cases to
methods.
check for valid reactions to parameter values that are invalid
If a generic test suite is produced in advance of ATSs, then it
with respect to the protocol specification. Parameter values
will be a useful step in the design process. If a generic test
that are valid with respect to the protocol specification(s) but
suite is produced after the production of at least one ATS,
outside the ranges stated in the PICS shall not be tested.
then it will provide a means of relating different test suites to
NOTE - The progression of the work on formal methods in conform-
one another and analysing where there may be gaps in their
ance testing may provide analytical approaches to assess the appro-
coverage.
priate coverage of an ATS, especially for the state/event variations,
as in b) above. This part of IÇO/IEC 9846, however, does not recom-
Guidance on the production of generic test cases is provided
mend any particular analytical approach.
in annex D.
10.5 TSS&TP compliance clause
12 Abstract Test Methods
The TSS&TP part shall include a compliance clause concern-
ing the development of test suites for that TSS&TP. That
12.1 Introduction
clause shall require, as a minimum, that a generic or ATS
complying with the TSS&TP part An Abstract Test Method describes an abstract testing archi-
tecture consisting of a lower tester, upper tester and test
a) consists of a set of test cases corresponding to the set
coordination procedures, and their relationships to the test
or to a subset of the test purposes specified in the TSS&TP
system and SUT. Each ATM determines the PCOs and test
part;
events (i.e. ASPS and PDUs) which shall be used in an
b) uses a test suite structure which is an appropriate sub-
abstract test case for that ATM.
set of the whole of the test suite structure specified in the
Each ATS shall be specified in accordance with one of the
TSS&TP part;
ATMs defined in this clause.
NOTE - The only subsetting of the test suite structure that should
is the omission of test purposes that are untestable in
take place
12.2 General principles
the chosen Abstract Test Method. In particular, for embedded test
12.2.1 Lower Testers
method variants, this will be necessary due to the limitations im
posed by the use of the protocoi(s) above the one that is the focus
In all the ATMs, the lower tester communicates with the SUT
of the test purposes.
via the appropriate underlying service provider. The physical
c) uses the same naming conventions for the test groups
medium is considered to be the service provider underneath
and test cases; the Physical layer.
d) maintains the relationship, if any, specified in the The general specification of the ATMs, given in this clause,
TSS&TP between the test purposes and the entries in the refers to an IUT in which the highest layer is numbered “w
PICS and partial PlXlT proformas to be used for test case
(for “top”) and the lowest is numbered “N; (for “bottom”). For
selection;
single-protocol IUTs, 4 is equal to N,. The same notation is
e) complies with this part of ISO/IEC 9646 and with ISO/ used to refer to layers within the SUT and within the lower
IEC 9646-3.
tester. The SUT may implement protocols in layers lower
of interest in the test method
than “Nb, but these are not
descriptions. Nevertheless, the SUT shall include the Physi-
cal layer. For all test methods, ATSs specify test events at the
lower tester PCO in terms of (Nb-l)-ASPs and/or (Nb) to (NJ-
PDUs.
ISO/IEC 96462 : 199l(E)
12.2.2 Upper Testers d) there shall be a mapping between the relevant ASPs
and their realization at the upper interface of the IUT;
The main difference between the ATMs is in the nature of the
upper tester and its coordination with the lower tester. e) the upper tester is located within the SUT;
In some test methods, a second PCO is employed, for the 9 the requirements for the test coordination procedures
upper tester. In these test methods, the definition of the test shall be specified in the ATSs, although the procedures
events at the PCO for the upper tester shall be specified in themselves shall not be;
accordance with the appropriate OS1 service definition and
g) if the upper interface of the IUT is a human user inter-
OS1 protocol specification. The activity at the upper tester
face, then the human operator of the SUT fulfils the
PCO shall not require that the SUT or IUT support ASP
requirements of the TCPs;
parameters, PDUs or capabilities that are not part of an OS1
h) if the upper interface is a standardized programming
International Standard or CCITT Recommendation.
language interface, then the upper tester is realized in soft-
If the PCO is at a humanly accessible interface, the SUT’S
ware and the upper and lower testers together fulfil the
user interface shall serve as the PCO.
requirements of the TCPs.
This test method is illustrated in figure 2.
12.3 General specification of the ATMs for end-
system IUTs
ATSs for the Distributed test method shall not themselves
specify an upper tester interface.
12.3.1 Introduction
In order to avoid placing requirements on the internal design
For IUTs within end-system SUTs there are four categories of
of SUTs, the ATSs shall not require that a programming lan-
ATMs: Local, Distributed, Coordinated and Remote.
guage interface is standardized for the sole purpose of test-
12.3.2 The Local test method
ing.
Abbreviation: L
NOTE - In the Application layer, until application programming inter-
faces are standardized to provide a common means of access to OS1
In this test method
Application services, the use of the Distributed test method is in prac-
a) the test events at the lower tester PCO are specified
tice limited to using human user interfaces to OS1 applications (e.g.
only in terms of (Nb-1)-ASPS and/or (Nb) to (NJ-PDUs;
File Transfer Access and Management initiators).
b) the test events at the upper tester PCO are specified in
12.3.4 The Coordinated test method
terms of (&)-ASPS;
Abbreviation: C
c) the upper service boundary of the IUT shall be a stand-
In this test method
ardized hardware interface which can be used for testing
a) the test events at the lower tester PCO are specified in
purposes; the test suites shall not place any requirements
terms of (Nb-1)-ASPS, and/or (Nb) to (w-PDUç plus TM-
on the realization of the interface in the SUT, additional to
those in the standardized hardware interface Specification; PDUs;
d) the specification of the hardware upper interface of the b) (Nd-ASPS are not used in the specification of the ATS;
IUT shall define the mapping between the relevant ASPs
no assumption is made about the existence of an upper
and/or PDUs and their realization at the interface;
service boundary of the
...