ISO/IEC 9646-5:1991

ISOllEC
I NTE R NAT1 O NA L
STANDARD
First edition
1991 -07-15
Information technology - Open Systems
interconnection - Conformance testing
methodology and framework -
Part 5:
Requirements on test laboratories and clients for
the conformance assessrnenf. process
Technologies de l'information - Interconnexion de systèmes ouverts -
Cadre général et méthodologie des tests de conformitë OS1 -
Partie 5: Spécifications pour laboratoires d'essais et clients pour le
procédé dévaluation de conformité

ISOll EC 9646-5: 1991 (E)
Con tents Page
Foreword . iv
lntroduction . v
1 Scope . 1
2 Normative references . 1
3 Definitions . 1
4 Abbreviations . 2
4.1 Abbreviations defined in ISOAEC 9646-1 . 2
4.2 Other abbreviations . 2
5 Overview of the Conformance Assessment Process . 2
5.1 Introduction . 2
5.2 Preparation for testing . 2
5.3 Test operations . 4
5.4 Test report production . 4
6 Preparation for testing . 4
6.1 Introduction . 4
6.2 Requirements for SUT testability . 4
6.2.1 Client role . 4
Communication between the test laboratory and the client . 6
6.3
6.3.1 Test Laboratory and Client Checklists . 6
6.3.2 Agreement on test methods and selection of test suites .
6.3.3 Management of technical issues . 7
6.4 Documentation for conformance assessment . 7
6.4.1 Overview . 7
6.4.2 Protocol Implementation Conformance Statement (PICS) . 8
6.4.3 Protocol Implementation extra Information for Testing (PIXIT) . 8
6.4.4 System Conformance Statement (SCS) . 8
6.4.5 TMP Implementation Statement . 8
7 lest operations . 9
7.1 Introduction . 9
7.2 Static Conformance Review . 9
7.2.1 Test laboratory role . 9
7.2.2 Client role . 9
7.2.3 Mutual role . 9
@ ISOAEC 1991
All rights reserved . Unless otherwise specified. no part of this publication may
be reproduced or utilized in any form or by any means. electronic or mechanical.
including photocopying and microfilm. without permission in writing from the publisher .
ISOAEC Copyright Office Case Postale 56 CH-1211 Genève 20 Switzerland
Printed in Switzerland
ii
ISOllEC 9646-5: 1991 (E)
7.3 Test selection . 9
7.3.1 Test laboratory role . 9
7.3.2 Client role . 9
7.4 Test parameterization . 11
7.4.1 Test laboratory role . 11
7.4.2 Client role . 11
7.5 Test Coordination Procedure verification . 11
7.5.1 Introduction . 11
7.5.2 Mutual role . 11
7.6 Test Campaign . 11
7.6.1 Introduction . 11
7.6.2 Test laboratory role . 11
7.6.3 Client role . 12
7.6.4 Mutual role . 12
7.7 Negotiated Exits during the Test Campaign . 12
7.7.1 Use of a Negotiated Exit . 12
7.7.2 Test laboratory role . 13
7.7.3 Client role . 13
7.7.4 Mutual role . 13
8 Test report production . 13
8.1 Conformance test reports . 13
8.1.1 Introduction . 13
8.2 System Conformance Test Report (SCTR) . 13
8.2.1 Test laboratory role . 13
8.2.2 Client role . 13
8.2.3 Mutual role . 13
8.3 Protocol Conformance Test Report (PCTR) . 13
8.3.1 Test laboratory role . 13
8.3.2 Client role . 14
8.3.3 Mutual role . 14
9 Compliance . 14
9.1 Test laboratory role . 14
9.2 Client role . 14
Annexes
A System Conformance Test Report (SCTR) . 15
A.l Introduction . 15
A.2 SCTR Proforma . 15
B Protocol Conformance Test Report (PCTR) . 18
B.1 Introduction . 18
8.2 PCTR Proforma . 18
C Skeleton PlXlT Proforma . 21
C.l Introduction . 21
C.2 PlXlT Proforma . 21
D Guidance for a PlXlT . 24
E Summary of Conformance Testing documents . 25
E.l Abbreviations in this annex . 25
E.2 Use of Conformance Testing documents by the client and test laboratory 25
Figures
1 Overview of the Conformance Assessment Process . 3
2 Preparation for Testing . 5
3 Typical Test Operations . 10

ISOll EC 9646-5: 1991 (E)
Foreword
IS0 (the International Organization for Standardization) and IEC (the
International Electrotechnical Commission) form the specialized system
for worldwide standardization. National bodies that are members of IS0
or IEC participate in the development of International Standards through
technical committees established by the respective organization to deal
with particular fields of technical activity. IS0 and IEC technical com-
mittees collaborate in fields of mutual interest. Other international or-
ganizations, governmental and non-governmental, in liaison with IS0
and IEC, also take part in the work.
In the field of information technology, IS0 and IEC have established a
joint technical committee, ISO/IEC JTC 1. Draft International Standards
adopted by the joint technical committee are circulated to national bod-
ies for voting. Publication as an Internaiional Standard requires ap-
proval by at least 75 % of the national bodies casting a vote.
International Standard ISO/IEC 9646-5 was prepared by Joint Technical
Committee ISO/IEC JTC 1, Information technology.
ISO/IEC 9646 consists of the following parts, under the general title In-
formation technology - Open Systems Interconnection - Conforn?ance
testing methodology and framework:
- Part 1: General concepts
- Part 2: Abstract test suite specification
- Part 3: The Tree and Tabular Combined Notation (TTCN)
- Part 4: Test realization
- Part 5: Requirements on test laboratories and clients for the con-
formance assessment process
Annexes A, 6 and C form an integral part of this part of ISO/IEC 9646.
Annexes D and E are for information only.

ISO/I EC 9646-5: 1991 (E)
Introduction
Conformance testing requires mutual understanding and agreement between the
test laboratory and the client. This part of ISOAEC 9646 addresses the roles of both
the test laboratory and the client during the conformance assessment process, the
need to reach mutual agreements between them, and the requirements on each of
them.
The conformance assessment process is the most visible process of conformance
testing, where the results of test suite standardization are put to real use. This is
also the stage at which there is potentially the most scope for variations to occur. As
this part of ISO/IEC 9646 is concerned with the relatively formal process of testing
implementations, it is important that the number and nature of such variations is
very limited. One of the major objectives of standardizing the conformance testing
process is to achieve an acceptable and useful degree of comparability of results of
conformance assessments of similar implementations. If this is to be achieved, not
only should the same source of tests be used (i.e. as specified in standards), but
also the methods of selecting and parameterizing these tests, and presenting their
results, should be, to a large extent, the same.
This part of ISOAEC 9646 addresses the issues which should be taken into account,
by both the test laboratory and the client, if the necessaryconsistencyof conformance
assessment is to be achieved. The target audiences for this part of ISO/IEC 9646
are the test laboratories and their clients.
The test laboratory is responsible for conducting the conformance assessment of an
OS1 implementation at the request of a client. Typically, test laboratories are
a) organizations developing or supplying OS1 implementations (first-party test
laboratories);
b) organizations willing to verify OS1 implementations themselves before using
them (second-party test laboratories);
c) organizations, independent of suppliers or users of OS1 implementations,
whose business is the testing of such implementations (third-party test laborato-
ries).
Clients may be implementors or suppliers of real open systems or other OS1 systems,
who are applying for their own implementations to be tested. Alternatively, they may
be procurers of those implementations, or any other interested party. The applicability
of this pari of ISOAEC 9646 is independent of the relationship between the client
and the implementation. During the conformance assessment process, the client is
responsible for the conformance statements accompanying the System Under Test
(SUT) and for the configuration of the SUT.
Secondary, but related, audiences to whom this part of ISOAEC 9646 could also be
of interest are
a) test realizers;
b) organizations responsible for the accreditation of first- , second- or third-party
test laboratories;
ISOAEC 9646-5: 1991 (E)
c) organizations responsible for the issue of test certificates which are based
upon the conformance test reports issued by test laboratories;
d) readers of conformance test reports.
Within this part of ISOAEC 9646, the conformance assessment process relating to
both the test laboratory and the client is divided into three phases:
a) preparation for testing;
b) test operation;
c) production of test reports.
An overview of these three phases is given in clause 5. Clauses 6 to 9 state
requirements on how to conduct these three phases.
For the purposes of this part of iSO/iEC 9646, it is assumed that a test laboratory
to provide a conformance assessment service.
is available and is already organized
The test laboratory is assumed to have acquired from a test realizer (whether or not
the latter belongs to the same organization) Means of Testing IUTs, for one or more
OS1 protocols and according to one or more Abstract Test Methods (ATM)s. This
part of ISOiIEC 9646 specifies requirements on the test laboratory with respect to
of the conformance assessment process for a particular client.
the conduct
Similarly, it is assumed that a client is ready to apply for conformance assessment
of an OS1 implementation. The client is assumed to be familiar with the appropriate
standards, with the concepts of conformance testing and Abstract Test Methods, and
to be ready to cooperate with the test laboratory. This part of ISOAEC 9646 specifies
requirements on the client with respect to both the testability of the proposed SUT
and the conduct of the conformance assessment process.
is recommended that this part of ISO/IEC 9646 be read in conjunction with ISO/IEC
It
9646-1.
of ISOAEC 9646 is also to be published as CCiTT Recommendation X.294
This part
(1991).
vi
INTERNATIONAL STANDARD ISO/IEC 9646-5: 1991 (E)
Information technology - Open Systems Interconnection -
Conformance testing methodology and framework -
Part 5:
Requirements on test laboratories and clients for the
conformance assessment process
1 Scope c) accreditation of test laboratories;
This part of ISOAEC 9646 specifies requirements on both d) certification of implementations of OS1 protocols.
the test laboratory and the client, for the conduct of the con-
formance assessment process. The requirements are those
necessary to achieve comparability of results of tests on sim-
ilar implementations performed by different test laboratories.
2 Normative references
This part of ISOAEC 9646 also provides some guidance on
the conformance assessment process.
The following standards contain provisions which, through
reference in this text, constitute provisions of this part of
The requirements include
ISO/IEC 9646. At the time of publication, the editions indi-
cated were valid. All standards are subject to revision, and
to agreements based on this part of ISO/IEC 9646 are
a) requirements for the testability of the implementation parties
with respect to ATMs; encouraged to investigate the possibility of applying the most
recent editions of the standards indicated below. Members of
IS0 maintain registers of currently valid International
IEC and
b) general requirements on the test laboratory and the
Standards.
client applicable to any conformance testing process;
ISOAEC 7498 : 1984, Information processing systems -
c) exchange of technical and administrative information,
Open systems interconnection - Basic Reference Model.
including a System Conformance Statement (SCS), a Pro-
tocol Implementation Conformance Statement (PICS) for
See also CCITT Recommendation X.200).
each relevant OS1 protocol standard, and Protocol Imple-
mentation extra Information for Testing (PIXIT) for each
ISO/TR 8509 : 1987, Information processing systems -
Abstract Test Suite (ATS) to be used for testing;
Open systems interconnection - Service conventions.
d) cooperation between the test laboratory and the client
(See also CCITT Recommendation X.21 O).
to reach an agreement on the definition of the Implemen-
tation Under Test (IUT), on the ATMs and ATSs to be used
ISOAEC 9646-1 : 1991, Information technology - Open Sys-
and on the conditions under which testing will be performed;
tems Interconnection - Conformance Testing Methodology
and Framework - Part 1: General concepts.
e) requirements for the structure and content of the con-
formance test reports that document the results of confor-
(See also CClTT Recommendation X.290(1991)).
l @
mance assessment.
ISOAEC 9646-2 : 1991, Information technology - Open Sys-
This part of ISOiIEC 9646 is applicable equally to those test tems Interconnection - Conformance Testing Methodology
laboratories which are affiliated to suppliers or procurers, and and Framework - Part 2: Abstract Test Suite Specification.
those which are independent.
(See also CClTT Recommendation X.291(1991)).
This part of ISOAEC 9646 is applicable to conformance as-
sessment of implementations of OS1 and ISDN two-party pro-
ISOiIEC 9646-3l), Information technology - Open Systems
tocol standards that comply with the relevant requirements for
Interconnection - Conformance Testing Methodology and
testability in ISOAEC 9646-2, based on conformance testing
Framework - Part3: The Tree and Tabular Combined Notation
standards specified in compliance with ISO/IEC 9646-2, and (TTCN).
using Means of Testing (MOT) in compliance with ISOAEC
9646-4. ISOAEC 9646-4: 1991, Information technology - Open Sys-
tems Interconnection - Conformance Testing Methodology
The following are outside the scope of this part of ISOAEC and Framework - Part 4: Test Realization.
9646:
(See also CCITT Recommendation X.293(1991)).
a) the production of diagnostic trace information, addi-
tional to that in the conformance log, for the results of
testing performed by the test laboratory, and its supply to
the client;
3 Definitions
b) aspects of test laboratory operations which are not For the purposes of this part of ISOAEC 9646, all the defini-
specific to testing implementations of OS1 protocols;
tions given in ISOAEC 9646-1 apply. The definitions in this
')To be published.
ISOll EC 9646-5 : 1 991 (E)
PICS : protocol implementation conformance statement
clause also apply to this part of ISO/IEC 9646.
PIXIT : protocol implementation extra information for testing
3.1 client test manager: The person identified by the
SAP : service access point
client organization as being responsible for all matters re-
lating to the conformance testing of the IUT.
SATS : selected abstract test suite
SCS : system conformance statement
3.2 test laboratory manager: The person identified by
the test laboratory as being responsible for all matters relating
SCTR : system conformance test report
to test laboratory operations.
SUT : system under test
3.3 test operator: The person or persons designated by
TMP : test management protocol
the test laboratory as being responsible for running confor-
mance tests against the IUT.
4.2 Other abbreviations
3.4 SUT operator: The person or persons designated by
the client organization as being responsible for operation of
For the purposes of this part of ISO/IEC 9646, the following
the SUT during conformance testing.
abbreviation also applies:
e
NE : negotiated exit
3.5 negotiated exit [NE] (from the conformance assess-
ment process): A point in time at which the test laboratory
and the client can mutually decide to terminate the confor-
mance assessment process.
5 Overview of the Conformance Assessment
Process
3.6 test laboratory checklist: A record of test-related in-
formation supplied to the client by the test laboratory during
5.1 Introduction
the test preparation phase of the conformance assessment
process.
Figure 1 illustrates the conformance assessment process.
ISOAEC 9646-1: 1990, 6.3 provides an overview of the
3.7 client checklist: A record of test-related information conformance assessment process. An overview of the three
phases (preparation for testing, test operations and test report
supplied to the test laboratory by the client during the test
production) is given as guidance in this clause.
preparation phase of the conformance assessment process.
5.2 Preparation for testing
4 Abbreviations
The preparatory phase includes
4.1 Abbreviations defined in ISO/IEC 9646-1
For the purposes of this part of ISOAEC 9646. the followina
a) general administrative steps, such as the application
'
abbreviations defined in ISOAEC 9646-1 : 1990, ciause 4 by the client, the provision of documents by the test labo-
ratory describing the general policy, terms and conditions
apply:
to be followed during test operations, and the provision of
information about the SUT by the client;
ASP : abstract service primitive
ATM : abstract test method
b) checking the completeness of the documents provided
by the test laboratory (PIXIT proforma), of those provided
ATS : abstract test suite
by the client (PICS, PIXIT, SCS), and also of the checklists
exchanged between the test laboratory and the client;
BIT : basic interconnection test
c) analysis of the SUT configuration and choice of a con-
IUT : implementation under test
formance testing standard(s) for the OS1 protocol(s) to be
tested, either agreeing that the SUT and the test labora-
MOT : means of testing
tory's MOT are both capable of supporting the ATM, or
using a negotiated exit if an agreement cannot be reached;
OS1 : open systems interconnection
d) preparation of the SUT and the MOT for the testing
PCO : point of control and observation
configuration that results from the choice of an ATM.
PCTR : protocol conformance test report
Requirements for test preparation, on both the test laboratory
PETS : parameterized executable test suite and the client, are given in clause 6.
ISO/IEC 9646-5: 1991 (E)
Start Assessment
Preparation for Testing
[T] ATM(s) & ATS(ç)
C
TEST OPERATIONS
-
Static Conformance Review
Test SelectioniParameterization
Results
-0
I
Test Reporl Production
Key
- = control flow I
- = information flow
Finish Assessment
W
Figure 1 - Overview of the Conformance Assessment Process
ISOllEC 9646-5: 1991 (E)
6.2 Requirements for SUT testability
5.3 Test operations
During the second phase, the test operations are carried out. 6.2.1 Client role
These include
6.2.1.1 General
a) the static conformance review, during which detailed
analysis of the PICS and PlXlT takes place;
The client shall ensure that the SUT is testable using at
least one Abstract Test Method (ATM) as defined in ISOiIEC
b) test selection and parameterization, applied to the 9646-2.
executable (or abstract) test suite; this determines the
Parameterized Executable Test Suite (PETS) that will be NOTE - This part of ISOAEC 9646 does not constrain the client to
agree to any particular ATM, as long as one method is made possible
executed;
by appropriately configuring the SUT.
c) one or more test campaigns, running:
Each of the ATMs described in ISOAEC 9646-2: 1990,
clause 12, imposes particular requirements on the SUT with
1 ) basic interconnection tests (optional);
respect to its testability. The requirements vary according to
the test method.
2) capability tests;
For the ATM by which the client claims the SUT can be tested,
3) behaviour tests.
the client shall ensure that the SUT provides the necessary
means of control and observation and that it can enable the
If difficulties are encountered during test operations, it is pos-
appropriate test coordination procedures to be performed.
sible for the client and test laboratory to negotiate a repetition
of a test campaign as a whole or in part. Alternatively, they
The next four subclauses state further requirements for SUT
can take a negotiated exit from the conformance assessment
testability for each of the ATMs.
process.
NOTE - The reasons for the negotiated exit are documented in an
informal test report. 6.2.1.2 Local test method
Requirements for test operations, on both the test laboratory
6.2.1.2.1 For the Local ATM, the client shall ensure that
and the client, are given in clause 7.
the IUT upper interface is realized in hardware and can be
connected to a test system.
NOTE -The only requirements are that the interface is standardized
5.4 Test report production
and that there is a well-defined mapping between the relevant ASPs
and the hardware interface.
After test operations are complete, the third phase com-
mences: an evaluation of the conformance of the OS1 imple-
6.2.1.2.2 There are no requirements on the client for the test
mentation is made. This evaluation is recorded in System
coordination procedures.
and Protocol Conformance Test Reports, the proformas for
which are specified in annexes A and B respectively.
6.2.1.3 Distributed test method
Requirements for test report production are given in clause 8.
6.2.1.3.1 For the Distributed ATM, the client shall ensure that
the SUT contains the means of generating, and indicating the
6 Preparation for testing
receipt of, the ASPs for the relevant Points of Control and
Observation (PCOs), that are appropriate to the envisaged
6.1 Introduction IUT.
NOTES
Clause 6 specifies requirements for test preparation on both
the test laboratory and the client. Figure 2 illustrates the
1 The only requirements are that means be realized within the SUT
preparatory phase of the conformance assessment process.
for the control and observation of the effects of the relevant ASPs and
During this phase, both parties ensure that the required doc-
that the upper service boundary of the IUT is either a human-user
umentation (including SCS, PICS, PlXlT and the checklists)
interface or a standardized programming language interface.
is completed to their mutual satisfaction: in particular, the
characteristics of the SUT which determine its configurations
2 Any realization will satisfy this requirement provided that the
and affect the choice of test method must have been precisely
appropriate ASPs can be generated and detected unambiguously.
defined. It is assumed that the requirements for SUT testa-
Example realizations are: pushing buttons, observing and controlling
bility have been met by the client before approaching the test
activities of OS1 protocols which use the service primitives concerned,
laboratory.
observing activity on peripherals and so on. There may be a one-
to-one, one-to-many, or many-to-one correspondence between the
means within the SUT and ASPs.
As a prerequisite to test operations, the client and the test
laboratory agree on the ATM and the conditions for the test
6.2.1.3.2 The client shall ensure that notification of the occur-
campaign. If an agreement is reached, the test laboratory will
select the MOT for the chosen test method, and proceed to rence of ASP events within the SUT can be communicated by
the SUT Operator to the Test Operator (at the test laboratory)
the test operations phase; otherwise, a negotiated exit may
when required by the test coordination procedures.
be taken.
ISO/IEC 9646-5: 1991 (E)
Client Application
I
SUT
L c Configuration
Client completes
- and
documents
Testability
c
-
T
Client and
Client Checklist Test Laboratory Test Laboratory
SCS (l), PICS (n) agree on
Checklist
ATM(s) and ATS(s)
-
-
client
Test laboratory
PlXlT Proforma (n)
Client identifies MOT
completes and completes
PlXlT proforma(s)
PlXlT (n)
e
MOT
Preearatio n
Key . .
I
- = control flow
Test operations (refer to Figure 3)
-+. = information flow
Figure 2 - Preparation for Testing
O = decision point
ISO/IEC 9646-5: 1991 (E)
NOTE - The means for transferring notification of the occurrence of 6.3 Communication between the test laboratory
generated and observed ASP events to the test laboratory is outside
and the client
the scope of ISOAEC 9646, but could be achieved by the use of a
separate communication channel, e.g. voice, telephone call, data
6.3.1 Test Laboratory and Client Checklists
call, etc. The way in which this communication takes place constitutes
part of the informal test coordination procedures, and is stated by the
6.3.1.1 introduction
client in the Client Checklist when applying to the test laboratory for
conformance assessment.
During test preparation, the test laboratory and client ex-
change test-related information in order that they can agree
on the definition of the IUT and on the choice of ATM(s) and
ATS(s) to be used during testing. This information, some of
which may be assembled as a result of discussion, is spec-
ified in a Test Laboratory Checklist and a Client Checklist.
Based on the exchange and review of the checklists by both
6.2.1.4 Coordinated test method
parties, an agreement can be reached on whether or not to
proceed with the test preparation phase. An agreement might
not be reached if the test laboratory is unable to offer a testing
6.2.1.4.1 For the Coordinated ATM, the client shall ensure
service which is compatible with the client’s proposed IUT. If
that the SUT can support at least one upper tester that is an
both parties agree to continue, the documents required for
implementation of a standardized Test Management Protocol
conformance assessment are prepared (refer to 6.4).
(TMP) appropriate to the envisaged IUT.
NOTE -This requirement does not imply the implementation within
the SUT of a real OS1 service boundary with real service primitives.
6.3.1.2 Test laboratory role
6.2.1.4.2 Once an ATS for the Coordinated ATM has been
During the test preparation phase of the conformance as-
chosen during test preparation, the client shall ensure that
sessment process, the test laboratory shall provide the client
the SUT supports an upper tester which realizes the TMP for
with a Test Laboratory Checklist, containing as a minimum
this ATS.
the following information:
NOTE - The client may use the assistance of the test laboratory
a) requirements placed on the client by the test laboratory,
during test preparation in order to meet this requirement. Alterna-
concerning the provision of the SCS, PICS(s) and PIXIT(s);
tively, the client may choose to implement the upper tester and TMP
for a particular ATS within the envisaged IUT before approaching a
test laboratory for conformance assessment. b) statement of compliance with ISOAEC 9646-5;
c) ATM(s) supported for each OS1 protocol for which a
testing service is offered;
d) statement of conformance with the conformance test-
ing standards for which a testing service is offered;
6.2.1.5 Remote test method
e) statement of whether or not the test laboratory offers a
comprehensive conformance testing service as defined in
each of the applicable conformance testing standards and
6.2.1 5.1 For the Remote ATM, once an ATS is chosen during
test preparation, the client shall document in the PIXIT the in ISOAEC 9646-2;
e
degree of control and observation that is possible within the
SUT to achieve the test coordination procedures that are
f) limitations, if any, of the lower tester(s), for the sup-
stated informally in the standardized ATS. ported test methods that are relevant to the client’s SUT;
NOTE - Of all the test methods, the Remote ATM imposes the
g) specifications of the upper tester, if applicable, and the
fewest constraints on the SUT The SUT, however, is expected to
test coordination procedures for the supported test methods
perform in accordance with the claims made in the PIXIT. These
that are relevant to the client’s SUT;
claims may imply some degree of control, for example, where the
SUT is required to initiate some event.
h) description of the test laboratory procedures which are
related to running the tests during the test campaign and
6.2.1.5.2 For those test events for which a claim of degree
which are relevant to the client, particularly those to be
of control or observation has been made in the PIXIT, the
performed by the SUT Operator;
client shall ensure that notification of the occurrence of such
test events within the SUTcan be communicated by the SUT
i) references to all documentation to be used and pro-
Operator to the Test Operator (at the test laboratory), when
duced by the test laboratory during the conformance as.
required by the test coordination procedures.
sessment process;
- The means for transferring notification of the occurrence of
NOTE
NOTE - The test laboratory may provide the client with
generated and observed test events to the test laboratory is outside
the scope of ISO/IEC 9646, but could be achieved by the use of a
separate communication channel, e.g. voice, telephone call, data a) Information on the testing services offered in the range of OS1
protocols of interest to the client;
call, erc. The way in which this communication takes place constitutes
part of the informal test coordination procedures, and is stated by the
client in the Client Checklist when applying to the test laboratory for b) Assistance in the implementation of an upper tester, if appli-
conformance assessment. cable for the chosen ATM;
ISO/IEC 9646-5: 1991 (E)
Information on whom to contact to obtain PIXIT proformas and NOTES
c)
any other necessary information;
1 The client may wish to select ATM(s) which impose no additional
requirements on the SUT other than those contained in the OS1
d) Estimates of the time required to complete the Test Oper-
ations and Test Report Production phases of the conformance standards to which the SUT claims to conform. In this case, the
assessment process; client should select a test laboratory which provides a comprehensive
testing service. (See ISO/IEC 9646-2: 1990, 11.7.2 and clause 9.1
of this part of ISO/IEC 9646).
e) Statement of test laboratory accreditation (if applicable).
2 An IUT can consist of a single protocol entity or multiple protocol
entities. If the IUT is multi-protocol, then single-layer embedded test
6.3.1.3 Client role
methods should be used incrementally. (Refer to ISO/IEC 9646-1:
1990, 7.6).
During the test preparation phase of the conformance as-
sessment process, the client shall provide the test laboratory
3 When using embedded single-layer test methods, the ATS is
with a Client Checklist, containing as a minimum the following
targetted at a single OS1 protocol.
information:
4 Several IUTs can be defined in the SUT if several conformance
a) statement of compliance with ISO/IEC 9646-5;
assessments are to be performed on the same SUT for different
combinations of protocols and test methods.
b) specification of what part of the SUT is proposed to be
the IUT and which protocols are to be tested;
6.3.2.3 Mutual role
c) a claim of SUT testability based on specific ATM(s)
and/or conformance testing standards;
After both parties have reviewed the information provided
in the checklists, in order that the conformance assessment
d) a statement of the test coordination procedures that process can continue, both parties shall agree on
are suitable for use with this IUT, and which correspond to
the proposed ATM(s).
a) the accuracy and sufficiency of the information pro-
vided in the checklists;
NOTE - The client may provide the test laboratory with
b) the definition of the IUT;
a) Information on any physical requirements for the SUT (for
example, space, air conditioning, etc.) if relevant, and any other
c) the ATM(s) and the corresponding ATS(s) which will be
practical information that may be needed during the conformance
used for the conformance assessment process.
assessment process;
If an agreement cannot be reached, a negotiated exit should
Information on whom to contact during the conformance as-
b)
be taken in order to terminate the conformance assessment
sessment process
process. If an agreement is reached, what has been agreed
shall be recorded in the System Conformance Test Report
(SCTR) issued at the end of the conformance assessment
6.3.2 Agreement on test methods and selection of test
process.
suites
6.3.2.1 Test laboratory role
6.3.3 Management of technical issues
The test laboratory shall review the Client Checklist and
There are no general requirements concerning procedures
shall determine if the test laboratory offers a testing service
for the resolution of technical issues between the client and
which is applicable to the client’s proposed IUT. The test
test laboratory that may arise during the conformance assess-
laboratory shall accommodate the client’s choice of ATM
ment process. However, should differences be discovered
for each protocol in the proposed IUT (see 6.3.2.2) and
between the conformance testing standard and the protocol
shall select the corresponding reference standardized ATS
standard, the protocol standard shall have precedence in
(including if relevant the TMP specification) to be used in the
problem resolution.
conformance assessment process.
NOTE - Unresolved issues of a technical nature, related to the
For each standardized ATS selected, the test laboratory shall
interpretation of relevant standards, can be referred to the appropriate
identify and use a MOT which complies with that standardized
OS1 standard defining group.
ATS and with ISOAEC 9646-4.
NOTE -The conformance testing standard(s) used should have the
6.4 Documentation for conformance assessment
highest available standardization status (see ISO/IEC 9646-2: 1990,
clause 5).
6.4.1 Overview
6.3.2.2 Client role After the test laboratory and client have agreed on the defi-
nition of the IUT and on the ATM(s) and ATS(s) to be used
The client shall review the Test Laboratory Checklist and during the conformance assessment, they exchange detailed
shall make the choice of which ATM is to be used for each information about the SUT. This information resides in four
protocol in the proposed IUT in accordance with the claims documents related to test preparation: the PICS, PIXIT,
for SUT testability and the testing service offered by the test SCS and TMP Implementation Statement. The next four
laboratory. subclauses state requirements on both the client and test
ISOll EC 9646-5: 1991 (E)
NOTE - There is one standardized ATS per protocol for single-
laboratory related to the production and exchange of these
layer and single-layer embedded test methods. Thus, for these test
documents.
methods, there is one PlXlT proforma per protocol.
6.4.2 Protocol Implementation Conformance Statement
6.4.3.3 Client role
(PICS)
6.4.2.1 Content of the PICS The client shall provide a PlXlT for each ATS to be used for
testing, by completing the relevant PlXlT proforma provided
by the test laboratory.
Detailed information on the role and scope of the PICS is
given in ISOAEC 9646-1: 1990, 5.4 and basic guidance on
NOTE - Wherever the information requested is extensive, the
the design of the PICS proforma is given in ISOAEC 9646-2:
PlXlT may refer to other documents which contain the necessary
1990, annex C.
information.
A PICS proforma is contained in each OS1 protocol standard
and in each OS1 transfer syntax standard which complies with
6.4.4 System Conformance Statement (SCS)
the requirements of ISOAEC 9646-2 for testability.
6.4.4.1 Content of the SCS
6.4.2.2 Test laboratory role
The role and scope of the SCS is mentioned briefly in ISOAEC
9646-1 : 1990,5.4. The format of the SCS is not standardized.
There is no requirement on the test laboratory for the provision
However, the SCS contains the following information as a
of PICS proformas for use by the client. However, the test
minimum:
laboratory may provide copies of the relevant PICS proformas
if necessary.
a) information related to both the SUT and the client:
6.4.2.3 Client role
1) administrative information to identify the client;
The client shall provide a PICS for each OS1 protocol standard
2) system information to identify the SUT, for example,
(and for each OS1 transfer syntax standard if applicable) which name and version number;
is implemented in the IUT and for which conformance is to be
tested.
b) information related to the protocols in the SUTfor which
a PICS is provided:
The client shall complete the relevant PICS proformas from
the relevant OS1 standards. The requirements for the provi-
1) the identification of the OS1 protocol (and OS1 trans-
sion of PICS information are stated in the relevant standards.
fer syntax) standards, including version numbers;
2) a reference to the related PICS.
6.4.3 Protocol Implementation extra Information for
Testing (PIXIT)
The SCS optionally contains an indication of whether an
SCTR and a PCTR, received from a previous conformance
6.4.3.1 Content of the PlXlT
assessment, are available for information.
The role and scope of the PlXlT is given in ISOAEC 9646-
1: 1990, 6.2. Requirements and further guidance on the
6.4.4.2 Test laboratory role
structure and content of the PlXlT is given in annexes C and
D of this part of ISOAEC 9646.
There is no requirement on the test laboratory for the provision
of an SCS proforma. The SCS provides test-independent
information about the SUT.
6.4.3.2 Test laboratory role
The test laboratory shall produce a PlXlT proforma for each
6.4.4.3 Client role
standardized ATS for which testing is offered, in accordance
with the skeleton proforma specified in annex C.
The client shall provide to the test laboratory an SCS as
described in 6.4.4.1.
Each PlXlT proforma shall include the augmented partial
PlXlT proforma accompanying the MOT and associated with
the relevant reference standardized ATS. It shall also contain
6.4.5 TMP Implementation Statement
the questions necessary to elicit all the information concerning
the SUT, the IUT, and the relevant ATS parameters, which
6.4.5.1 Content of the TMP Implementation Statement
is not contained in the relevant SCS or PICS, and which is
needed by the test laboratory in order to run the executable
The role and scope of the TMP Implementation Statement is
version of the standardized ATS.
given in ISOAEC 9646-2: 1990, clause 14. A proforma for
the TMP Implementation Statement is contained in the TMP
The test laboratory shall provide a PlXlT proforma to the
part of each conformance testing standard which uses the
client for each ATS that is to be used during the conformance
Coordinated ATM.
assessment process.
ISO/IEC 9646-5: 1991 (E)
6.4.5.2 Test laboratory role
7.2.1.3 No additional checks are needed for items which are
indicated in the status column as
The test laboratory shall provide the client with the proforma
for the TMP Implementation Statement for each ATS based on
a) an unqualified option (boolean: do or not do);
the Coordinated ATM which will be used during conformance
b) not applicable.
assessment.
7.2.1.4 The test laboratory shall also check the consistency of
the information presented in the PIXIT and SCS documents
6.4.5.3 Client role
provided by the client.
The client shall provide a TMP Implementation Statement for
7.2.1.5 The test laboratory shall inform the client of the results
each ATS based on the Coordinated ATM which is to be used
of the static conformance review before continuing with the
1 for testing. This is done by completing the relevant proforma
conformance assessment process.
provided by the test laboratory.
I
7.2.2 Client role
7 Test operations
The client shall review the results of the static conformance
7.1 Introduction
review.
0 Clause 7 specifies requirements for test operations. Figure 3
7.2.3 Mutual role
shows typical test operations. For the purpose of this clause,
it is assumed that a single-protocol IUT is being tested using
If the results of the static conformance review reveal that, in
a single standardized ATS, a single PICS and a single PIXIT.
the view of either the test laboratory or the client, to proceed
However, this clause does not exclude a multi-protocol IUT
with testing would not be productive, then a negotiated exit
being tested incrementally, using a series of single-protocol
standardized ATSs. may be taken.
if a multi-protocol IUT is to be tested incrementally, then a
standardized ATS will be used for each protocol to be tested, 7.3 Test selection
and so there will be a PICS for each protocol and a PIXIT for
each corresponding standardized ATS. 7.3.1 Test laboratory role
The test laboratory shall select all those abstract test cases
7.2 Static Conformance Review appropriate for the IUT, based on the information in the PICS
and PIXIT, in accordance with the documentation of the MOT
7.2.1 Test laboratory role and with the requirements of its reference standardized ATS.
This means that the test laboratory shall select the following
7.2.1.1 During the static conformance review, the test lab- test cases provided that they are testable according to the
to the following PIXIT
oratory shall analyse the PICS according
criteria:
...