Version 9.10 of the MUMPS Validation Test Suite (MVTS) is a revised Version 9.05, July 31, 1995. It has been designed to meet for validating the MUMPS implementations against the Standard in ANSI/MDC X11.1-1995 Draft.

It includes such features as (1) computer detection of integrity of the validation test suite, (2) reporting "pass/fail" conditions for individual tests, (3) checkpointing test runs to enable interrupted test to resume at the most suitable position, (4) instructions and warnings at adequate points, and (5) fully automated tabulation of the Validation Summary Report (VSR). Provision of such features will facilitate evaluation of the validation process by relegating these time-consuming tasks to the computer with assistance by a few experienced personnel.

VSR is for reporting the test results of Part-77, Part-84, Part-90, and Part-94.

The current Validation Test Suite V.9.10 includes Phase II extensions, which mostly consist of the fundamental tests for Trasaction Processing added in ANSI/MDC X11.1-1995 Draft. It is composed of Part-77, Part-84, Part-90, and Part-94. It contains a total of valid 4,318 tests.

MVTS V.9.10 has omitted tests for the standard error features in ANSI/MDC X11.1-1995 Draft. These features have been schemed to be tested during validating Transaction Processing facilities using $STACK features.

The previous error handling tests (cotained in VVE and VE) for 92 specific syntaxes defined as errors until the ANSI/MDC X11.1-1990 as to be detected by the standard implementation, requiring a series of manual intervention guided by the driver and the restarter, have been removed since the previous MVTS V.9.05.

Validation sequences of Part-77, Part-84, Part-90, Part-94 have their execution control drivers; VV1, VV2, VV3, and VV4 (and VV4TP) assisted by their restarters, V1, V2, V3, V4, respectively.

Part-77 was originally developed for validating the ANSI X11.1-1977 standard specifications including those minor changes in the ANSI/MDC X11.1-1984. It contains 2,156 tests (currently 2,105 valid tests, and 51 tests withdrawn, suppressed, or moved to Part-90 for 1990 ANSI extensions), plus 8 optional and 2 withdrawn tests for other reasons. There are 266 routines plus 3 supportive routines, its driver (VV1), restarter (V1) , and a routine (V1PRESET) that is externally referenced by the main tests.

Part-84 contains 224 tests (currently 169 valid tests, and 55 tests withdrawn, suppressed or moved to Part-90 for 1990 ANSI extensions). It has 27 routines plus 3 supportive routines, its driver (VV2), restarter (V2), and an externally referenced routine. The tests are targeted for testing conformance only to the revised and/or extended part of the specifications in the ANSI/MDC X11.1-1984.

Part-90 consists of 1108 tests (currently 617 valid and 13 suppressed tests) for the proposed and revised ANSI/MDC X11.1-1995 Draft, in 175 routines including supportive routines.

Part-94 consists of 923 tests (currently 911 valid and 12 suppressed tests) for the revised ANSI/MDC X11.1-1995 Draft, in the routines starting with "V4" as routine names including supportive routines.

It must be emphasized that the driver for the tests of Transaction Processing is VV4TP and not VV4. But the restarter of the test remains as V4.

Transaction Processing opens a very important horizon for the future of multi-user-multi-task Client/Server MUMPS environment. When successfully implemented, this challenging environment should be able to replace so many critically important installations in the circulation enterprises. Implementers are required not to dramatically decrease the current performances of their implementations. This must be the main cause for the implementers of Transaction Processing to be taking great amount of time and efforts to such an extent to let them try different approaches from the specifications defined in ANSI/MDC X11.1-1995 Draft. There are implementations which refuse subtransactions (nested transactions) so that they refuse implementing $TL[EVEL] intrinsic special variable. (If implemented, the value of $TL[EVEL] could only be 0 or 1.)

In the tests for Transaction Processing (VV4TP) it should be noted that the tests for $TLEVEL intrinsic special variable are omitted.

One micro floppy disk contains the following documents:

1) CHAP1: Overview of the MVTS V.9.10

2) CHAP2: Instruction, with an example of Validation Summary Report

and the Flow Tables of Part-77, Part-84, Part-90, and Part-94.

3) CHAP3: Contents of Validation Items of Part-77.

4) CHAP4: Contents of Validation Items of Part-84.

5) CHAP5: Contents of Validation Items of the Part-90.

6) CHAP6: Contents of Validation Items of the Part-94.

7) CHAP7: Sample files of the Validation Summary Reports for "Part-77", "Part-84 ",

"Part-90", and "Part-94".

1) VV1.RTN: Part-77 validation routines with driver and restarter V1, in sequential files.

2) VV2.RTN: Part-84 validation routines with driver and restarter V2, in a sequential file.

3) VV3.RTN: Part-90 validation routines with driver and restarter V3, in a sequential file.

Some implementation specific restore routine may fail to install a routine which has legally permissible routine line length. It may stop at certain point of a routine and may not install the rest part. Therefore, the causes for missing routines should be sought before a new package is requested.

VINT9 is the first MUMPS program to be run after installing the validation suite to assure the operator that the validation suite has been successfully installed. During Integrity Checking each routine is expressed by a dot on the CRT and any damage in the routines will be reported. If one routine is missing, the integrity checker will stop suggesting that there is a missing routine.

The Integrity Checker,VINT9, reports the INTEGRITY STATUS to the environment file to be reported in the VSR. The Integrity Checker is to be run only at the time of initial installation.

The four test groups of validation are independent from each other. Validation process can be started from any one of VV1, VV2, VV3, VV4, or VV4TP. Each of them will first check for the necessary environment information and if necessary will trigger the process before entering the intended series of tests.

While VV1, VV2, VV3, and VV4 plus VV4TP (for Part-77, Part-84, Part-90 and Part-94 respectively) are designed so that they may be run independently, the test results of these four Parts will be reported by the report writer program, VSR, at any time even before completion of all the four Parts.

Part-77 has its driver VV1. VV1 controls the flow of routines to the end of Part-77. There will be no session summaries printed in Version 9.10.

Part-84 has its driver VV2. VV2 controls the flow of routines to the end of the Part-84. There will be no session summaries printed in Version 9.10.

Part-90 has its driver VV3. VV3 controls the flow of routines to the end of Part-90. There will be no session summaries printed in Version 9.10.

Part-94 has its driver VV4. VV4 controls the flow of routines to the end of Part-90. There will be no session summaries printed in Version 9.10.

VSR is the program to be run to generate the Validation Summary Reports (the VSR) on site. It may be run before having completed any one of the four Parts. In such a case VSR will simply report the serial number of not-executed tests as "*SKIP*". However, the SUBTOTAL and the GRAND TOTAL of the statistics will include these "*SKIP*" results in the number of "FAILing" tests.

A session title (title of routine) always precedes the series of tests. Routines in each Part have their own unique Routine Numbers, representing the sequence to be executed in that Part. In each Part, a series of unique sequential numbers are allocated to the individual test in the test sequence.

Whenever a test is executed, the unique serial test number is stored in the global database (VREPORT) with the result and the checkpoint indicating the test to be run next. Checkpointing is secured using the VREPORT file and a more direct checkpointing file NEXT.

The number of routines and associated session titles increase whenever old programs are divided or new programs are added. However, the total ID numbers of parent tests remain unchanged for maintainability, unless a uniquely new test with its new ID number with a new proposition (i.e., the syntax element or specification requirement to be tested) are added.

A section title with or without an ID number may precede a series of tests. IDs of the tests are represented by integers preceded by the Part number. Sometimes the ID number of a test has a series of child IDs, in which integers follow a dot (.) after the parent ID number, or the child IDs may have grand-child IDs with integers following after a second dot. Thus, the child and grandchild tests are the component of the parent and the child tests, respectively. However, failures in the child or grandchild tests are recorded individually and not as failures of their parent tests. The extent of the failures in the family structured test would pinpoint the location and severity of problems in the parent test.

Validation items of are printed each time the line labels of routines are entered, with their unique absolute Test Numbers, and with the page and the section number in the MUMPS Language Specification, followed by the specific document clarifying the error status. Each error detecting MUMPS capability is examined by 4 different tests.

The test IDs were numbered originally in program listing sequence, not according to the program execution sequence. However, the structure tables of the routines illustrate the execution sequence of routines, so that reference to the routine name is made easier.

Session Title, Routine #, Routine Name, Test ID, Reference, Serial Test #,

Test #, and Proposition.

=============================================================================== P.I-22 $TEXT <<<<<--- (Session Title)

4---VETEXT <<<<--- (Routine # and Name)

E-4 P.I-22 I-3.2.8 $TEXT(lineref), $TEXT(+intexpr)

An error will occur if the value of intexpr is less than 0.

<<<--- (Test ID, Page ANSI/MDC X11.1-84,Section Title,

and Statement of the Specification)

E0013 13---E-4.1 $TEXT(+-1) <<---Serial Test #, Test #, Test ID,

E0014 14---E-4.2 S A=$T(+-99) and proposition)

E0015 15---E-4.3 $T(TEXT+2-3)

Session Title, Routine #, Routine Name, Section Title, Test ID,

38---V1BOA3 <<<<--- (Routine # and Name)

Algebraic Difference <<<--- (Section Title)

I-30 expratom=0 <<--- (Test ID and proposition)

10466 I-30.1 0-0 <---|

10467 I-30.2 1-0 <---| (Above Test is subdivided

10468 I-30.3 000-2 <---| into these Child-tests

10469 I-30.4 0-+.999 <---| with Serial Test #s.,IDs

10470 I-30.5 00000-00000.00000E2 <---| and propositions)

===============================================================================

Session Title, Routine #, Routine Name, Section Title, Test ID,

Child-Test ID, Child-Test Title, Grandchild-Test, Serial Test #,

Unary operators -10- <<<<<--- (Session Title)

30---V1UO4A Unary operator -10- <<<<--- (Routine # and Name)

I-801 Multiple Unary Operators <<<--- (Section Title with Test ID)

I-801.1 Duplicate Unary operators and a numlit <<- (Child-Test ID with title)

10328 I-801.1.1 ++0 <---| (Above Child-Test is subdivided 10329 I-801.1.2 +-0 <---| into these Grandchild-Tests

10330 I-801.1.3 +'0 <---| with Serial Test #s., IDs

10331 I-801.1.4 -+0 <---| and propositions)

==============================================================================

Merging of ID numbers has occurred by uniting two or more groups of tests under one or more logical propositions by integrating child tests. When two items were merged, they were made to carry the original ID numbers in the form "I-238/239".

If a test is withdrawn for some reason the test ID number and its unique sequential number will be left in its original site without an executable test, but with the date of nullification, not to be replaced by any other test, and the report file (VREPORT) will be filled with a "*WITHDR*" mark, denoting there was a test which has been withdrawn*). If a newly added test is to be ID-numbered, it shall take a new ID number regardless of the site of its insertion.

The Validation Suite documentation contains such statements as "The last test ID number of Part-77 is I-847," denoting that the most recently created test (not the one added or divided for clarity) is "I-847" which will be found somewhere in the routines with a new proposition. This forces the next newly created test to take the ID number "I-848".

--------------------------------------------------

*) Only two *WITHDR* marks represent the two tests created before and abolished after ANSI X11.1-1977 was instituted. All other *WITHDR* marks are added due to any one of the reasons: 1) modified and moved to other Parts after being extended for ANSI/MDC X11.1-19xx, 2) obsolete, or 3) temporary suppression in order not to force implementations to any semantics other than resolved by the MDC.

The results of validation are checked for "passing" or "failing" for the valid and optional tests. GRAND TOTAL statistics in VSR will count all the results except those of optional tests. The Validation Summary Rreport generated by the program VSR will not require manual operation for filling blanks or whatsoever. The reports will be ideally output by request after completion of all the validation Parts. They consist of about 16 pages, which include matrix tables of the serial test numbers and their results (see Chapter 2, 2.4.3 The Validation Summary Report, and Chapter 7, Sample Validation Summary Reports).

During execution of the routines, a "passing" test is reported by a "PASS" sign followed by the ID# and the test proposition, on the Principal Printer or CRT each time one test is run. A "failing" test is reported by a "** FAIL" sign followed by the ID and the test proposition. The basis for detecting as "failure" is reported with comparison of the computed against the correct results. The cause of "failure" has frequently been requested to be reported, but it will only be detected by the implementor of the system.

The VREPORT file keeps the serial numbers and the results of all individual tests, in the forms classified into "PASS", "*FAIL*", "*FAILO*", *ABORT*", and "*SKIP*". Withdrawn, suppressed, or moved tests from the Validation Suite remain in the Suite with their serial numbers, ready to be reactivated in case an earlier FIPS MUMPS implementation is to be tested for conformance. The VREPORT file has those correct results of the suppressed, moved, or abolished tests filled with a "*WITHDR*" mark. These "withdrawn" tests will not be counted in the VSR. Arbitrary inactivation of the test item by the operator will be reported simply as "*SKIP*".

Those tests causing "error" interruptions, in the tests of Part-77, Part-84, Part-90, and Part-94 are stored in the result file as "*ABORT*". "Aborted" tests will be counted as "failures", but if the interruption has been made by human error, the test can be restarted by the restarters.

Rigorousness of this Validation Suite is due to its complete coverage of syntactic elements in a minimum number of required combinations and their semantic behaviors. The Validation Suite is limited in that it does not provide tests beyond the boundaries specified in ANSI/MDC X11.1-1995 Draft and its revision specifications. Namely, it does not provide such testings as:

a) How far the boundaries of the standard specifications are extended; as numbers of characters in a string literal beyond 255, existence of control characters in the string subscripts, acceptance of non-ASCII characters, or acceptance of non-MUMPS commands and functions, etc.

b) If a MUMPS process B, initiated by a JOB command in process A, continues without disturbance when the triggering process A ends and exits from the partition.

c) Database performance and robustness tests, including such tests if certain data pointers are lost during repeated SETting and KILLing of global variables, resulting in accidental loss of data values.

An implementation detected for some failures in the validation test should immediately be instructed to correct the "errors" in order to pass these particular tests. Whatever question arises, it should not be left unanswered.

End of Chapter 1. MUMPS Validation Test Suite Overview