What SQA Resource are you searching for?

Skepticism, like chastity, should not be relinquished too readily. – George Santayana NB. This document brings fond memories – was originally prepared in 1978 using a 30 cps Teletype 43 terminal with UPPER CASE characters when writing 3GL programs was adventure in the embedded computer systems world where most software was written in assembler. The document was provided to the IEEE Software Engineering Standards Sub-Committee in 1981 – the IEEE guidelines for audits and reviews evolved many years later. ABSTRACT Brevity is the soul of wit. – William Shakespeare, “Hamlet”, Act 2 scene 2 A set of guidelines for review and auditing of software documents is provided. The guidelines contain checklists for auditing the documents produced at the various phases of the life cycle. Also, rules for conducting reviews are given and comments about possible pitfalls are included. Psychological aspects of the review process are emphasized. The guidelines are organized in a way that enables systematic application for the analysis of a given document. KEYWORDS: audit, review, software quality assurance, software configuration management, walk-through, software standards. 1. INTRODUCTION top It is quality rather than quantity that matters. – Seneca, Epistles Auditing and reviews are both a major software quality assurance activity and one of the four functions of configuration management: * identification; * configuration control; * status accounting; * auditing. Extensive software tool support can be provided to all the other configuration management functions except auditing. Auditing and reviews are by nature a manual process that can be supported by proper guidelines and some minor software tools. The process of auditing and reviews is the main means of feedback for management and buyer. This is especially true for material that is not readable by the computer. * This is to assure that specifications conform to requirements and the code to the specs. * The audits are methods by which management and buyer can give their approval to continuation of the development effort. * At a less formal level – it is used for internal checks and verifications. * There are several types of reviews: * REVIEW: used to check progress on a periodic basis * AUDIT: used for checking all aspects of the product, usually at a milestone. * CODE INSPECTION: a peer inspection of code during development. * WALKTHROUGH: a simulation of what the system is supposed to do. Preparing a set of test cases and tracing system transformation in response to assumed conditions does it. Although it is done usually at a code level, it may also be done at a higher level. 2. ORGANIZING FOR AUDITS top 2.1 ROLES The lady doth protest too much, methinks. – William Shakespeare, “Hamlet”, Act 3 scene 2 * The people taking part in an audit may vary according to type of the audit (or review). At least the following should participate: * PRODUCER. The person responsible for the product being reviewed. This may be the manager of the department – if it is a big product – or the actual writer of the document. * Her role is to present the product and to respond to comments of other participants. * If the review is an informal one, the producer is not expected to react immediately to comments but to elaborate them afterwards. * COORDINATOR. The person summoning the meeting and scheduling the audit. The coordinator is the manager of the audit. His duties include deciding who will be summoned, distribution of invitations, distributing of material and preparing of the room. During the actual meeting the coordinator will act as moderator, and after the meeting he will prepare minutes and distribute them to the participants. * SCRIBE. The person taking notes, he may be the coordinator. * USER REPRESENTATIVE. * QUALITY ASSURANCE REPRESENTATIVE. Will consider issues of testability, reliability, maintainability, conformance to standards and other quality criteria. * Other reviewers may be included according to the nature of the product, managerial responsibilities or contractual agreement. The configuration manager and the person maintaining the system are likely participants for reviews. * The reviewer team should preferably not exceed 5-6 people. * Reviewers must be highly qualified personnel and capable of writing the material they review. * As development proceeds according to a hierarchical manner – each document is a refinement of a document at a previous level and sources to next document – the reviewers may include representatives of the authors of the previous level and intended authors of the next level, e.g.: the author of the requirements will be represented in the review of the preliminary design document, and will check if the requirements were correctly understood. The author of the detailed design document will also be present and will analyze the feasibility of the preliminary design. 2.2 PREPARATIONS BEFORE THE MEETING top The coordinator has to gather the material and distribute it. This gathering activity must be done if there are more than one source of documents. In this case the ‘producer’ will not be the actual author but he will be THE PRESENTER, rather than the ‘producer’. The coordinator sends the invitation and assures the appropriate time for the meeting. * The PRODUCER has to prepare the presentation. If a walk-through is intended the producer has to prepare some test cases. If the producer is also the author it is better that someone else should prepare the test cases (or scripts). * The participants must prepare themselves for the meeting by reading the material and analyzing it. It is possible to supply them with such support reports as a ‘keyword under context’ and cross-references : arranged according to data items, according to various groupings. This can be done if the reports are written in some semi formal language like the Problem Statement language (PSL). In that case there are special programs that can produce the support reports. The structure of the preparation will be described later. 2.3 CONDUCTING THE REVIEW top No legacy is so rich as honesty. – William Shakespeare, “All’s Well that Ends Well”, Act 3 scene 5 * The coordinator calls the meeting to order, and gives a brief explanation of the problem, and then introduces the participants to each other. * The producer takes the floor. If it is a follow-up of previous meetings the producer may review the action taken to react to the previous comments, and then presents the product, piece by piece. At the end of each piece comments are requested. * If a walk-through is executed – the order of presentation is not necessarily the order of the document but the order of the test cases. As branching points arise (multiple choices) a certain method has to be devised, assuring that all possibilities are considered. The order may be according to levels (for each point where more than one possibility arise – terminate the review of all the choices of the first level and than proceed for each following level the same way). * During the walk-through the scribe takes notes. The notes do not have to be taken verbatim unnecessary. * At the end of the review a vote may be taken upon recommendations concerning the product. 2.4 PSYCHOLOGICAL PROBLEMS top Do not speak ill of the dead. – The Seven Sages, from Diogenes Laertius, Lives of Eminent Philosophers * The moderator must be aware of various dangers and problems of human nature: * The meeting may evolve into a personality clash. The coordinator must be careful to direct comments to the product being reviewed rather than to the producer. The producer must also be prevented from defending the product by personal attack. This is rather difficult if both top management and the user are present. * Care must be taken that the discussion will not drift to other topics or will turn into a social chat, although some casual talk may, of course, be permitted in order to allow for a relaxed atmosphere. * Arguments about minor topics must also be avoided, especially arguments about style. * The coordinator must pay attention to shy participants who must be encouraged to participate actively. This may be done by eliciting comments according to some predetermined order (like the order of seating) without waiting for the participant to initiate the comment. * The coordinator has to be sensitive to the fine psychological inclinations and reactions that reflect political or psychological power struggle but are disguised as content comments. 3. Guidelines and Checklists I never cease being dumbfounded by the unbelievable things people believe. – Leo Rosten 3.1 PHASES OF AUDITS The phases of the reviews to be discussed here are: * Software Requirements Review. * Preliminary Design Review. * Detailed Design Review. * Code Inspection. * Formal Qualification Review. 3.2 Checklists top The following requirements are applicable to all phases of reviews: 3.2.1 FORMAT CHECKLIST * The format of the document should be checked and conform to company standards with respect to style and external form and items to be included. The following should be checked: * Table of contents. * Glossary. * Breakdown into small items. * Proper indentation and spacing. * Use of tables and drawings. * Proper identification of pages, chapters, figures and tables. * Proper date and change designation. * Bibliography. * Use of footnotes. * Check style and use of complicated expressions. * General appearance and typesetting. * It is advised that the reviewer becomes familiar with the document from the point of appearance and layout. * Look first at the table of contents and digest the form and structure of the document. * Prepare a plan for reading the document. The document may be organized in a way that may be difficult to read sequentially like in the order of the alphabet. * Study the glossary. This will give the reader also a general view concerning the objects that are dealt with in the document. * Then study the general format of the document. If it is found to be deficient or not compatible with standards or conventions it should be noted. 3.2.2 GENERAL CHECKLIST The general checklist is suited to all phases of the reviews. Each item in the checklist can be checked for each paragraph for each level of the document internal structure. (1) ACCURACY. * Are the facts accurate? * Check computation (when feasible) on a random basis. (2) COMPLETENESS * Is any function missing? * Is any input, output or condition missing? * Is any possibility overlooked? * Avoid mental blocks by using various creativity enhancement methods (see De Bono [8]). (3) CONSISTENCY * Usage of terms has to be unique. The same term may not be used for different ideas. * An idea or object must have only one term describing it. Pay special attention to usage of synonyms, different structures of sentences. The use of abbreviations for long terms is allowed. * The abbreviations must be defined in an abbreviation table and in the glossary. (4) USEFULNESS * Is there a purpose to each feature? * What is the end use of an interim feature? (A feature that has no meaning for the user). (5) TESTABILITY * How will each feature be tested? * How will internal features (not visible externally) be tested? (6) MODULARITY * The system and documents describing it have to be modular. Modularity means independence of modules: * Maximize internal binding, minimize coupling. * Modules must be of limited size. * Structure should be hierarchical. * Repeated usage of the same module (a module may be a software module, and documentation module – a paragraph) must be properly cross-referenced. (7) CLARITY * The meaning of each item is understood. * There is a single interpretation of each item. * Each item’s description is exact and not vague. * The specification is easy to read. (8) FEASIBILITY For higher level documentation like requirements – consideration should be given to implementation issues. If implementation is not clear some indication that it is possible must be given. It may suffice to point out similar requirements that were solved elsewhere. (9) RELIABILITY * What will happen in case of system failure? * How is the system going to respond to unexpected input? * What diagnostics are to be produced? * For critical software a special reliability checklist may be prepared by Quality Assurance and a special Reliability Review may be held. (10) MAINTAINABILITY * The development group has to consider changes, as follows : * List the features that are not likely to be changed. Grade them according to probability of change. This can be done before the review by sending questionnaires to all participating groups. During the review these lists may be modified. The items not in the list are likely to be changed. * For items that can be changed a list of expected changes is compiled. The creativity methods mentioned earlier may be also used. Each item within the system can be checked in view of these changes. (11) HUMAN ENGINEERING FACTORS (12) MINIMAL COMPLEXITY Minimize complexity across: CONCEPTUAL INTEGRITY * The system has to have a clear unifying line of thought. Command and system reactions must have the same syntax. Internal tables must have similar form and criteria for the design decisions must be the same. * The user interface has to show a measure of uniformity and harmony. The semantics of the commands has to be similar. An inherent symmetry with respect to syntax, names, operands, defaults. COMPONENT INDEPENDENCE The system has to be partitioned into its components so that the dependence of the components is minimized. The high frequency dynamics of the system falls within the components and the inter-component interactions represent the lower frequency dynamics. HIERARCHY – The hierarchy is a method for simplifying the description of a system. Check if this method was used properly. (13) TRACEABILITY This is an important feature. Each item of documentation must have its source clearly stated. 3.2.3 AMBIGUOUS WORDS TO BE OBSERVED The following words may have ambiguous meaning: Some different possibilities are given for each example below: A (AN): (one and only one? some? at least one?) “The data set will contain an end of file character. ” AFTER: (immediately? somewhere after?) “The control number comes after the quantity.” ALL THE TIME: (initially and never reset? Every time?) “The interrupt status is set to ‘optional’ all the time.” AND: (either…or? double condition?) “The sequence ends on a flag and an end of file.” CURRENT: (current for program? current for real system? current in other sense?) “The control total is taken form the current record.” FOLLOWING: (following immediately? Following somewhere away?) “The checksum is on the following summary card.” FROM: (starts from designated number? Starts somewhere after the number?) “The registry number start from 100.” LARGEST: (what is the set and sequence?) “The field will be set to the largest possible value.” (character or number?) LAST: (last for the program read? last referenced? last updated? last in file?) “The control total is taken form the last record.” LATEST: see previous word. MAY: (can it be something else not mentioned too?) “The return code may contain an integer or a blank.” OR: (either…or? only one?) “The sequence ends on a flag or an end of file.” SAME: (same value? same format?) “All customers have the same control field.” SHOULD: (must? ought?) “The operator should mount the designated disk pack.” SMALLEST: see LARGEST. THEIR: (whose exactly?) “A family unit consists of a mother and father and all their children.” THEY: see THEIR. TO: see FROM UNTIL: (stopped now? stopped by other condition also?) “Elements are added until the element with the present date has been added.” WE: (who exactly – the user, the developer and the user, who else?) “We will create an operator’s guidebook.” WHEN: (the only way? are there other ways?) “The terminal session ends when the sign-off command is issued.” WHEN: (by the time? the first time? only when? whenever?) “The counter is set to zero when the subroutine is invoked.” WILL: (who will? is it a fact?) “The pointer will be set before the data value is set.” - HOW TO USE THE LIST OF WORDS: A. Each paragraph can be read with the list of items in mind. This is possible after the list has been more or less memorized. B. If the document is in machine readable form – a computer search can be made for statements containing the listed words. Then they can be examined for ambiguities. 3.2.4 AMBIGUOUS EXPRESSIONS TO BE OBSERVED 1. UNPROVED CERTAINTY. “There can be no more than fifty lines per invoice.” Is it a hope? What is the proof? Push the search for a proof as far back as possible. That is – the proof, too, needs a proof. 2. REQUIREMENT OR FACT? “The flag is set to zero.” Is it a fact or a requirement? 3. VAGUE WORDS: SOME, SOMETIMES, OFTEN, USUALLY, ORDINARILY, CUSTOMARILY, MOST, MOSTLY. “Exception processing will be required for some of the line items.” What is the exact number? 4. INCOMPLETE LISTS. Watch for: ETC., AND SO FORTH, AND SO ON, SUCH AS. “The permitted inputs are ‘A’, ‘B’, ‘C’, etc. The rest of the list may contain two more items or A-Z letters. Try to complete the lists. 5. UNSTATED ASSUMPTIONS. “The valid codes range from 100 to 1000.” Is it only integers? hexadecimals? 6. LISTS WITHOUT EXAMPLES (OR TOO FEW EXAMPLES) See previous rule. 7. VAGUE VERBS. HANDLED, PROCESSED, REJECTED, SKIPPED, ELIMINATED. “Unmatched detail records are to be rejected.” What has to be done on rejection? 8. PERSUASIVE WORDS. CERTAINLY, THEREFORE, CLEARLY, OBVIOUSLY, AS ANY FOOL CAN PLAINLY SEE. “Clearly, the dataset control pointer will be set before any data value is set.” Is it a fact? Perhaps a requirement? 9. PASSIVE VOICE. “The parameters are initialized.” By whom? may be by nobody? 10. COMPARATIVES WITHOUT REFERENTS. “The final total is to be placed in the NEW file.” NEW – compared to what? It may depend upon the time of reference. 11. PRONOUNS. IT, HE, SHE, THEY, HIS, HERS, ITS, THEIR, WE, US, OUR, YOU, YOUR. “Module ZAP is called by module ZIP. It must be given the security code.” Who – ZIP or ZAP? 3.2.5. TRANSFORMATIONS FOR CLARIFICATION The meaning of a paragraph may be clarified or ambiguities revealed by subjecting the text to various transformations: 1. VARY STRESS PATTERNS. “The EXCEPTIONAL cases are handled by the terminal operator.” (Only those cases!) “The exceptional cases are handled by the TERMINAL operator.” (No other operator!) 2. TERM SUBSTITUTION. Substitute the definition of the terms instead of the terms themselves. “The ‘current status file’ becomes the ‘old status file’.” “The ‘direct access file containing the current status of all active accounts’ becomes the ’sequential file of accounts carried over from the previous period’.” Notice the disparities in the definitions. 3. PICTURE THE STRUCTURE. When a structure or a relation is described in words – try to picture it. See if the relations among the objects are unique. Try to change the relations and see if it fits the description in words. 4. EXPRESS EQUATIONS IN WORDS. 5. EXPRESS CALCULATIONS IN AN EQUATION. 6. WORK TWO EXAMPLES OF AN EQUATION BY HAND. 4. PHASE GUIDELINES My salad days, when I was green in judgment. – William Shakespeare, “Antony and Cleopatra”, Act 1 scene 5