- Forward


The Basics of Testing and Debugging


Prof. David Bernstein
James Madison University

Computer Science Department
bernstdh@jmu.edu

Print

Desk Checking
Back SMYC Forward
  • Definition:
    • Verifying that a program satisfies specifications by carefully reading the code
  • Desk Checking for Students of Introductory Programming:
    • Did you forgetting to initialize variables?
    • Are your Boolean expressions correct?
    • Did you forget to check for a condition (e.g., divide by 0)?
    • Did you compare values appropriately (e.g., consider rounding errors, compare reference types)?
    • Did you misuse integer division?
Testing
Back SMYC Forward
  • Definition:
    • Verifying that a program satisfies specifications by executing the code
  • Testing for Students of Introductory Programming:
    • Unit testing should be performed when a module is completed
    • System testing should be performed after (the current version of) the program/application is completed
Debugging
Back SMYC Forward
  • Definition:
    • Locating and repairing logic errors
  • Debugging for Students of Introductory Programming:
    • Don't worry about formal terms like failure, fault, and defect; use the informal term error instead
    • Do distinguish between the symptom (i.e., the observable manifestation of the error) and the error itself (the logic error in the code)
The Relationship between Testing and Debugging
Back SMYC Forward
  • The Process:
    1. Create test cases that can be used to identify symptoms (i.e., incorrect behaviors)
    2. Execute the code and identify incorrect behaviors
    3. Form a hypothesis about each incorrect behavior
    4. Instrument the relevant portion(s) of the code (using debug output statements or a debugger)
    5. Execute the instrumented code
    6. Prove the hypothesis (or create a new one and repeat)
    7. Fix the error
    8. Run the tests again
  • Some Observations:
    • There is some creativity involved both in the creation of tests and in the formation of hypotheses
    • It is important to use a good process (and be able to describe the process you use); haphazard debugging is almost never successful
Instrumentation Using "Debug Output Statements"
Back SMYC Forward
  • Defined:
    • The inclusion of calls to print() that are designed to provide information about the program while it is executing
  • Best Practices:
    • Include a description of what is being output in every call (otherwise it will be impossible to interpret)
    • Don't delete these statements (until the program is correct), just comment them out (because you'll need them later)
Developing Tests
Back SMYC Forward
  • White Box (a.k.a. Open Box and Clear Box)
    • You know how the algorithm works
    • You should try to devise tests that execute all statements/branches/control paths
  • Black Box (a.k.a. Closed Box)
    • You only know the inputs and outputs, not the workings
    • You should generate random/representative test inputs (and tests inputs based on the specifications, if you have them)
White/Clear Box Testing
Back SMYC Forward
  • Statement Coverage:
    • Make sure every statement is executed
  • Branch Coverage:
    • Make sure every "branch" (resulting from conditionals) of every statement is executed
  • Path Coverage:
    • Make sure every "path" through the code is executed (which is often impractical in practice)
White/Clear Box Testing: An Example
Back SMYC Forward
public static int calculate(int x, int y) { int a, b; a = 1; if (x > y) { a = 2; } x++; b = y * a; if (y <= 0) { b++; } return b; }
  • One Test Can Cover Every Statement:
    • calculate(5, -2)
  • Two Tests Can Cover Every Branch:
    • calculate( 5, 2)
    • calculate(-2, -1)
Black Box Testing: A Method with one Positive Input
Back SMYC Forward
  • A Seemingly Good Idea:
    • Use all possible inputs
  • The Difficulty:
    • All outputs need to be verified
  • A More Realistic Approach:
    • Some random tests
    • Some heuristic tests
Testing - Black-Box Unit Testing (cont.)
Back SMYC Forward
  • Random Testing:
    • Generate test cases randomly
  • Heuristic Testing:
    • Generate test cases using "rules of thumb" (e.g., boundary value analysis)
Testing - Black-Box Unit Testing (cont.)
Back SMYC Forward
  • Value Heuristics:
    • Extreme values are thought to be the most likely to cause a failure (e.g., include a test case with a large positive value, a small positive value, 0, a large negative number, a small negative number)
    • Include type mismatches (e.g., characters for integers)
  • Array Heuristics:
    • Include a small array, large array, 0-length array, 1-length array
    • Include unsorted, sorted ascending, and sorted descending arrays
    • Include arrays with one value (e.g., all negative, all positive, all 0)
Testing - Black-Box Unit Testing (cont.)
Back SMYC Forward
  • File Name Heuristics:
    • Include multiple files with the same prefix.
    • Include multiple files with the same suffix (i.e., file type).
    • Include files that reverse the prefix and suffix.
  • String Heuristics:
    • Include strings that vary only in case (e.g., all upper case, all lower case, mixed case)
    • Include zero-length strings
    • Include long strings
    • Include short strings
    • Include strings that consist of one repeated character
    • Include strings that are in reverse order
    • Include strings that swap underscores and dashes
    • Include strings that swap ones and lower case ls
    • Include strings that swap zeros and Os
Test Execution "Tools"
Back SMYC Forward
  • Stub:
    • A program that simulates the activity of missing components
  • Driver:
    • A program that passes test cases to a particular component
  • Test Harness:
    • A system that facilitates the organization and execution of tests
There's Always More to Learn
Back -