Integers and Integer Operations in C

Integers and Integer Operations in C
Vulnerabilities, Attacks, and Mitigations

Prof. David Bernstein
James Madison University

Computer Science Department

bernstdh@jmu.edu

Motivation

The Fundamental Issue:
- Integers have fixed limits
The History:
- These limits have often been ignored because they were sufficient and/or because testing was too expensive
The Reliability/Robustness Concern:
- The limits aren't always sufficient (e.g., Comair had to halt ground operations in 2004 because their crew-scheduling software limited the number of changes to 32,768 in a month)
The Security Concern:
- Attackers will intentionally violate these limits

Wraparound Vulnerabilities

Definition:
- A vulnerability that arises because of a defect involving integer wraparound
An Example:
- JPEG files contain a comment field that includes the 2-byte length of the comment
- The length of the comment alone is, thus, length - 2 + 1, which is the value passed to malloc()
- This can be used to cause integer wraparound

Conversion Vulnerabilities

Definition:
- A vulnerability that arises because of a defect involving conversion

An Example:

void initialize_buffer(int size)
{
  if (size < MAX_SIZE) buffer = malloc(size);
  else                 // handle the error;
}

The Defect:
- malloc has a parameter of type size_t so the int must be converted, which could result in a too-large value if size is negative

Truncation Vulnerabilities

Definition:
- A vulnerability that arises because of a defect involving truncation

An Example:

unsigned short int total;

total = strlen(first) + strlen(last) + 1;
char* both = (char *)malloc(total);
strcpy(both, first);
strcat(both, last);

The Vulnerability:
- An attacker might provide arguments such that the sum can't be represented by an unsigned short int
The Example Revisited:
- The right-side is 65500 + 36 + 1 meaning total will be assigned the value 65537 % 65536 (i.e., 1)

Mitigation

Use Appropriate Integer Types:
- Use a type that can fully represent the range of values
- Be explicit in the use of signed or unsigned
Use rsize_t (C11):
- Defined to be size_t but explicitly used to hold the size of a single entity
- Functions that have parameters of type rsize_t detect constraint violations for values greater than RSIZE_MAX
Use typedef for Readability and Portability
Use Secure Integer Libraries:
- Operations either succeed or report an error