fatal error: limits.h: No such file or directory

On OSX, I can't compile simple programs from the command-line when I use an externally-provided compiler (e.g. gcc, or a custom clang install). System headers like limits.h cannot be found.

Example error messages:

fatal error: limits.h: No such file or directory

Or:

fatal error: stdio.h: No such file or directory

Or:

fatal error: 'stdlib.h' file not found

and so on.

What's wrong?