Learn your Debugger
It's really helpful to get to grips with the debugger, whether it be text-based, full IDE or some blend thereof. You don't give much details so I'll describe the general case:
1) Breakpoints
In addition to just stopping at a line of code, many debuggers let you specify to break when a condition arises (e.g. "x > 5"), after a number of passes through the code or when some memory changes value. This is very useful for understanding how your code gets into a bad state, e.g. watching for when a pointer becomes null rather than catching the crash when it is dereferenced.
2) Stepping through code
You can step into functions, line-by-line along code, jump over lines ('set next statement') and then 'up' out of functions. It's a really powerful way of following your code execution to check that it does do what you think it does :-)
3) Evaluating expressions
So you can put variables into a Watch list/window and see their value change when you hit a breakpoint or step through code, but you can also usually do complex expression evaluations too, e.g. "x + y / 5" will be evaluated. Some debuggers let you put function calls into the watch lists too. You can do things like "time()", "MyFunction(...)", "time()" and get clock timing of how long your function took.
4) Exceptions and Signal handling
So if your language supports exceptions and/or signals, you can usually configure the debugger for how to react to this. Some debuggers let you break into the code at the point where the exception is about to happen, rather than after it has failed to be caught. This is useful for tracking down weird problems like "File Not Found" errors because the program is running as a different user account.
5) Attaching to a process/core
So sometimes you have to use the debugger to jump on an existing process which is going awry. If you've got source code nearby and the debug symbols are intact, you can dive in as if you'd started in the debugger in the first place. This is similar for core dumps, too, except you usually can't continue debugging in those (the process has already died).
Build Configuration
There are a number of build variations you can make through turning on or off features like debug symbols, optimisations and other compiler flags:
1) Debug
Traditionally, this is a plain build with no special characteristics, which makes it both easy to debug and predictable. It varies a little by platform, but there may be some extra head-room on, e.g. allocations and buffer sizes in order to ensure reliability. Usually a compiler symbol like DEBUG or Conditional("Debug") will be present so debug-specific code is pulled in. This is often the build that is shipped, with function-level symbols intact, especially if reliability and/or repeatability are a concern.
2) Release/Optimised build
Enabling compiler optimisations enables some low-level code generation facilities in the compiler to make faster or smaller code based on assumptions about your code. The speed increases possible are irrelevant if your algorithm choice is poor, but for intensive computations this can make enough of a difference through Common Subexpression Elimination and Loop Unrolling, etc. Sometimes the assumptions made by the optimiser are incompatible with your code and so the optimiser has to be cranked down a notch. Compiler bugs in optimised code have also been a problem in the past.
3) Instrumented/Profiled build
Your code is built with specific instrumentation code to measure the number of times a function is called and how long is spent in that function. This compiler-generated code is written out at the end of the process for analysis. Sometimes it's easier to use a specialised software tool for this - see below. This type of build is never shipped.
4) Safe/Checked build
All the 'safety valves' are enabled via preprocessor symbols or compiler settings. For example, ASSERT macros check function parameters, iterators check for non-modified collections, canaries are put into the stack to detect corruption, heap allocations are filled with sentinel values (0xdeadbeef being a memorable one) to detect heap corruption. For
customers who have persistent problems that can only be reproduced on their site, this is a handy thing to have.
5) Feature build
If you have different customers that have differing requirements of your software product, it's common to make a build for each customer that exercises the different parts when testing. For example, one customer wants Offline functionality and another wants Online-only. It's important to test both ways if the code is built differently.
Logging and Tracing
So there's writing some helpful statements to printf() and then there's writing comprehensive, structured trace information to data files as you go. This information can then be mined to understand the runtime behaviour/characteristics of your software. If your code doesn't crash, or it takes some time to repro, it is helpful to have a picture of e.g. list of threads, their state transitions, memory allocations, pool sizes, free memory, number of file handles, etc. this really depends on the size, complexity and performance requirements of your application, but as an example, game developers want to ensure that there are no 'spikes' in CPU or memory use while a game is in progress as that will likely affect frame rate. Some of this info is maintained by the system, some by libraries and the rest by the code.
Other Tools
It's not always the case that you have to make a different build to cover these scenarios: some aspects can be chosen at runtime through process configuration (Windows Registry tricks), making alternate libraries available with higher priority to the standard libraries, e.g. efence on your loader path or using a Software ICE or specialised debugger to probe your software for runtime characteristics (e.g. Intel v-Tune). Some of these cost big money, some are free - dtrace, Xcode tools.
