Decoupling (Bypass) Capacitors
A lot of the capacitors you see in circuits, especially those featuring an integrated circuit, are decoupling. A decoupling capacitor's job is to supress high-frequency noise in power supply signals. They take tiny voltage ripples, which could otherwise be harmful to delicate ICs, out of the voltage supply.
In a way, decoupling capacitors act as a very small, local power supply for ICs (almost like an uninterruptible power supply is to computers). If the power supply very temporarily drops its voltage (which is actually pretty common, especially when the circuit it's powering is constantly switching its load requirements), a decoupling capacitor can briefly supply power at the correct voltage. This is why these capacitors are also called bypass caps; they can temporarily act as a power source, bypassing the power supply.
Decoupling capacitors connect between the power source (5V, 3.3V, etc.) and ground. It's not uncommon to use two or more different-valued, even different types of capacitors to bypass the power supply, because some capacitor values will be better than others at filtering out certain frequencies of noise.
While it seems like this might create a short from power to ground, only high-frequency signals can run through the capacitor to ground. The DC signal will go to the IC, just as desired. Another reason these are called bypass capacitors is because the high frequencies (in the kHz-MHz range) bypass the IC, instead running through the capacitor to get to ground.
When physically placing decoupling capacitors, they should always be located as close as possible to an IC. The further away they are, they less effective they'll be.
To follow good engineering practice, always add at least one decoupling capacitor to every IC. Usually 0.1µF is a good choice, or even add some 1µF or 10µF caps. They're a cheap addition, and they help make sure the chip isn't subjected to big dips or spikes in voltage.
Power Supply Filtering
Diode rectifiers can be used to turn the AC voltage coming out of your wall into the DC voltage required by most electronics. But diodes alone can't turn an AC signal into a clean DC signal, they need the help of capacitors! By adding a parallel capacitor to a bridge rectifier, a rectified signal like this:
Can be turned into a near-level DC signal like this:
Capacitors are stubborn components, they'll always try to resist sudden changes in voltage. The filter capacitor will charge up as the rectified voltage increases. When the rectified voltage coming into the cap starts its rapid decline, the capacitor will access its bank of stored energy, and it'll discharge very slowly, supplying energy to the load. The capacitor shouldn't fully discharge before the input rectified signal starts to increase again, recharging the cap. This dance plays out many times a second, over-and-over as long as the power supply is in use.
If you tear apart any AC-to-DC power supply, you're bound to find at least one rather large capacitor. Below are the guts of a 9V DC wall adapter. Notice any capacitors in there?
There might be more capacitors than you think! There are four electrolytic, tin-can-looking caps ranging from 47µF to 1000µF. The big, yellow rectangle in the foreground is a high-voltage 0.1µF polypropylene film cap. The blue disc-shaped cap and the little green one in the middle are both ceramics.
Energy Storage and Supply
It seems obvious that if a capacitor stores energy, one of it's many applications would be supplying that energy to a circuit, just like a battery. The problem is capacitors have a much lower energy density than batteries; they just can't pack as much energy as an equally sized chemical battery (but that gap is narrowing!).
The upside of capacitors is they usually lead longer lives than batteries, which makes them a better choice environmentally. They're also capable of delivering energy much faster than a battery, which makes them good for applications which need a short, but high burst of power. A camera flash might get its power from a capacitor (which, in turn, was probably charged by a battery).
Capacitors have a unique response to signals of varying frequencies. They can block out low-frequency or DC signal-components while allowing higher frequencies to pass right through. They're like a bouncer at a very exclusive club for high frequencies only.
Filtering signals can be useful in all sorts of signal processing applications. Radio receivers might use a capacitor (among other components) to tune out undesired frequencies.
Another example of capacitor signal filtering is passive crossover circuits inside speakers, which separate a single audio signal into many. A series capacitor will block out low frequencies, so the remaining high-frequency parts of the signal can go to the speaker's tweeter. In the low-frequency passing, subwoofer circuit, high-frequencies can mostly be shunted to ground through the parallel capacitor.
When working with capacitors, it's important to design your circuits with capacitors that have a much higher tolerance than the potentially highest voltage spike in your system.
Here's an excellent video from SparkFun Engineer Shawn about what happens to different types of capacitors when you fail to de-rate your capacitors and exceed their maximum voltage specs. You can read more about his experiments here.