The ICSP header allows you to plug a programmer onto the board so you can download new code to the microcontroller, (uC) as well as debug code that's on there. You can also use the programmer to read data off the uC.
There are other standards besides ICSP (In Circuit Serial Programming) it's used by the Microchip products. I use the PIC microcontrollers made by Microchip because they are very efficient and beat the Atmel chips at low power applications.
You can Google ICSP pinout to get the low down on the pins but it's pretty straight forward. MCLR is used to put the chip into programming mode, connect that to the MCLR pin on your uC (the uC datasheet will have all this listed out in the pin descriptions).
+5V, power and GND, ground. It needs power and ground like everything else.
DAT this is the data line where serial data (e.g. your code) gets transferred
CLK clock is used to negotiate timing between the programmer and the uC
Sparkfun.com has a nice Eagle part for this header which has the pins slightly offset so the header sticks in there and is easier to solder (or to just hold and program if you don't want a soldered in header).
To make a microcontroller board you just need power, a voltage regulator like the LM7805, a programming header like this one and your uC.