Nothing much. The SD card is there for your convenience. You'll lose your photos and music and anything else you've downloaded. If you've never installed an app to the SD card, you'll still have your apps, too.
Even if you install apps to the SD card, you can always re-download and reinstall them. It's no big deal if you wipe it. It's just only annoying if you have a lot of data on the card that you have to put back on.
PS: I switched the memory card that came with my Droid 1 to my Droid 2. The Droid 1 was a fine and so was the Droid 2.