Always use TWRP as your recovery, and always Factory Reset, wipe cache, wipe dalvik cache, wipe system.
Only time you can get away with wiping caches is with nightly builds, like twa_priv releases a new CM10 build every couple days or whatever, then no data wipe is required, but when changing roms you absolutely need to do a full wipe.