I ended up reverting back to the "M" stable image yesterday (was waiting for the June image to be available). No issues with the "N" to report, except one custom app that I use was giving me some issues.
Anyway, thought I'd share my "downgrade" story
If you do the downgrade the right way (un-enroll from the Beta program), and get the OTA back to the latest "M", it will likely go well - because it performs a factory reset on your phone.
I however, didn't wait to set my phone up from scratch (lazy), so I flashed the factory image for the latest "M" (and skipped userdata.img). Bad idea. Flash went thru' successfully, but I ended up with pretty much an unusable phone because two system processes kept crashing in a loop after booting up. I then tried ADB sideload for the new OTA image file - same results. Wiped cache partition - no difference.
I then remembered that some apps (like Google Maps) had different versions for "N", and realized that the issue must be because of apps like that not being compatible with "M". Thankfully, I had a nandroid backup of the "M" (May security release) from before I installed "N", and I installed that - and then, updated to the June security release of "M".
Whew!
So, in summary, If you are rooted, take a nandroid backup in TWRP before you install "N" preview. If not, plan to factory reset your device after going back (or while going back) to "M" - no shortcuts.