Tbh it depends on the device.
Like on my old galaxy s3 and nexus7 it makes a huge difference in speed of apps opening and even scrolling smoothness.
But on my new phone (Oneplus One) I notice no difference and (I know benchmark apps should be taken with a pinch of salt but) my phone actually scores lower with ART than Dalvik.
I say give it a try and see what you think. The worst that will happen is that some apps won't work but that's very rare now.
Don't be alarmed when you change it and the first reboot takes forever, it will boot eventually

Let us know how it goes and what phone you're on