Yes, apps are universal.
Speed issues? No, nothing like that at all.
Rarely do you see an app that is designed to care about a processor architecture - things like the Tegra Zone games come to mind (and then you simply install a nifty tool to defeat that dependency).
But speed? Never.
Some apps may need updating to work with the ICS (newest) version of Android found on the Galaxy Nexus, but most all are in process of that upgrade already and a great many don't need upgrading at all to work with ICS.