We're not talking about Windows or Linux, we're talking about Android. The same speed memory that's used for a swap partition is used for storage, so whether you're swapping the app in from the swap partition or the storage partition makes no difference in speed.
Remember, Android doesn't waste time moving an app and its data to another chunk of memory if it needs that piece that's being used. An app is responsible for always keeping its data stored. If the Android memory manager needs the space the app is in, it just kills the app. No swap partition, running on ANY class of card, is anywhere near that fast, regardless of the size of the app being killed or swapped. (It takes just as long to kill a 100KB app as it does to kill a 100MB app.) When you return to that app, it's loaded back, still has its data and you never notice that it spent some amount of time not being there.