Partly right, I just want to expand on this. If you take a dual-core @ 2.8ghz, and a quad version of the same chipset at 2.4ghz, the dual will be faster in anything that uses only one or two cores. Going from dual to quad wasn't nearly as drastic as single to dual because most system processes only use the first core (meaning the main app can be partially offloaded to the second). The problem with quad-core is after the second core, there's rarely any spill over unless it's designed (optimized, as you said), or you're encoding.
So, I'll take a cual-core phone over a single-core, but quad isn't a necessity for me at this point. Now, for my desktop, that's different, as I do a TON of encoding, and I use it as an internal web server.