My problem: I had hundreds of contacts in Google Contacts that I did not want to show up in Voice Search. At the same time, I had my Droid phone contacts set up to only display and sync with my corporate Microsoft Exchange server. The problem was that Voice Search would not find any of my Exchange /Outlook contacts, however, I could not get it to stop suggesting names of people it pulled from Google Contacts who I last spoke to 5 years ago.
I tried master resets and clearing caches, but nothing worked.
Finally, after a bunch of testing, I realized that Google Voice Search only searches those contacts that are in your Google profile. It searches everything in "My Contacts" by default, in addition to any other categories you select.
Solution: I exported my Outlook contacts to CSV, and imported them into Google Contacts as a new category called "phone". I then removed from "My Contacts all 500 or so contacts that were there. I set up synchronization to only synchronize the "phone" category from Google. I disabled my Exchange / Outlook contact sync.
This solution seems to work now. I do have to add any new contacts though to Google now instead of to Outlook / Exchange.
------------------------
Updating, because it looks like now that I have removed all my contacts from "My Contacts", the Google Voice search does not find any contacts at all. Maybe you have to keep the ones you want in "My Contacts". Unfortunately, Google gives you no easy way of taking one category, and putting all selected into "My Contacts". The Voice Dialer still works though, so I might just give up on this and use that when I want to make a call.