Not "dumb" at all, as many of us have been bitten by that one.
You can get to the same needed screen, either by tapping PHONE, then the CONTACTS tab; or, just tap the CONTACTS icon.
Once within that Contacts-focused screen, hit the menu key. Then, choose DISPLAY OPTIONS. You will see your gmail account address, preceded with a round silver arrow pointing to the right. Tapping that gmail addy opens a drop-down list, reflecting your groups. You'll likely see that MY CONTACTS is pre-selected. Just tap the group you want. You may need to scroll, as ALL OTHER CONTACTS is listed last.
Tip: if you want to use ALL contacts, and your list is long-ish, you may want to try freebie
StarContact. Makes it much easier to get to your desired contact, via a few taps per searching.
OR, go into the website for Google Contacts:
.............. gmail >>> Contacts (on left), or
..............
http://www.google.com/contacts
where you can rework your group assignments, including your favorites (Starred). Will save you some taps long-term.