If you have big hands, you'll be fine. I do, and I actually prefer the slight bigger size because typing is a breeze and it gives me more to hold on to. Beyond that, though, I think people over-analyze something like the size when it comes to a cell phone. The X is bigger than a DInc or the upcoming Droid 2, but you get a lot of screen real estate which makes it more than worth it. I've been regularly watching videos on my X during my subway commute, which is something I never did on my Droid because of the screen size.
Let me put it this way: I think you need to think about how the size translates to a larger display and less about the physical size. If you're someone that will do a lot of web surfing, email, anything that would be made better/easier by 4.3 inches of screen, definitely get this phone.
I've been through a lot of phones in the past year (HTC Touch Pro, Palm Pre, Droid, HTC Evo 4G, now the X) and this is by far the best.