OEM skins are only cosmetic, they don't affect security or app compatibility and can be largely ignored (you can't do anything about e.g. Samsung hiding menu items in different places from Google, but since Settings has a search function that's not the usability problem it once was). Hence I don't see them as a fragmentation issue.
The combined WiFi/Data toggle is just another example of Google being rubbish at UI design. Sadly they always have been. That is one respect in which others not being bound to using their decisions is an advantage for Android.
I'm brand agnostic myself. I've never actually found adjusting to different UIs terribly difficult, so changes of version or brand don't really bother me, and I use very few pre-installed apps regardless of which brand of phone I'm using. So swapping brand actually has very little effect on me. That's partly because I can almost always find an app I prefer to the manufacturer's version (I don't rate either Google's and Samsung's apps), perhaps partly because I used to change ROMs every month and so built-in apps were never a thing for me, and partly because it's a personal policy never to become tied-in to a particular manufacturer and hence I prefer not to use proprietary apps or features. I'd say the only Samsung feature I used on my s21 is the "side apps", where there are 2 3rd party widgets I use, and the only Samsung apps I use are the dialer and contacts apps, which were also the only Google apps I used on my previous Pixel. Given that I also use a 3rd party launcher, really the only difference changing brands makes to me is the layout of the system Settings and the options for the notification slide.