XL was going to be too big physically for me. I was coming from an iPhone, and wanted a bigger screen, the XE wasn't that much physically bigger, so plumped for that.
+ve for me on XE are expandable memory on XE not XL
same headset, same s/ware on both
+ve to XE 1.5GHz CPU, XL "only" single core
obviously bigger screen on XL, but again for me the physical size was a -ve to me.
try the "pocket" test, thats what I did, see them both side by side, and physically hold them in a shop, even the dummy units I would suggest. Apart from the above not much difference, to me anyway.