I wouldn't typically have both devices. S-Pen is always the "x factor" for me, regardless of the other phones on the market.
If I had purchased both, I'd sell the S5 for a premium price (which it would get right now, since it just got released) and then hold that money aside for the Note 4 and purchase that at the full retail (non-subsidized) price.
I wouldn't be able to go back to a phone with anything less than a 5.5 screen. And the S-pen will consistently be an "x factor" for me. Since my days with Windows Mobile (prior to what's now known as Windows Phone), most of my phones had a connected stylus. The only time I abandoned the stylus was when upgrading to the HTC Evo. Since I gave Samsung a try again with the Note 2, I don't know if I'd be able to go back to not having a stylus (especially a digitizer like the S-pen, which senses pressure).
