I have the S5 and the Note 4.
FWIW, the screen is bigger on the Note 4 which is why I got it in the first place.
Other than that, I don't notice any real differences....
I am ignoring the S-pen in this equation, as I don't use it.
Actually, I do notice a difference: The Note 4 is much faster than the S5 and does not crash hardly at all.
Whereas, the S5 would lock up if I left a whole bunch of used apps floating in the 'recent tray'.
That little penchant has instilled a habit in me.
I do a Close All several times a day... and that had ended the crash syndrome....
yeah, I know: Android is supposed to do its' own housekeeping.... but I am here to tell you different. It don't do the job well enough to keep the phones happy.
The Note 4 will lock up sometimes if I let 15 or so old apps sit in the recent tray.
Easier for me to just keep it clean.... I have not restarted the phone in so long, I can't remember when it was.