TBH it's not as bad as you'd think, but this is an incredibly subjective area. I'd consider the old iPhones way too small, for some people it's perfect for them. The best way to know for sure is to try one out, either in a shop or via a friend. Like yourself I used to own an S4, and upgraded to a Note 4, and was lucky that my GF owned a Note 2 so I could give it a proper test, see how it felt in my pocket etc. Or you could always buy a dummy Note 4 (the display models they sue in shops which don't do anything) which is identical in weight and size and cost £10-15 on eBay, see how it feels carrying it around with you.
For me personally (also male, wearing jeans, front pocket), standing up it it's fine and fits in my front pocket like the S4 did. Sitting down it's still fine but a bit more noticeable, and you might want to take it out.
There is the odd time when I have to use 2 hands but it's definitely usable mostly with one hand. Plus you've got options to resize the screen as you need it, change the keyboard for one handed use etc. Weight-wise there's no complaints from me, never felt tired using it for long periods of time or making calls.
Now I love it and couldn't go back to a regular screen size. If you don't mind the bigger size then I'd strongly recommend the Note 4.
Feel free to to ask any other questions

, you're right to be diligent before looking into a new phone