The phone is closer to a computer than calculator. Imagine if after 2 years your computer stopped receiving updates and you were told to buy a new one.
I had a Note 3. I can't understand how you'd call the Note 4 recent. It's unfortunate to lose the updates but you can imagine how difficult it is to support so many iterations of hardware/software config because it all needs to be tested to satisfaction. Get a more modern Samsung and you'll feel the difference in performance - such a huge step up from my Note 3 to my S8+. Edge screen is nice for edge apps (quick dialer, calculator, etc.)