What do you want the caller to hear? Different ways of doing it will produce different results.
Most phone apps let you block a caller. I'm not sure what that sounds like to the person making the call (I should experiment).
Most also let you route a particular number to voicemail. At first this will seem like your phone was turned off or you were out of cover, so it works in the short term. Eventually they'll wonder why your phone is never connected and why you never return their calls though, so it's only short term.
If you just set a silent ringtone for them it will always ring out (from their point of view). Again, at first they may think that you just didn't hear it, but eventually they'll wonder why you never do.
As for them not suspecting, it really depends on what they know about phones and how long you want to do this for. If they know anything at all they will start to suspect eventually.