I guess that's Google's Messages app (and the other is probably your service provider's app). I don't use that, but some message apps make different sounds when you are in a conversation than when the app is in the background (WhatsApp does this, for example). I think the idea is that when you are actively exchanging messages with someone you don't need a loud tone when a new message comes in, but of course if you just put the phone down with the conversation ongoing it may be insufficient.
Maybe check the app's settings and see whether there is any option to control this behaviour. No guarantee it will let you play a louder sound though: Google apps are usually amongst the poorest for customisation (Google always think they know best), and it's more common for apps to offer on on/off toggle for sounds when in a conversation rather than an option to choose what that sound is.
If the app won't let you change this I can see just 2 options: try another SMS app that doesn't do this (or lets you control it) or press the home button when you put the phone down (or maybe turn the screen off will do it?). You can go to the Play Store and post a bad review/complaint, but if I'm right about whose app this is they are not likely to take any notice.
(One problem with the "find another app" solution, which is my default solution to the inadequacies of pre-installed apps, is if you use RCS ("Chat") functions. Many SMS apps don't support RCS, so if you use that feature then it will limit what apps you can use.)