So what happens? You find yourself back on the home screen, or in another app? Is it any particular app or apps that does this? Are you touching the tablet or interacting with it when it happens?
It sounds like either you are inadvertently touching something that returns you to a different application (the home screen is also an app, so I include that here) or something else is interrupting and taking over focus (which a different app should not do). But without more detail it's hard to guess.
You talk about the "square button", but is that an on-screen button or a physical or touch button in the bezel? (We don't know what device this is, so don't know what physical buttons it has). Just wondering because a square on-screen button would imply that you are using the Android 8-or-earlier 3-button navigation, as opposed to the Android 9 2-button navigation or the Android 10 gestures, but if you have actual off-screen buttons those could cohabit with the gestures, and the "back" gesture in Android 10 is a swipe from either side of the screen, which could be triggered accidentally. You should be able to check in your system settings.