Hello,
we are also facing this problem regularly. There are some softwares, which are able to detect where inputs such as mouse movements are coming from. For example, Kaspersky is actively "blocking" mouse movements over their windows, if the mouse is moved by an application like input director instead of a physical device.
Maybe with such a technique it could be possible to implement an option, that the master does not switch to the slave if the input to do so comes from an application (e.g. TeamViewer). This solution would be software agnostic and would also work for other remote-control software.