Once I had a bit similar problem in a complex circuit with micro controllers and amplifiers. To save energy, I wanted to put the amplifiers in standby mode, but I needed the processors running. It did not work.
The reason might be the same as for you and the solution could be easy.
When you remove the supply voltage, the internal (positive) rail of the circuit will drop below the positive input signal level. Inside the circuit you will have diodes that are normally reverse biased towards that rail and possibly other parts of the circuit.
Through the input, you will then "supply" a current into the circuit. That current goes wherever the path is open and can cause both strange behavior or even damage.
Just looking at the rail, could easily be understood that a strange behavior would appear when the IC would be powered with the input pulses!
My solution was to add a simple "resistive isolation barrier" between the two circuit groups.
In my case I put 470Ω in series with the input pins of the powered down circuits. That was low enough to keep the signals running without too much voltage drop and by slowing down the edges too much. It was also high enough to isolate the input signal from the powered down circuitry, without causing any negative influence.
So my suggestion:
Try to put some different resistors in series with SW3 and SW4
(for instance in the range between 10Ω and 4.7kΩ)
It's easy to test and might bring the solution