Can you draw a schematic of the power supply showing how you connect and disconnect the power. Also show how you connect to the 5v from the USB which (I presume) should power the circuit when there is no mains-derived power.
Note that I am most interested in the regulation you've used and where you're switching power on and off. Have you tried switching between your regulator and the circuit to see if it makes a difference?
How do you prevent your power supply from feeding back into the USB power, and how do you prevent the USB power from feeding back into your power supply?
Can you also note if the disconnect happens when you turn off power or when you reconnect. Is it immediate?
