It sounds like (literally) that you have discovered the wonderful, mysterious, tear-your-hair-out, world of ground loops. Entire volumes have been written and published on how ground loops are formed and methods to defeat them. As you have discovered, sometimes removing a "ground" connection actually helps. Sometimes it doesn't.
The problem is this: the little "ground" symbol is supposed to represent an equipotential surface. There should be
NO potential difference between any two points labeled as "ground," but in the real world there always is. It may be anything from barely measurable picovolts to several hundred volts, depending on the circumstances. For audio ground loops, the voltage levels are generally in the microvolt to millivolt range.
There are many types and many causes of ground loops, but the most often encountered are
"common mode" ground loops. A pretty good article that discusses ground loops, and how to prevent them, can be
read here.
Often the problem arises when heavy-current equipment, such as a motor, shares conductors, especially "ground return" conductors with sensitive electronics. Separating the two "grounding" paths will help to eliminate this common mode noise source. A so-called "star grounding" system is also mostly effective when properly implemented. Do some reading on this for more ideas, but trial-and-error is also an effective troubleshooting technique when pursued logically and systematically. Bypassing the power supply leads at, or as near as possible to, your electronic components is always a good idea.
And, yes, moving the circuitry from a solderless breadboard to a strip-board or even a full-fledged printed circuit board can often do wonders to reduce noise. At the very least it will look tidy.