Use fixed resistors to find the max resistance for an acceptable low volume. The value will be somewhere in the range 100 Ohm ... 330 Ohm. Once you know the max. resistance, find a pot with similar rating.
Or use one of those pots you've already tested. Find the position where the speaker volume is "off" to your ears. Measure the resistance of the pot from wiper to start in this position and select a pot with fitting resistance.
Harald