You are correct. For that reason I would make dummy battery slug and not even opening the powered gizmo and not installing "or" tied diodes. If the capacitor is charged by USB or solar cell it may work when the battery is dead. Pull battery out, put in the dummy and look for replacement battery. Eventually things will run that way. There is energy harvesting research for IoT, so someone will get it done. I see some new car dashcam video recorders use capacitors to save file and shut down when the ignition is turned off.
Maxim makes some LDO voltage regulators with P-channel pass through for minimal quiescent current. It is just hobby right now but helps to learn new things thanks to support like you provide. Thanks.