Your circuit looks very strange, shouldn't the ground terminal of the scope be connected to ground.
I would not know how to calculate the values but R needs to discharge C fast enough to follow the wave form. The DC offset will affect the discharge rate and may allow 100% modulation to be detected without distortion.
I think if you started off with an RC of 5 times the 1/fm (modulation frequency) and see how you go. There is a lot of contention over what formula to use. So try this first and see how you get on.
Adam