Good comments from GPG. I assumed that since the sensor output -0.5 to 0.5V it was already amplified by an op amp, but apparently not. So a buffer amplifier is required before the summing (or differencing) amp.
One more thing, though:
Even a rail-to-rail amp will not allow the input to go all the way to zero with a single supply. So you should either use dual supplies, or map the range to something not including 0, such as 1-5V if you prefer to use a single supply.
Bob