Re your code in post #19, there is an obvious problem: the
while(1) is inside the loop! I don't think that's valid C code; a
do without a corresponding
while() isn't valid syntax.
I would approach this problem using some debug code to output diagnostic information from the serial port. To start with, I would just output two decimal numbers in ASCII format, with a space between them and a carriage return and linefeed after. Transmit at 115200 bps and capture the data with a PC running a terminal emulator program.
The numbers should be res and pwm_duty. Generate the ASCII string and transmit it out the serial port each time round the loop. Then you can vary the input to the circuit and watch what's happening on the terminal. You should see the res vary over the range 0~1023 (your comment says it's a 10-bit ADC) and the pwm_duty value should vary over the range 0~255.
There's probably a bug with your code to convert R_IN and R2_IN to R_OUT and R2_OUT:
if (RD4_bit &&RD5_bit &&RD6_bit)
If the condition is true, you perform two calculations to set R_OUT and R2_OUT, but if it's not true, no values are assigned to R_OUT and R2_OUT.
Also, identifiers in ALL CAPS are traditionally only used for constants and #defined values; variable names are usually in lower case or mixed case.
You have done the conversion in a very complicated and inefficient way. Floating point operations are very expensive on the PIC. In fact, anything bigger than 8 bits (char or unsigned char) is expensive, but floating point is probably about two orders of magnitude more expensive!
If you simply want to take a 10-bit value in the range 0~1023 into a proportional value from 0~255, you just shift it right by two bits. You could replace all of your floating point calculations with that operation and see whether it fixes the problem. I assume this won't be a permanent solution because you will be changing the conversion once you get the simple version working.
When you do change your logic, I would strongly recommend that you avoid floating point as much as possible. If you do need to use floating point, try to get the compiler to precalculate as many values as possible. Your code in post #19 you are converting a 10-bit raw ADC value to an 8-bit PWM control value using the following steps:
1. R_IN = res * VREF/1024.0;
2. Conditionally, R_OUT = R_IN * .82;
3. pwm_duty = R_OUT * 51.2;
Step 1 requires at least one, possibly two floating point operations. The multiplication must be performed with a floating point multiply; the division (VREF/1024.0) could be optimised out by the compiler if it was smart enough to notice that VREF is a const, but I suspect it wouldn't, because VREF is actually a variable.
Step 2 conditionally performs another floating point operation, and step 3 performs another.
You can eliminate some inefficiency by only scaling once, and more by getting the compiler to precalculate VREF / 1024.0.
If you can't eliminate floating point entirely, there are several ways to improve efficiency. You could perform the scaling calculation using the raw value from the ADC, giving a floating point result in the range 0~1023 (assuming the scaling will be less than 1.0), put that back into an integer variable, and scale it from a 0~1023 range to a PWM control value (0~255) in the integer domain using two right shifts (value >>= 2).
Or you could scale your ADC raw values to floating point numbers in the range 0.0~1.0 by dividing them by 1023, perform the scaling, then translate back to PWM control control values by multiplying them by 255.
In all cases you should consider how rounding will be performed by the conversions to and from floating point.
Finally,
const unsigned short VREF = 5.00 is confusing, if not invalid syntax; short integers cannot have decimal places!
If you just want to apply scaling and offset, and don't need the MCU for anything else, this can be done with op-amps and precision resistors.