I can't tell you how any particular program calculates the checksum, but it's not going to be complicated. If it's an 8-bit checksum, it's usually the bottom eight bits of the sum of the data bytes. If it's a 16-bit checksum, or a 12-bit or 14-bit checksum in the case of a PIC, it's probably the bottom bits of the sum of the program words.
It should be calculated on the whole program address space, with words that are not defined in the hex file being assumed to be all 1-bits, so that it will match the checksum obtained by reading the program memory and adding all the words together.
It may be an exclusive-OR of the data, rather than a sum.
I suggest you write a small program (just a few instructions), load it into an emulator, view the program memory, add all the word values together, and see if the result matches the checksum reported by the tool.
The program memory in the PIC12F508 has 512 locations, each containing a 12-bit opcode. Say your program consists of just a NOP (opcode 0x000) at address 0. The rest of the program memory will be blank. So the program memory will contain:
0x0000: 0x000
0x0001: 0xFFF
0x0002: 0xFFF
0x0003: 0xFFF
...
0x01FE: 0xFFF
0x01FF: 0xFFF
So you can calculate the 12-bit checksum of those values by multiplying 0xFFF (4095) by 511, the number of blank locations. The NOP at 0x0000 is 0 so there's nothing to add. 4095*511 is 2092545. The bottom 12 bits of that value are 3585 or 0xE01. If the tool displays a checksum of 0xE01, then you know that's how the checksum is calculated, although I would try one or two other short programs just to make sure.
If the checksum is something different, you'll need to try some other ways of combining the data to see whether you can come up with the same number.
The PIC doesn't fit with the standard hex file format in several respects. The program space is 12 or 14 bits wide, instead of the 8 or 16 bits that the hex file is designed around. How is this handled? My guess is that each program memory location is allocated 16 bits in the hex file, and the top 2 or 4 bits are zero.
PICs have an RC oscillator calibration value programmed into the top word of program memory. This value isn't defined in the hex file, and it doesn't make sense to include it in the checksum either, because the calibration value is programmed into each chip at the factory and can be different for each chip, so if it was included in the checksum, different chips with the same code would have different checksums.
The PIC also has a configuration word which is not part of the program memory space, although it is defined by the program, and needs to be in the hex file unless the programming software uses another file that contains that information. If the configuration bits ARE in the hex file, I don't know how this is handled for the 12F508. For other variants, the configuration word pretends to be at some location outside the main address space; 0x2007 I think. Is it included in the checksum? I don't know.
So if you can't find documentation with a Google search, you'll have to figure it out for yourself using this information and anything else you can find.
