This is how I understand the circuit:
Assume any of the inputs A, B = LOW. Then the output OUT = HIGH. This will turn on MN5. MN5 will apply Vdd to the source of MN2. Presumably MN5 has a low conductance (being an NMOS connected to VDD), thus the source of MN2 will not be lifted up all the way to Vdd. Instead, it The voltage on the source of MN2 will rise only a bit above 0 V. This source voltage will increase the voltage necessary to turn on MN2 by the input voltage on input B and consequently on input A -> the threshold for a logic HIGH input is raised.
You can now perform the same thought experiment for the P-channel MOSFETs and the HIGH->LOW transition on the inputs MP5 will reduce the threshold voltage MP1...MP4 when the output is low.
An increased threshold for the LOW->HIGH transistion together with a reduced threshold for a HIGH->LOW transition (input side) makes for the hysteresis. The value of the hysteresis voltage will depend on the transistor parameters.