A potentiometer is not a good way to control the brightness of a LED. A potentiometer is not designed for comparatively high currents as required for a LED. Use the potentiometer instead to control a transistor which drives the LED current.
A modern way of controlling LED brightness is by using
PWM. Much less power dissipation at the cost of higher parts count.