Zymkey binding lost with startx in rc.local

This is my rc.local

exec 1>/tmp/rc.local.log 2>&1 # send stdout and stderr from rc.local to a log file
set -x # tell sh to display commands before execution
su joe -c startx
exit 0

With startx in rc.local the blue led goes into overdrive and blinks continuously while if I remove startx line it works just fine i.e. normal single blink after every three seconds.

Commenting out other lines has no effect. Using Buster Raspberry Pi OS - 64 bit, lite version with added packages for x display.

Any clues?