The click sound is the starter solenoid. It's always there, but normally drowned out by the starter.
It could be that the solenoid isn't making good internal contact, so try jumping it with a screwdriver and see if it starts reliably that way.
If not, check your wiring: check for tight, clean connections. Then voltage drop test. If the wiring checks out, the starter is going bad.