wierd starter problem

[wildside41]

Need some advice here guys. I have a 2001 police electroglide that lately doesnt want to start. When I hit the start button nothing happens not even a click. Wait a while and it will turn over no problem. i know it will happen because the engine light that usually goes out after the fuel system pressurizes starts to flash fast and then goes out. Now here is a wierd one. If I hold the starter in and wait sometimes it will just start up like normal. My model you have to have the clutch lever in to start could that switch be the problem? This start problem happens any time cold or hot. Just think its wierd that sometimes if I just wait 40 seconds holding the starter button down it will sometimes turn over. There is no dimming of the lights ect. When I hit the start button its just like its dead. Thanks for any info.