The swingers startdown is the biggest hurdle. The startdown should be deliberate, positive, and heavy.
For hitters, I often use many different analogies to fix this problem.
1. I tell them that they cannot make the club move fast until after impact.
2. I use a music analogy that the club would sound like a crecendo. It starts softly and gets louder. (Slower to faster)
3. As a drill, I'll have them make the downsing last three times longer than normal.
4. Finally, I'll try to get them to start down with the entire body remaining still (zero pivot).
Try some of these things to see what gives you the best feeling and result.