As long as you start the swing (up or down) with the hips, you have pivot-controlled hands, which not only transfers your weight to either side, but also releases "Lag", including the "Pivot" one (or zone I), and all you describes.
I can easily start my swing with my hips and yet direct my #3 pressure point at my aiming point or ball. Look at Hogan - I doubt he was a pivot controlled hands player. Not with his accuracy and length.