[Yoda]

...what are the downfalls of manipulated hands swing?

There is no downside to Manipulated Hands Swinging. In fact, Homer Kelley specifically discouraged True Swinging due to the terrific problem of Ball Location.