...what are the downfalls of manipulated hands swing?
There is no downside to Manipulated Hands Swinging. In fact, Homer Kelley specifically discouraged True Swinging due to the terrific problem of Ball Location.
Are we, therefore, saying that any hand manipulation is best done at address as opposed to during the actual swinging process?