It's more than likely your shooting form - movement, trigger control, breath control - and not about flinching.
Offhand is a whole different world than off bench, and separates the sportsmen from the hunters (4 and 2 legged

).
Try this - shoot offhand, but brace the rifle against a tree, and see what happens.