"Fast" is less light bounces than "Ultra" so you end up with deeper/darker shadows sometimes. It's not just cutting corners, but it can also be an aesthetic choice.
Ultra will always have smoother lighting though. So if all you're looking at is the smoothness of the light, then yeah... Ultra is the way to go.