Is there a downside to inviting a large number of athletes to join my campaigns?

There is no downside to inviting a large number of athletes. The more athletes you invite, the more traction your campaign gets!