I second what ob said. .40 can be "unstable"/ snappy. If you shoot it you'll know what I mean. .45acp is pure fun and kickass stopping power. Lol, 9mm is very stable and reliable.
One thing to consider when selecting caliber, especially with a carry gun, is the magazine capacity. A Springfield xds in 9mm holds 7 + 1, and in .45 holds 5+ 1. You may feel more comfortable with the extra rounds, but you may want the stopping power of a .45.
Me personally, I prefer 9mm. I like the extra rounds, and to me, 9mm kills bad guys too.