You're right, bigger isn't always better in any audio application, and neither is more. Each application is different and we shouldn't make sweeping generalizations from past experience, hear say, or even factual information. Like Pat Brown says, "It Depends...".
The mounting of a driver, its location in referance to other drivers, and the overall acoustic environment all affect the sound of a system. Low frequency drivers are generally omidirectional until more drivers outputting the same frequencies are added. The interation between the sound waves may cause comb filtering or even cause frequencies to become more directional due to cancellations.
The environment also plays a role in the overall sound of a system. The absoprtion abilities of the materials in the environment and the angle of incidence due to driver placement could be the difference between a system that sounds "good" and one that sounds "bad".
