Brain lateralization is commonly interpreted as crucial for human brain function and cognition. However, as comparative studies among primates are rare, it is not known which aspects of lateralization are really uniquely human. Here, we quantify both pattern and magnitude of brain shape asymmetry based on endocranial imprints of the braincase in humans, chimpanzees, gorillas, and orangutans. Like previous studies, we found that humans were more asymmetric than chimpanzees, however so were gorillas and orangutans, highlighting the need to broaden the comparative framework for interpretation. We found that the average spatial asymmetry pattern, previously considered to be uniquely human, was shared among humans and apes. In humans, however, it was less directed, and different local asymmetries were less correlated. We, thus, found human asymmetry to be much more variable compared with that of apes. These findings likely reflect increased functional and developmental modularization of the human brain.