My theory is that Homeworld is a caste system. It runs like this:
Diamonds are at the top
Other gems are like a poor or middle class
Pearls and Fusions are at the bottom. Pearl and Jasper speak of Pearl by implying that there's more Pearls. Pearl even says that she needs to serve someone to feel secure, which is why she served Rose. Garnet's reason for fighting against Homeworld was because Earth was the only place she could live freely. Maybe Fusions are seen as weak for this reason.
It kind of makes sense. In all of the Homeworld outfits, Diamonds are prominent. Even Pearl's space outfit in Space Race had a diamond on it. Jasper says that Fusion is a cheap tactic to make weak gems stronger, maybe a Homeworld belief, but Garnet says its a sign of love, Rose's belief.
Plus, you never hear much of the other Gems, like Peridot for that matter, because she's in the middle class.