(Sorry if this has already been made by other users, I'm always late.)
Okay, before you look at the title and already get ready to type a long paragraph about how I'm wrong or a link to this being already made, let me explain.
First, I'd like to mention that pearls could be servants to any Gem, with the possible exception of some (one noted Gem race that doesn't get pearl servants are peridots, since the now Crystal Gem Peridot states that peridots are lucky if they get their own pearl). Speaking of Peridot, and as stated on the Gems page, in "Back to the Barn", Peridot says that pearls are servants, and take orders and hold other Gem's stuff. I honestly see that the main Gems that would need their stuff "held" would be Quartzes and maybe just bigger Gems- which then would include jaspers, rose quartzes, amethysts, carnelians, possibly bismuths, and of course the diamonds. I can also possibly see sapphires having pearls too. However, due to rubies being "common" (and Eyeball exclaiming that she'll "get her own pearl" for defeating a rose quartz in "Bubbled"), it can be said that rubies don't normally have pearls.
Now that I've said that, let's take a look at two Gems that many suggest Pearl belonged two- Pink Diamond or White Diamond. I'm going to skip over "Pearl belonged to White Diamond"- due to the lack of appearances of White Diamond and lack of evidence that our Crystal Gem Pearl was hers. However, Pearl being Pink Diamond's is very possible...or is it? Let's take a look at our two known Diamond's pearls. We honestly can't say much about Blue Diamond's Pearl, since she had a very small window of screen time and didn't even talk in said time. Yellow Diamond's Pearl, however, despite also not having much screentime, is obviously much more confident and serves as an informant to her diamond.
Where am I going with this? Well, we all know that our Crystal Gem Rose Quartz allegedly shattered Pink Diamond. If our Pearl was like the other Diamond's Pearls, then why would she join the Crystal Gems after her diamond was killed? I know there could also be many possibilities, such as she was convinced by Rose to be a Crystal Gem, or she thought Rose was right the entire time, but take a look at Jasper. Jasper was also formerly one of Pink Diamond's followers, and as of "Earthlings" it was revealed that she is still devastated about her death and was trying to take revenge. Wouldn't it make sense that if Pearl belonged to Pink Diamond, she would also plot revenge? Even so, it does not look like Pearl has any plans for that, since it can be seen that Pearl ultimately cherished Rose and even had romantic feelings for her.
Now I know the argument will come up "But Pearl's space suit has pink diamonds at the toes! So does Sardonyx!". Well, I know how to answer that. Let's take a look at Peridot's outfit, there's a yellow diamond near the chest area, and two yellow diamonds over her knees. Yes, she was one of Yellow Diamond's followers, but was she a servant? No, that's a job for a pearl. Yes, Yellow Diamond sent her on a mission with Jasper and Lapis to check on The Cluster- but that still doesn't make her a servant or anything significant to Yellow Diamond. If you still think that makes her something special to Yellow Diamond, that is like saying the five rubies are also special to her (which they are probably anything but special, since they're so common), since they were also sent on a mission, which was to find Jasper.
Now the good and last part of this entire thing before my hand cramps, how does all of this add up to Pearl belonging to Rose Quartz? Well first, I said that the most likely Gem races to receive pearls would be quartzes- Rose is a quartz. On Rose's page, it says Rose trusted Pearl with information she'd never tell Garnet or Amethyst, which is apparent when Pearl gets upset that Rose withheld information from her. Like I also stated earlier, Pearl had romantic feelings for Rose, and never really got over them until "Mr. Greg" (possibly). So if Pearl was Pink Diamond's, why would she become so attracted to someone who shattered her owner? Finally, when "The Answer" took place, Rose and Pearl were the only members of The Rebellion. Therefore, I think Pearl in a way co-founded the Crystal Gems.
So like always, please add your input and tell me if I've made mistakes!