I think gem is another word for god.
Peridot said in catch and release, "Help me get off this lousy, gem-forsaken planet!" which is another term for god forsaken. This would mean she would be a homeworld god, and the crystal gods, also Steven's half-gem half-human like a demi-god which is half-god half-human, and it is said gods came down to earth to have children.