Well remember the first half of the series where Steven was a whiny kid who constantly screamed "GARNET! AMETHYST! PEARL!" to the point where you could make a drinking game off of it? Plus in the return and episodes after that he had to worry. Well maybe back more to roses scabbard as that was probably when Steven realized that the gems blamed him for rose.