Does Gem / bust actually matter?

At the moment, I don’t recruit any busts at all.