Trained LoRA of myself just creates someone who only roughly looks like me

Hey community :)

TL:DR: I'm in the process of making two LoRAs, one of my wife and one of me using kohya_ss. But I fail on getting a LoRA that actually gives results that really are her or me.

I have two sets with images of her and me and each one contains about 30 carefully picked pictures. Including full-body shots in different poses and not necessarily all front-view but also more or less from behind. More or less pictures that show each one of us from all angles. And in there are also pictures with different clothing and backgrounds.

I think this is the reason why the generated images are showing our bodies quite correctly. That's OK.*

In these sets of images there are also some portrait shots from different angles that also show her glasses and my beard style clearly. And of course the shape of our faces, our jawlines, the form of our noses and the color of our eyes. But the LoRAs I create don't give results that are actually us. The persons look more or less like similar people but they are not us. I have tested several checkpoints and created at least up to 50 pictures with them and there was not only one that got me or her right. (I did test the LoRA of her and the one of me separately.) It even fails on the our hairstyle, me beard, her glasses and the color of the eyes. **

(Side-note: One of my first LoRAs showed eyes that had an almost white pupil. This was because I had not completely compensated for the red-eye effect in the input images. After I went through the training set and made the pupil color much darker by hand and then ran the training again, this strange effect was gone. So I have witnessed the effect of changing the quality of the images used to train the LoRa on.)

I have tried more training steps and/or epochs but that gives me overcooked LoRAs without showing a change in the quality of reproducing our faces.

Would I need even more portrait shots? (And maybe take out some full-body shots instead?)

Oh, and I train the LoRAs on the standard 1.5 model. Training them on Photon does not seem to give better quality. Does this even have an effect beside that the LoRA just might work better with Photon?

* Edit: OK, now that I think about it, her breast size varies quite heavily in the pictures she takes. Sometimes it fits, sometimes they appear smaller, but most of the time they are too big.

** Second edit: It even makes the person on the created pictures looking really old sometimes, like in their 70s or 80s. Yes, we have laugh lines and such on a few of the input images. But the images created are in a completely different league. It's not even the same sport anymore... Sometimes the female in the created pictures looks so androgynous almost like a man. Why that? My wife has a slightly stronger jawline then some other women but this is ridiculous.