Do you take out the diaphragm when wiping out your M5 in between pumps?

I have the Momcozy M5 and use pump wipes in between a few of my pumps a day. I’ve been taking the diaphragm out each time and tediously wiping it down on both sides and then wiping down inside the cup where it goes. But I just realized maybe I don’t need to take it out and do that each time? Since the milk doesn’t get under the diaphragm, I really could just wipe it out while keeping it in the cup. Right? And then just taking it out to fully clean each night?

Additional question: if you have the M6 and/or M9, what are your thoughts on those? I heard the M6 is quieter than the M5 and both the M6 and M9 look more discreet, so I’m looking to buy one of those to wear at work. Thanks!