Territorial about litter box?

I've noticed that whenever i even scoop the litter box of our recently re-homed kitty (4yo Russian Blue male - indoor) he seems to get very skittish and territorial. He doesn't like me cleaning or scooping or even refilling the litterbox, or taking poo bags anywhere (whilst he is also there). But naturally he does expect it to be immaculately clean 24/7.

How normal is it for cats to be extremely territorial with their poo? i can't so much as scoop it in earshot of our little guy or he will go bezerk and try to follow me to see where i am taking his sacred poop.

I get it, like i get why they cover it, and that it's a vulnerability / safety / threat thing etc. I do wish that there was some way i could do it all when he isn't there, but he is literally glued to me so i have to just make the most of every minute he isn't to sort it out which is not ideal in terms of my own schedule. Alternatively do i just keep doing it in front of him until he chills out and realises it's ok?