Urgent care missed ankle fracture a month ago, now doc wants me to wear a boot!

Hello, and I am sorry about the long question. So about a month ago I dropped a chair on my ankle. The pain passed very quickly, but it got super swollen and bruised badly over the next few days. A week later I got tired of my family yelling at me to see a doctor, so I went to urgent care. Xray and "exam" were OK and I was told to use compression and rest. The bruising is now completely gone, there is no pain, but there is still some swelling. Today (a month after the injury) I saw my podiatrist for another issue, they noticed the swelling and x-rayed it. The doc said I have a fracture and gave me a boot to wear for 3 weeks. I can't imagine what 3 weeks of sitting around will do to my physical and mental health, so I questioned if it is really necessary at this point. I asked the Dr if it would just take longer to fully heal without the boot (with just compression and more rest), but could not get an answer. Any comments/ shared experiences would be greatly appreciated. Thank you!!