Is my man secretly gay?

I have known my man for a long time, he has always been in long term relationships and likewise. We were both single for several years when we caught up again. It felt like we were meant to be, he became and still is my best friend, we can't go a single day without communicating (we are doing this long distance). The only problem is, we still haven't had sex. We are in our 40's. When we get together and spend the weekend or days at a time together, I'll initiate but he just can't get it up. He claims it's his condition but he can get it up and wank off several times a day when we have ph sex and he wants me to talk about other guys. It gets him hard. I've laid next to him in bed at night and just silently cried because I just can't understand how someone can show me so much love and affection, more than I've ever had from anyone in the past, and yet be so cold and indifferent when it comes to sexual intimacy with me. We've had several discussions about it where he will blame his condition(he could never get it to stay up in previous relationships either and would actually go years without sex with them) but surely not being able to get it up has nothing to do with wanting to sexually touch the person you are with(which he never has).. lately I've brought up the idea of swinging and that he should try being sexually intimate with a guy.. and it makes him hard. He tries to deny he wants to do that but then indulges in sex talk about it when goaded and he comes really hard. He likes talking about their 'nice hard' cocks.. he likes me to describe licking it and sucking it.. He swears he loves me, and I know he does, but I also know he's not sexually attracted to me in any way despite his denial of this. What do I do. Is it only a matter of time before he does switch to the other side? Will he just do it behind my back? And why would he want to stay with me when there's no sexual intimacy? I can't force him to want me in that way. It breaks my heart because I love him so much. But I feel like im cheating us both because I know what i need and I'm starting to think I know what he needs to, even if he's in fierce denial(he was brought up catholic and a mother who would probably be horrified and think he was going to hell if he came out). Any thoughts??