"not with others"

everytime i tell my boyfriend i have something to do or someplace to go that isnt normal the first thing he says hto be with the only reason i am doing it is so i can go out with some other guy. when i ask him to stop all he will say to me is that its my problem not his and he doesnt have to change what he says to me. i dont want to leave him but i cant take having him think so little of me.