Tell them, there is a time and a place for everything. If your so called friend can't take a joke, I don't think you should be friends with him, apologies then say something like I don't think we should hangout anymore or anything like that.
You should have just told him to stop the jokes. Making a comeback will make it worst.
And lots of people probably pick on him in that subject so he is not too happy about jokes like that.
It's just he probably said it in spit of the moment. Try to talk to him and straighten everything out and it will go well.
I wrote a longer post but I guess it didn't add.
