My situation, what's the smartest thing to do?

[Simba]

Well self defense isn't fighting for no reason, it's not only legal it's perfectly valid and acceptable, you can't let him push you around, it will just continue and others may even join in, seeing you as an easy target.

It's the sad reality of it.

Personally, I think you and your friend should work him over a bit to let him know you're sick of it.

Thank you for your advice, my friend also told me he want to beat him up one time since he got enough of it, like I do. So should we together fight him back?