i understand you...but if i were you i would try to not to focus in this small detail...
more if he is a good source of income....i would think that there is worst! i just prefer it that others bad manners.
Sometimes we focus on small things, yes can put you off but try to think...that it is just a word!
i do not like myself hun, darling, baby and blabla...and if he is a new punter is difficult that i am going to see him...but if a regular call me in a special way..i do not care! just do not listen or start to call him in a special way you too! LOL he will understand in this way!
try to joke about that with him!
these are just suggestion, of course if put you off really do not see him again!