Chelsea have been criticised for the way they handled their complaint against Mark Clattenburg by Alan Leighton, the national secretary of the referees' union Prospect.
And in an interview with BBC Radio Five Live, Leighton also claimed that Chelsea's behaviour could deter other dissuade players from reporting instances of racial abuse in the future.
Chelsea's complaint was based on Brazilian midfielder Ramires' allegation that Clattenburg used inappropriate language against team-mate John Obi Mikel.
But observers have been critical of the way Chelsea made their grievances against Clattenburg public, before it was later ruled that the Premier League referee had no case to answer.
"We think this is the wrong way to go about combating racism, and in fact what Chelsea have done could put people off making legitimate claims when they have been subjected to racial abuse," Leighton said.
Clattenburg has not officiated since the incident but he will return to match action on Sunday as fourth official between Tottenham and West Ham while he will be in charge for Southampton v Norwich on Wednesday.
"It's been terrible for him, he's been through a living hell," Leighton added. "He's had the press camped on his lawn. He's had to live with walking down the street and people either pointing at him or him thinking people are thinking, 'Oh, that's the racist referee'.
"Everybody always remembers an allegation but not everybody always remembers the outcome."
Chelsea have so far made no apology to Clattenburg for his ordeal, and noted in their statement earlier this week that the Football Association felt the complaint had been made in good faith.
Leighton believes Clattenburg wants to avoid a legal battle such a scenario and put the matter behind him. "If Chelsea want to make an apology, pay compensation and get Mark to sign an agreement saying he won't take legal action, I'm sure we can do that."
Meanwhile, Leighton hopes it is not too long before Clattenburg takes charge of another Chelsea game.
"I think if there was too long a delay in refereeing Chelsea then I think the issue gets dragged out again,'' he said.
"The key issue for Mark is getting back to refereeing and getting back to normal, and that means refereeing Chelsea as soon as he possibly can."
Information from the Press Association was used in this report.