The Society of Black Lawyers (SBL), which is leading the drive for a black players' union to be set up in Britain, has threatened to report Tottenham to police over what it alleges is anti-Semitic abuse at White Hart Lane.
The organisation said it would contact the Metropolitan Police "if neither Tottenham FC nor the FA are willing to take a stand".
The society has put forward a ten-point plan to combat racism in football, which includes players being found guilty of racial abuse being sacked and banned for nine months.
It also wants referees to be given the power to call games off in the event that there is racial abuse from the stands.
Chairman Peter Herbert said he believed the use of the words "Yids" and "Yiddos" at White Hart Lane needed to be addressed.
Although the terms are widely recognised as insulting, some Tottenham fans chant "Yid army" at home games as an act of defiance against those who discriminate against the North London club's sizeable Jewish following.
But Herbert said: "It does not make a difference if it is Tottenham fans doing the chants or away fans - if they continue to do it we will report it to the police. There has to be zero tolerance and, if that catches out Spurs, then so be it.''
He said it was also not acceptable for Jewish fans to use the terms, and warned: "The report will be made if this behaviour does not cease by November 20. We will have monitors in attendance to observe what occurs."
A Tottenham official said the club's fans had often been subjected to taunts about the Holocaust, and stressed: "Our position on this topic is very clear - the club does not tolerate any form of racist or abusive chanting.
"Our guiding principle in respect of the 'Y-word' is based on the point of law itself. The distinguishing factor is the intent with which it is used, i.e. if it is used with the deliberate intention to cause offence.
"This has been the basis of prosecutions of fans of other teams to date. Our fans adopted the chant as a defence mechanism in order to own the term and thereby deflect anti-Semitic abuse. They do not use the term to others to cause any offence - they use it as a chant amongst themselves."
The statement said the club felt that "real anti-Semitic abuse, such as hissing to simulate the noise of gas chambers, is the real evil and the real offence".
It added: "We believe this is the area that requires a determined and concerted effort from all parties, and where we seek greater support to eradicate.''
Meanwhile, the SBL said it believed referee Mark Clattenburg should be suspended from officiating while he is investigated over Chelsea's allegation that he used "inappropriate language" towards midfielder John Obi Mikel.
The language he is alleged to have used is understood to have been interpreted as being racist. He denies the claim, which is being investigated by both the police and the Football Association.
A statement said: "The SBL expresses serious concern that the FA has not suspended [Clattenburg] in light of the allegations of racial abuse he faces.
"Suspension is commonly used in employment situations where allegations of misconduct are made... suspension does not signal any guilt at all."
Clattenburg did not referee a game last weekend. He will not do so this weekend so that he is able to concentrate on helping the FA and the police with their inquiries.
Information from the Press Association was used in this report