Tottenham fans vented their fury at their side during an insipid display that allowed Wigan to claim only their second win at White Hart Lane. The home support were also furious at Villas-Boas for his decision to withdraw the in-form Defoe and replace him with Emmanuel Adebayor in the second half.
Defoe himself seemed upset at the move too, sluggishly sauntering off the pitch before grabbing his jacket and walking down the tunnel.
"For sure it is not something that is very pleasing to see," Spurs manager Villas-Boas said. "The most important thing for the players to realize is that we do things for the benefit of the team. We understand individual frustrations but we, as a team, are more important.
"The game wasn't going well for Defoe in our thought process. Maybe he could think the opposite but we don't take him off to punish him. We took him off to improve the team."
Adebayor was expected to be Villas-Boas' first-choice striker this season following his permanent move from Manchester City.
The Togolese, who was Tottenham's top scorer while on loan last season, is yet to start a Barclays Premier League game for the club this term, but it appears Villas-Boas may recall him for Sunday's clash against former team Manchester City.
There have been reports that Adebayor is unhappy at his lack of game time this season, but Villas-Boas insists there is no problem with the player's attitude, citing recent injuries as the reason why he has not started games.
"We're quite happy with his attitude," Villas-Boas said. "He had an injury which took him out for six games and then another injury took him out for two games. We have the injury records if you want to look at them.
"From the 11 that we've played, I think he's been out for eight of them and he wasn't here at the start of the season. In the games when he was available he showed the quality striker that he is. We have no doubts whatsoever. We have him fit now and hopefully ready for the next upcoming games. We know his ability."