Louis van Gaal was left deeply frustrated after his Manchester United team allowed Chelsea to score a late equaliser as the two teams played to a 1-1 draw at Stamford Bridge.
Jesse Lingard's goal looked to have earned United a first win over Chelsea since October 2012, but Costa capitalised on Daley Blind's stoppage-time error to tuck in his seventh goal in nine games.
Van Gaal bemoaned his team's inability to "reward themselves" and hold on to their lead after looking like the better team for much of the match.
"Deep frustration because we were the better team. Again. And we didn't win. We had the better chances but we didn't win," Van Gaal told Sky Sports after the match.
"We played a superb game until the last quarter. Then we did not keep the ball in the way I think we have to. Then, we were unlucky and we get a goal against us. It is unbelievable. We don't reward ourselves.
"This is also because Chelsea is pushing with long balls and more aggression but nevertheless you have to keep the ball and be composed when you defend; you are not kicking the ball way without any meaning.
"Also we could have scored again after the first goal but you don't reward yourselves. When you play such a game against a team like Chelsea, you have to reward yourself."
Van Gaal admitted that the draw further complicates United's efforts to close the gap with the Premier League's top four, as they are now six points behind fourth-placed Man City and 12 behind leaders Leicester.
"When your aim is to be third or second or first then you need to win because the gap is already big," Van Gaal said. "The gap is now bigger than before so that's not good.
"The equaliser we conceded is also defensive organisation, you have to be composed and then you can play them out, not kicking the ball away. And then all the free-kicks. I don't think all the fouls were free-kicks but this referee gave a lot of free-kicks at the end of the game.
"The [injury] time was already over, the six minutes, and then he let [Chelsea] kick the corner and when we had a free pitch for us he whistles.
"He did that also in the first half. I think you cannot do that. Otherwise, you don't let him kick the corner, because I don't think that is fair for two teams."
Daley Blind, who was out of position for Costa's injury-time winner, maintained that United had played well despite the final scoreline.
"Of course we are disappointed. The late goal was disappointing but I think overall we played a good game," Blind said. "They were stronger for the last 15 minutes but overall we can be happy with our play.
"We played with a lot of confidence. We passed the ball quickly from side to side but we had some things we can work on.
"It is a crazy league but we have to look to ourselves, game by game."