Vidic brought down Villa striker Gabriel Agbonlahor inside the area and appeared to be the last man, but the defender was not even booked, though a penalty was awarded and James Milner converted it to gve Villa a 1-0 lead.
United went on to comeback and win the game 2-1 and O'Neill believes the decision changed the outcome of the match, admitting he could not understand the referee's choice.
"I think it would be universally accepted they should have been down to 10 men,'' O'Neill told BBC One. "There's no other decision to make. It's not a good decision by an otherwise fine referee.
"It's the major point of the match. In two or three weeks it's forgotten and the trophy's on their mantlepiece, but it's a poor decision.''
However, Vidic himself stated after the match that the referee had got the decision not to give him a red card spot on.
"I can react better but Agbonlahor did well," Vidic said. "I don't think it was a red card because I got in front of him, not behind. If I had got behind then yes.
"It's always nice to win big games but it was hard. They played with two forwards but I thought we deserved it.''
Despite the disappointing nature of the defeat, O'Neill backed his side to recover and book another Wembley date in the FA Cup final.
"It's hard to take,'' he said. "I thought we played splendidly, particularly in the first half, and got in front. Manchester United got the equaliser but at half-time I felt we were very, very capable of winning.
"I'm disappointed for the team but they're resilient, they're strong, they're young and they'll fight back. We felt we were capable no matter what side they played. We have to try and get back here again in May because it's a worthwhile journey.''
While O'Neill's anger with Dowd was clear to see, Villa defender Richard Dunne, who was at fault for United's equaliser, refused to blame the referee's decision for their defeat.
"It's very disappointing but we gave it a good go,'' he said. "It was a great start, that was what we wanted to do - to get in front and try and hold on. But I made a mistake and after that it was tough. Unfortunately it wasn't our day.''
On the possible sending-off, he added: "Possibly, but the game went on and it was an even game all the way through. They got a man sent off at Villa Park and it didn't make any difference. It was probably better to play against 11, it made it more enjoyable.''
Villa keeper Brad Friedel insisted Phil Dowd's failure to send off Vidic was a crucial factor. Friedel said: "We started the game really well but we have seen the penalty incident now and it really should have been a sending-off.
"You feel really hard done by after you have viewed it afterwards because you would have liked to be playing against 10 men. We feel a bit hard done by that we didn't play with a man advantage after that, especially on a pitch as heavy as that and cutting up like that.
"With 10 men, they would have really tired towards the end but it wasn't to be.''
Villa defender Carlos Cuellar echoed Friedel's sentiments and said: "For me it is a penalty and a sending-off. I think everyone says the same. You saw the sending-off of (Juliano) Belletti of Chelsea for something similar on Saturday. I think this was even more clear.
"But different referees make different decisions and you never know what would have happened had we had the extra man.''
Midfielder James Milner admitted producing a decent performance was no consolation for Martin O'Neill's team. He said: "We didn't play too badly but that is not what it is about and, in finals, it is about getting a result and winning the game.
"We came up against an experienced side who have played in a lot of finals and maybe that told in the end. But I don't think we did ourselves any harm. We put in a good performance, had chances.''