To make matters worse for beleaguered manager Tony Adams, the Swans' 2-0 win came at Fratton Park and an on-loan Southampton player opened the scoring.
Collecting Jason Scotland's cute floated pass, Dyer chested the ball down and drilled it past James into the bottom right corner.
Things then got worse for Portsmouth moments before the break when Nadir Belhadj brought down Jordi Gomez in the area and Jason Scotland converted the penalty to make it 2-0.
Andrew Johnson and Bobby Zamora spared Fulham's blushes away to non-league Kettering, firing home late goals as the Cottagers won 4-2 at Rockingham Road.
It looked like it might be easy for the Premier League side when Simon Davies gave them a 12th minute lead with a sweet volley, but Craig Westcarr fired in a deflected free-kick 10 minutes before the break.
Danny Murphy restored Fulham's advantage but Westcarr quickly tied it up again with an 83rd-minute penalty.
But with the hosts dreaming of a replay, quickfire goals from Johnson and Zamora decided it late on.
Chelsea made hard work of Ipswich at Stamford Bridge before prevailing 3-1.
Michael Ballack opened the scoring as he turned in Ashley Cole's cross after quarter of an hour, but Alex Bruce levelled for the Tractor Boys when he stabbed home from close range when Chelsea failed to clear a free-kick.
Chelsea struggled to impose themselves, but went back in front when Ballack curled in a free-kick for his second before Frank Lampard's late goal wrapped things up.
Earlier, a quickfire double in the final minutes of the first half gave West Ham a 2-0 win over Hartlepool but it was not without controversy as referee Lee Mason gave the Hammers a dubious penalty.
Switzerland international Valon Behrami rifled in a shot from the edge of the area in the 44th minute and moments later Mason pointed to the spot when Michael Nelson handled the ball, albeit outside the area. Mark Noble made no mistake with the spot-kick.
Ian Ashbee's wonder strike capped a 2-0 win for Hull over Millwall after Michael Turner's headed opener.
Jobi McAnuff had a hand in all four Watford goals as the Hornets beat Crystal Palace 4-3 to go through.
McAnuff teed up Jay DeMerit and Jack Cork before Palace got back in it through Clint Hill early in the second half.
But the Hornets then made sure of progress as McAnuff crosses picked out Will Hoskins and Grzegorz Rasiak, making sure a late Paul Ifill double was not enough for the visitors.
A dramatic late goal from Martin Paterson earned plucky Carling Cup semi-finalists Burnley a replay at home against West Brom following a 2-2 draw.
The Clarets had led through Graham Alexander's penalty but West Brom looked good for the win as goals from Robert Koren and Kim Do-Heon turned it around before Paterson's late intervention.
High-flying Aston Villa face a replay after being held goalless away by Doncaster, while Sunderland and Blackburn must also try again after a goalless encounter at the Stadium of Light.
Middlesbrough left it late to beat Wolves at Molineux, with Marvin Emnes coming off the substitutes' bench to fire in the winner seven minutes from time.
Sam Vokes had earlier cancelled out Afonso Alves' opener.
Goals from Danny Webber and Lee Hendrie gave Sheffield United a 2-1 victory over Charlton and a ticket into the fifth round.
Charlton got a 69th-minute goal back through Chris Dickson but were well beaten.
Elliott Ward's late goal broke Torquay hearts as Coventry won 1-0 at Plainmoor.
Paul Scholes and Dimitar Berbatov struck within a minute of each other to see Manchester United into the fifth round of the FA Cup at Tottenham's expense.
Roman Pavyluchencko gave Spurs a fifth-minute lead but the advantage looked shaky long before Scholes levelled with a shot that flicked off Tom Huddlestone.
And within a minute, former Tottenham favourite Berbatov had belted home what turned out to be the winner, brilliantly controlling Michael Carrick's chipped pass with his first touch and steering his second into the corner.