England were 304 for four after tea, with Kevin Pietersen, the only batsman able to score with relative freedom, unbeaten on 131.
Alastair Cook (24), Andrew Strauss (37) and Jonathan Trott (35) all fell to disciplined bowling after lengthy but not particularly productive stays at the crease.
South Africa struck a crucial blow shortly before the tea interval when Ian Bell was caught at first slip off Jacques Kallis for 11, leaving Pietersen to rebuild the innings together with new cap James Taylor, who scored his first runs in Test cricket when he drove leg-spinner Imran Tahir past mid-off for four in the final over before tea.
Strauss and Cook had to combat some hostile bowling in a morning's play curtailed by rain, which stopped play for an hour and a half.
Philander made the breakthrough when Cook played around a straight delivery and was given out leg before wicket by umpire Steve Davis. Cook unsuccessfully sought a review which showed the ball was hitting his middle stump.
Strauss was caught behind off Dale Steyn after a two-and-half hour stay when he pushed at a ball from the fast bowler to edge a straightforward catch.
Trott batted for two hours before losing patience and flashing at Steyn to be caught at first slip. Pietersen showed attacking intent from the start of his innings and hit seven fours in reaching 43 off 83 balls by tea.
Brief scores: South Africa 419. England 304-4 (Kevin Pietersen 131 not out, James Taylor 38 not out