The Swiss legend won a gold in doubles in Beijing in 2008, but has never been on the top of the medals podium in singles despite lingering frustration at knowing he should have been involved at least once.
"It's different pressure because the form is best of three so you feel like you have a bad five minutes and that could be it. That's kind of what cost me against (Tomas) Berdych in Athens, against (James) Blake in Beijing," said Federer.
"And then next thing, you know, you're out of both tournaments, and I think I was seeded 1 or 2 in either one of them."
Federer finished fourth in Sydney in 2000, was knocked out in the second round by Berdych in 2004 and lost a 2008 quarter-final to Blake.
"Obviously I'm very happy to represent Switzerland. I did way too good back in 2000 (where he finished fourth), didn't do so well in Athens.
"In Beijing, I was really close to a medal in the singles and then in doubles I got the gold," added Federer, who took the title with Stanislas Wawrinka.
"So I feel very relaxed going into the Olympics. I don't feel like this is a must win for me or anything like that. I probably would feel a bit more pressure if I wouldn't have won Olympic gold in doubles in Beijing."
On Monday, Federer kicked off his 50th straight major with a 6-2, 7-5, 6-3 win over Germany's Tomas Kamke at the French Open to equal Jimmy Connors' record of 233 Grand Slam match victories.
Federer is now 233-35 lifetime at the majors.
Monday's win was also his 50th against 12 losses at Roland Garros having made his debut as a wildcard in 1999 where he went down to Australia's Pat Rafter in the first round.
Since that time, he has missed just one Grand Slam -- the 1999 US Open where he failed to qualify.
"It's a big record, because that was longevity. Jimmy is obviously one of the greats of all time, and was around for 20 years," said Federer.
"This is my 13th or 14th French Open. It all started back in '99 against Rafter, and I have hardly missed any Grand Slams.
"I love the big tournaments. I have been so successful for such a long time and to already tie that record which is 30 years old is pretty incredible."
Federer, who is likely to set the new Grand Slam match record in the next round where he faces Romania's Adrian Ungur, now has his sights on another Connors record.
"He had 109 career victories, I'm at 74 now. Is it possible for me to equal Connors' titles? 110, that would be a round figure. That would be incredible. But that's a dream. I go year after year, and we'll see."
Despite Connors' standing in the sport, Federer admitted that his relationship with the great American has not always been close.
"He didn't talk to me much when he was working with Andy Roddick. He was quite distant at that time, which is a bit weird, because I got on very well with Roddick," explained Federer.
"But I think he respects me very much. He is much closer to Nadal than to me. Maybe that's the reason."