The Queen was not at home at the time, the Buckingham Palace said.
According to police, the man tried to enter via the palace's north centre gate at about 11.30 a.m. But he was stopped immediately. He was arrested on suspicion of trespassing on a protected site and possessing an offensive weapon and was taken to a London police station, BBC reported.
The north centre gate is the one furthest to the right in the commonest view of the palace (looking from the Victoria Memorial). It is the entrance to the palace which is in everyday use.
The knife was discovered when the man was searched, said police, adding that nobody was injured during the incident.
Last month, a man was arrested on suspicion of burglary, trespass and criminal damage after police said he had scaled a fence to get into Buckingham Palace.
The incident was believed to be one of the most serious intrusions into the royal residence since 1982, when the queen woke up on a morning and was taken aback to see a stranger in her room.