His question was a response to a more serious discussion regarding the age of the universe and its vast number of stars suggesting that extraterrestrial life should be common.
They then had a more serious discussion regarding the chances of humans observing faster-than-light travel by some material object within the next ten years, which one colleague put at one in a million, but Fermi put it closer to one in ten.
Fermi then made a series of rapid calculations using estimated figures (Fermi was known for his ability to make good estimates from first principles and minimal data). According to this account, he then concluded that Earth should have been visited long ago and many times over.
This discussion occurred in 1950 and Fermi's question regarding the conflict between an argument of scale and probability and a lack of evidence became known as the Fermi Paradox.
Part of the paradox, "the argument by scale", is a function of the raw numbers involved: there are an estimated 200–400 billion stars in the Milky Way and 70 sextillion in the visible universe. Even if intelligent life occurs on only barest fraction of planets around these stars, there might still be a great number of civilizations in the Milky Way galaxy alone.
(info gathered from http://en.wikipedia.org/wiki/Fermi_paradox)
This assumes that the Earth is not special, but merely a typical planet, subject to the same laws, effects, and likely outcomes as any other world.
If there is life on other planets, they would be in need of a Savior just as we are. I believe the Earth is special; created by a loving, intelligent God who wants to have a relationship with His crowning creation: humankind.
In the beginning, God created the heavens and the earth.
John 3:16 For God so loved the world, that He gave His one and only Son, that whoever believes in Him will have everlasting life.