If time travel is possible, then why haven't we been visited?

I know this topic has been addressed numerous times on the web and in an endless stream of science fiction books, movies and television shows, but I'd like to see if anybody out there has their own ideas. The idea is that time travel cannot be possible, because if it were possible then a working time machine would inevitably be engineered sometime in the future.

Once a time machine was in the hands of human beings, there would be all kinds of weirdness happening. What could stop us from traveling back in time to "fix" things to our advantage? Surely if time travel is possible then a time machine will exist in the future and someone will abuse it.

Given the amount of time in the future (theoretically infinite) someone traveling back in time to visit us is simply unavoidable if time travel is possible.

So why didn't we catch any people from the future taking their vacations in the Ming Dynasty?

Possible explanations for why this doesn't seem to have happened:

1. It has already happened, but the government has covered it up.
2. It has already happened, but a secret future agency of time cops has been traveling back in time and short circuiting the offending time travelers before they can commit their abuse of the past.
3. We don't have as much time as we think we do. The sun will go nova (or another global catastrophe) way before we think it will and there is nobody left on future Earth to travel back in time.
4. Time travel simply will never be possible no matter how far our scientific and technological advances take us.
5. Update: Greg Lange of greglange.com has suggested that it may only be possible to travel into the future, and not the past. Forward-only time travel certainly seems possible given the phenomenon of time dilation based on velocity close to light speed in the theory of special relativity (see the Wikipedia article on the twin paradox).
6. Update: Another possibility involves imagining that traveling back in time instantly shifts the traveler into an alternate universe exactly like our own, except with the traveler temporally located in a time in the past instead of his native time. This would explain why no one has ever seen a backwards time traveler -- we're all still in this universe. The time traveler doesn't travel back in time to our time, but backwards in the time line of a completely different universe.

What do you think?