I have had that experience too, where the other station is in a state of disbelief about my 5 watts. Often I get a power report from the other station, and I've been told of anywhere from 300 to 900 watts, and I'm doing just fine with my 5!
It all really comes down to propagation doesn't it? I've worked Australia a couple of times at 5 watts and once I was given a 599 RST, that's over 10,000 miles away, and with a simple wire antenna.
I love DXing at QRP. Sometimes I'm hard to read, but most of the time if I can hear the other station they can hear me fine, sometimes even better than I am reading them.
Honestly, I really do wish I had a directional antenna, as I've been working the same areas in Europe and North America over and over again. I really hanker to work more of Africa and Asia as well as Hawaii. But my time will come, I'm sure. I just don't have the space for radials or a place to put up a tower. :(