I loved the first GT, but skipped GT2 simply because it didn't seem as polished. GT3 was such a huge leap in visuals, I became a huge fan...plus my coworkers had setup a LAN for playing multiplayer races.
As far as the GT3 to GT4 comparision, I probably would have felt similarly if it wasn't for the fact I bought the Driving Force Pro (my first force feedback steering wheel) for GT4...and it made the driving experience feel so much more real and exciting. I also loved the sheer amount of cars and tracks in the game...and nothing in previous GTs compared to the sense of speed one got hauling ass on the Nurburgring especially with the DFP. After awhile, I didn't care that the AI still stuck to the driving line like glue or that you could get away with a few things you couldn't do in real life...the driving still feels more realistic than any driving game I have ever played (including the original Forza...haven't played Forza 2).
My biggest gripe with GT4 isn't the lack of crash damage, but more to do with the lack of online play...which had been promised and then dropped. And my biggest concern with GT5 is that the crash damage won't initially be part of the package (rumors suggest it will come as a downloadable add-on after the release)...which makes me question how they will handle car contact and penalties in their online competition modes.
GTHD is not really a demo of GT5...Prologue will be a more appropriate gauge. That said, if GT5: Prologue is anything like GT4: Prologue, there will likely be even more improvements to visuals and driving physics before GT5 comes out. The physics have definitely improved in GTHD, so they are obviously heading in the right direction.
I'm very curious how many cars will be on the track simultaneously in GT5. Screen shots already show as many as 8 cars on the track...I'm hoping for more.
In-car view is already confirmed for GT5...and, as you can see from the screens, the cars are quite detailed inside AND out. One story I read states that it takes 180 days to model each car!