One thing that does hold back the NSX is the lack of snob appeal. No matter how good of a performance car the NSX is or could be, it won't have the snob appeal of a Ferrari, Lamborghini, McLaren, etc. Many people that spend $100K+ on a car want that snob appeal because it's all about image for them.
It's funny. I took my NSX to a local EuroSunday event a couple of weeks ago, and out of the 55 pictures posted on the EuroSunday FB page, my NSX was in two of them (one direct shot and one shot of a Gallardo and mine)! The car got many looks and compliments, and I'm always getting compliments and thumbs up from people when I drive or park somewhere. The same happened at a Cars & Coffee event. So it's not a matter of people thinking the car sucks because it doesn't, as evidenced by the reactions.
I think it really does come down to snob appeal, or as some would say, brand appeal. But let's be honest, it's really snob appeal. When someone drives around in a Ferrari or Lamborghini, they know that people know they spent a ton of money on the car, so they must be someone special!