The lighter the car and the larger its side area, the more easily it will get deflected by cross winds. However, I think the trick is for the front to get deflected the same as or less than the rear. If the car is designed so that the center of pressure lies at or behind the center of gravity, the car will either just get displaced laterally or actually kind of self correct because the nose will want to point a bit into the side wind. Arrows and missiles are both aerodynamic and stable – the two are not mutually exclusive.
I agree, it would be interesting how a VW XL1 reacts to cross winds and whether Volkswagen even paid attention to that when designing a slow car.
In my experience, with its nice long tail, our aerodynamic NSX is stable at 300 km/h. A colleague of mine drove my NSX at almost 300 km/h when it had a completely stock underbody and he said the car felt more stable at that speed than his BMW did at 250. I have found my NSX can get a bit twitchy at 300 km/h if the alignment is off, however.
Edit: just a thought – to improve crosswind stability, you could try playing with the alignment settings. When a crosswind hits a car, it doesn't just push the car at the front and back, it also tries to tip it. That reduces the load on the wheels facing the wind so the wheels on the downwind side carry a higher percentage of the car's weight. If you increase front toe-in, the load-bearing downwind front wheel will try to steer a bit into the wind. Theoretically, at least. I don't know how much of an impact that would really have.