For one thing they were never marketed well here. And we never got the 'good' options which made them better cars than the ones we got.
However, the primary reason is cost. I recall a GT being listed at 9000 DM in the early 70's (new) in Germany. That car was around $3000 new here. Even without considering the exchange rate, that car would cost about $10k in 1970's dollars to sell in the US. Add inflation and we're talking a WHOLE lot more money. In fact, we'd be looking at BMW/Audi prices for the most part, with the current inflation, monetary exchanges, and equipment levels of modern cars.