The Concorde...
was also a lot faster than the average passenger jet- and look where that ended up. Obviously speed isn't the number one priority amongst travelers. Safety, affordability and availability to the masses will always come first. This whole idea has already been tried in the civilian world and fell flat on its face... why would this be any different?