Many reasons. For one, an oversquare engine is far better for higher rpms. Less side loading on the thrust face of the cylinder wall, lower piston speed, easier to increase the connecting rod ratio. For German autobahn driving, having an engine that can run at 6000 rpms all day is a plus.....without wearing it out in 20k miles.
But certainly metallurgy figures into the picture, the original CIH engine was brought out in '65. Modern materials, plus low-friction coatings, the need for low rpm torque, stricter emissions standards (small bore=less flame travel) not to mention oil technology....they all factor into the current trend of building square and undersquare engines. Oversquare engines are still the domain of purpose-built racing engines however.