Mismanagement destroyed most of it.
When Congress took over passenger service in 1971 they also inherited thousands of miles of track. The story is the same nation wide, but here are a few local cases.
There are dozens of hiking/biking/snowmobiling trails across much of NH, like the Rockingham Recreational Trail, that are built on the rail beds of shutdown express tracks.
In most cases the feds allowed the salvage of the rails to help pay for development of the trails. Railways not directly taken over by the feds were often seized by local municipalities or illegally removed by developers. Several miles of express rail were illegally removed near the Massachusetts / NH border by developers preventing the establishment of a commuter train from Derry NH to Boston. Instead of rebuilding several miles of rail they are spending almost $1 billion to widen over 20 miles of I93.
A project to restart direct rail service along the seacoast from Boston to Portland Maine has been stopped because one city illegally ripped up the express rail in town for a walking trail. That particular stretch of track was still considered active rail by the feds.
