The Mineral Products Association says the decline threatens to impact the government's housing plans.