Younger Americans who feel they have little hope of owning a home are making riskier financial decisions, research shows.