Aid groups and MPs have raised concerns over the move to prioritise defence spending over aid.