Dating to between 1860 and 1940, more than 50 photographs depicting the impossible are on view in a new exhibition at the Rijksmuseum