High-mass dijet cross sections in photoproduction at HERA