Since you mention POE as the voltage source, my experience is that various POE want a supply side voltage (and the voltage injection end of the POE circuit) and a minimum voltage at the far end of the POE circuit (to account for long runs of POE wire, and the resistance of the wire causing voltage drops). Do you have a good estimate of what the minimum voltage into the LM317 could actually be ?
n.b. I once set up something custom, to a device rated for maximum 24v POE, but from deep cycle FLA batteries, which were typically delivering 12.6-12.0v. It worked fine, because the POE device was designed for a minimum voltage of 10v.