Thanks for the answers so far.
I realize my post was not clearly defining the context or applications I had in mind.
So, the goal is not to create a network of any kind - there are already perfectly fine solutions these days - but to include an optical data link of some kind for point-to-point communication between devices. I'm mostly interested in the physical interface part, the protocol is already mostly take care of and doesn't have to be anything standard. Also, it would not be just for one-shot projects, but for a more general solution that one could integrate in products.
SFP modules would certainly be the easiest path these days. But apart from cost (which admittedly can be mitigated, because yes one can find relatively cheap SFP modules these days, especially in quantities), they are pretty bulky, so anything smaller would be a great plus here.
TOSLINK and derivatives are definitely an option, actually. Of course this is a relatively simple solution that works only up to a few meters or a few tens of meters depending on the front-end. But Toshiba has bidirectional front-ends that work up to 250 Mbps up to 20 m, which would actually fit my requirements. Namely, the TODX2402 would have been a fit. Unfortunately, it's hard to get ahold of, and the last prices I've seen are not that competitive compared to the cheap SFP modules. So the only benefit here would be size (the TODX2402 is much smaller.) Power draw is also a factor. SFP modules tend to draw signifiant power - but I have not investigated this thoroughly yet.
I'll definitely have a closer look at SFP modules and what I can do with them. If anyone has another idea, or knows of SFP modules that are smaller than the typical ones, don't hesitate to chime in.