When I list the audio devices on the beaglebone, I see a couple of mcasp devices. they are listed at i2s as well.
These are defined in the Device Tree; see e.g.
this Adafruit PDF or
this BeagleBoard blog post.
Basically, a device tree is a description of the available hardware, provided to the Linux kernel at bootup. The source (.dts) is human-readable, and is compiled to binary form. Typically, you have one main device tree binary file, plus any number of
overlays (.dtbo). It looks like your current Device Tree already has a
mcasp0: { ... } node, so you only need to adjust it, i.e. create a modified overlay.
After looking at the documentation, it looks like only P8 pins 31, 32, 33, 34, 35, 36 and P9 pins 25, 27, 28, 30, 41B, 42B can be used as S/PDIF inputs or outputs; and that simultaneous HDMI audio and S/PDIF audio is not possible.
I did a quick search for an S/PDIF DT overlay for BBB, but could not find any. I could try writing one myself, but not having a BBB myself, it would be pretty annoying to try to write one "blind"...
I don't use the optical, just RCA, and have driven that with paralleled buffer ICs in the past.
I've only used the optical ones, plus misused TOTX173/TORX173 with Toslink plastic cables for other stuff (inverted high-baud rate UART, among other things). It's the electrical isolation that is most useful for me.
I would definitely not connect an RCA cable directly to any I/O pin on an SBC. The ground loop issues alone give me the heebie-jeebies.
At minimum, I'd use a digital isolator, say ISO7710 (1 channel) or ISO6720 (2 channel, same direction), which can also do 3.3V/5V level translation; those are only about 1€ in singles at Mouser, so definitely not a cost issue. (They also need just two 0.1µF supply bypass capacitors on each side, so they're easy to use.) The transmitter circuitry (optical or RCA) does need power, of course, so you'd also need something like a 5V-to-5V isolated DC-DC converter, bypass capacitors, and a linear regulator to something like 3.3V or whatever your transmitter requires (AIUI it is just 0.6Vpp). RECOM RFM-0505S is a very commonly used isolated converter (for up to 200mA output), but below 20mA load, like many others of its kind, its output voltage can shoot above 10V!
I have to figure out how to get Alsa talking to the mcasp pins.
As soon as you boot with a Device Tree overlay providing the sound interface, ALSA will pick it up.
USB S/PDIF cables or dongles are easier to use, because they implement USB Audio (UAC), and therefore use standard Linux USB audio drivers. Using one also provides some protection –– the USB port is much more rugged than GPIO pins –– but I don't think they generally implement electrical isolation.
Since 48 kHz 32-bit stereo is only 384 kbytes/second, a simple one or two S/PDIF output dongle will likely use only USB 2.0 Full Speed (12 Mbit/s, about 1000 kbytes/second in one direction), in which case you can isolate the entire dongle/cable using a cheap (~$10) ADuM3160-based USB isolator off eBay; these implement the ADuM3160 datasheet circuit, and only vary by which isolated 5V DC-DC converter they use. I often use these with microcontrollers and usb to serial cables, and they work just fine; I do check the datasheet for the isolator first, though (for maximum load, and that it has no minimum load and output is regulated). If the dongle has other features, it may use USB 2.0 High Speed, and thus require a high-speed isolator, which tend to be much more expensive.