Just a quick look at your code
; Initialise SIO A
LD A,$00
OUT (SIOA_C),A ; select regester 0
LD A,$18 ; D3 D4 command 3
OUT (SIOA_C),A ; Command 3 ( Channel Reset )
LD A,$04
OUT (SIOA_C),A ; select regester 4
LD A,$C4 ; 0b11 00 01 0 0
; D2 + D3 1 Stop bit
; D6 + D7 X64 clock mode
OUT (SIOA_C),A
LD A,$01
OUT (SIOA_C),A ; select regester 1
LD A,$18 ; 0b11000
; D3 + D4 INT on all RX
OUT (SIOA_C),A
LD A,$03
OUT (SIOA_C),A ; select regester 3
LD A,$E1 ; 0b11 100001
; D0 RX enable
; D5 Auto enable
; D6 + D7 Rx 8 bits char
OUT (SIOA_C),A
LD A,$05
OUT (SIOA_C),A ; select register 5
LD A,RTS_LOW ; RTS_LOW .EQU 0EAH 0b11101010
; D1 RTS
; D3 Tx Enable
; D5 + D6 Tx 8 bits char
; D7 DTR
OUT (SIOA_C),A
For simple testing, I would want simple polled mode for send and receive. This means that before transmitting a character you make sure that the transmitter output is empty before character output ( looks like your test code is doing this.
For receive you would check if there is received character before reading the character.
When your simple SIO output code is working, I would modify a copy of the test code to just echo received character to the output. Then you should see what you type on the PC echoed back to PC. A simple step.
=======
For SIO register 1 you are enabling Rx interrupts.
Might not matter when Z80 has disabled interrupts, but?
Then for SIO register 3 you have D5 Auto enable
This enables DCD & CTS to control receive and transmit enables. You will not transmit(CTS) or receive(DCD) unless the pin is low
Make sure that Z80 SIO pins 18,19 & 31 are all connected to gnd.
If you change this setting then 18 & 19 are just status bits in SIO register..
Just to summarize your code
.ORG $4000 should be RAM
.ORG $0000 should be ROM
SIO A data register is at I/O address $00
SIO A command register is at I/O address $02
DTR & RTS pins should go low when you write to SIO register 5 and stay low
Now this is not an easy design to test. The way the 74LS138 is currently connected only an address match is needed to make an output to go low. A read or write to memory of address range of XX00 to XX07 will also cause 138's pin 15 to go low for that memory access.
You do not have to have complete address decode for a computer to function. Said a different way now you should have many Address What SIO will respond to (SIO CS pin goes low)
$00 SIO A data register
$01 SIO A command register
$02 SIO B data register
$03 SIO B command register
$04 SIO A data register
$05 SIO A command register
$06 SIO B data register
$07 SIO B command register
To make testing easer I would make a change to 138's connections.
If you were to connect Pin 5 to Z80 IORQ instead of A7 then the outputs of the 138 would only go low for valid I/O access. The side effect is that more I/O addresses will cause a output to go low, but this is not a problem.
After change with the SIO connected to 138's pin 15 like now, the SIO could be accessed from some additional addresses, but you gain a great output to trigger your scope.
After the change the Address what SIO will respond to (SIO CS pin goes low)
I/O $00 SIO A data register
I/O $01 SIO A command register
I/O $02 SIO B data register
I/O $03 SIO B command register
I/O $04 SIO A data register
I/O $05 SIO A command register
I/O $06 SIO B data register
I/O $07 SIO B command register
I/O $80 SIO A data register
I/O $81 SIO A command register
I/O $82 SIO B data register
I/O $83 SIO B command register
I/O $84 SIO A data register
I/O $85 SIO A command register
I/O $86 SIO B data register
I/O $87 SIO B command register
The code should work fine.
After this change OUT ($08),x will cause 74ls138's pin 14 go low for a short time while the instruction is executed. The actual data does not matter. Your current code will cause two short low outputs. I would remove the second.
If you add a
OUT ($20) just before the POP AF then you will get a short low on 138's pin 12 just before sending a character to output register.
Note:
Don't rip out the SIO if you go back to 68B50. Add it to what you have so you could use it to test and get the SIO working. If the SIO's CS pin is high then it's the equal of NOT THERE.