I suspect there are 3 viable options:
1) Bit bang for 5-bit,
2) Pack your 5-bit data into 8-bit bytes, or
3) Pad your data to give 8-bits.
I suspect you are not interested in bit banging, but here's an example I got from Mike (K8LH) years ago in case you are curious of what it looks like in PIC Assembly. It can be adapted to 5 bit, but you will need to also get the receive end (e.g., Get232).
;******************************************************************************
; PUT232
; Put232 (9600 baud) -- Mike McLaren, K8LH (Jan-09)
;******************************************************************************
Put232
movlb 2
movwf txbyte ; save Tx data byte
movlw 10 ; 1 start + 8 data + 1 stop bit
movwf bit_ctr ; setup bit counter
bcf status,0 ; C = 0 (start bit)
DelayCy (104*usecs-10) ; 104 usecs minus 10 cycles(added)
goto SendBit ; send start bit
NextBit
DelayCy (104*usecs-10) ; 104 usecs minus 10 cycles
bsf status,0 ; always shift in a 'stop' bit
rrf txbyte,f ; put data bit in Carry
SendBit
movf LATD,W ; read port
iorlw 1<<1 ; set Ser_out pin bit to 1
btfss status,0 ; if data bit = 1 skip, else
xorlw 1<<1 ; set Ser_out pin bit to 0
movwf LATD ; precise update intervals
decfsz bit_ctr,f ; done? yes, skip, else
goto NextBit ; send next bit
retlw 0 ;
"DelayCy" is a macro. You will need to insert an appropriate delay for your baud rate.