ASM has hit just about the right level of pain - what next?

[paulca]

The more annoying and harder to fix issue is UART overruns.  Oh I hear you fancy modern day MCU devs nodding, but serious, you have DMA and a FIFO you have NO EXCUSE.  I have one byte,  If I dont read it before the next frame starts I loose one.

The irony ... if I don't look at the OVRN flag I only drop 1 byte out of 10,000s.  If I check the OVRN flag I drop 1 in a 100.

Can you generate an interrupt when a byte has been received?  Then the ISR could read in the byte and store it in a circular buffer.

At 9600 baud and 1.8Mhz I doubt it.  That's the catch.  You would think, lets use a FIFO, but the time and instructions pushing and popping the FIFO are in the critical path as well.

What I did was shorten the protocol I was using to 3 byte bursts.  I hot loop the CPU on "char ready", consume 3 then go and "parse them", then I send "ok" and let the other end send 3 more.  So in the critical path are basically a memory write and a pointer advance, check if newline.... and it will still drop the odd byte.

If you went down to a single byte you probably wouldn't lose any.

[Analog Kid]

On a different note, without leaving ASM, I just got more confident and more adventurous.

When it came to parsings command-line I didn't let myself think, "Oh shit, string utils!", I just started writing them.  Turns out you can get quite "nifty" in ASM with strings...

Yes you can. Welcome to the wide, wild world of ASM.

Suggestion: To parse text, rather than using a bunch of string-decomposition functions (instr(), substr() and friends), consider doing what I do:

Implement a FSA (finite state automaton). Actually very easy to do in any assembly language. Create a state table with pointers to stub routines. Process the text to be parsed character by character. At each row of the table, you jump to the routine corresponding to the character just seen.* (You can even jump based on classes of characters, like numeric digits.)

Works like a charm, is generally smaller than iterative code, and is very easy to tweak because it's data-driven.

The first requirement here is to draw out the parsing (actually tokenizing) sequence on paper. The code & data then follow directly from that diagram. It's like magic.

* This part of the scheme is helped immensely by the use of the REP SCASB x86 instruction, which you lack in your instruction set. You'll have to code this up discretely, but that shouldn't be too hard: just construct a list of characters to be used for comparison (all your tokenizing chars.) and loop over that list until you find a match to the current character (or don't).

[paulca]

I only wrote what I needed.  Keyword and hex.  In pairs, str_scan_hex (and it's aggregates, hex8, hex16) with str_print_hex.  No "greedy" operators, even str_cmp takes a length.

The Z80 may have served it's purpose, a stepping stone, I have a few more things with it, then I might move onward to the 68000.

Thinking about the UART rate.  To get 9600 baud I divide the clock by 192.  So in theory I should have something like 1920 clock cycles per byte/frame. 

The Z80 averages something like 4-8 clocks per instruction with some taking way longer, but "in theory" you should be able to shift bytes easily fast enough with a ~240 instruction budget.

So maybe my byte loss cause is else where. 

When I upload a program of maybe 300bytes in the format

>AF\n
<ok\n
>A9\n
<ok\n

It will occasionally miss the \n and stall the state machine.  It's easy to reset with a further character which will reset the state machine, but it happens maybe 1 in ten times 

I suppose with old NMOS chips it could be noise or anything tripping it out.

1 in ten times sounds awfully familar to loading stuff on tape back in the day.

[Peabody]

Sorry, I haven't been following closely enough.  Are you bit-banging, or do you have an actual UART chip?  Which chip?

How about a a lower baud rate - 4800, or even 2400?

[woofy]

The Z80 may have served it's purpose, a stepping stone, I have a few more things with it, then I might move onward to the 68000.

I felt the same a while back, so I built this:
https://hackaday.io/project/177988-68k-mbc-a-3-ics-68008-homebrew-computer
However it was "just another processor" without the magic that came with the Z80. My first ever computer was a Nascom1 built from a kit. I did nothing useful with it but had a huge amount of fun doing so. I programmed it in assembler (hand assembly on paper to start with) and grew to love the Z80. Nothing since has come close to the awe and wonder of those days.

I guess nostalgia isn't what it used to be.

[paulca]

Sorry, I haven't been following closely enough.  Are you bit-banging, or do you have an actual UART chip?  Which chip?

How about a a lower baud rate - 4800, or even 2400?

It's a period Z80 DART IC.

It's not a big deal a byte dropping here and there.  It will just incentivise me to write a timeout.

[paulca]

The Z80 may have served it's purpose, a stepping stone, I have a few more things with it, then I might move onward to the 68000.

I felt the same a while back, so I built this:
https://hackaday.io/project/177988-68k-mbc-a-3-ics-68008-homebrew-computer
However it was "just another processor" without the magic that came with the Z80. My first ever computer was a Nascom1 built from a kit. I did nothing useful with it but had a huge amount of fun doing so. I programmed it in assembler (hand assembly on paper to start with) and grew to love the Z80. Nothing since has come close to the awe and wonder of those days.

I guess nostalgia isn't what it used to be.

It's not nostalgia I was after really.  It just so happens the two CPUs I used as a kid where the Z80 and the 68000.

The last thing on the plan for the Z80 is a 3v3 IO Playground.  Bringing the board stack to:

Core CPU + IOToyBoard (period), 3v3IOPlayground (modern).

The 3v3IOPlayground is looking like a first version with just level shifters and a socket for an Ice40 FPGA "Upduino", itself with 8 IO.  The FPGA is there to open the door in the floor to go "down" if I wish.  Starting with replicating the period Mode 2 interface and maybe a custom UART/SPI/I2C affair, or if I get a bit bogged down I can switch to the IP Blocks for the same in the FPGA.

There is temptation to respin both of the first two boards with upgrades and move the FPGA and it's shifters to the CPU core board so I can do "MAXED out Z80" with stupid memory paging etc.

It shouldn't be wasted, because I think the 68000 is probably capable of where I want to play next.  Fully abstracting the memory away from the CPU, via a bus master controller which owns the memory and the CPU becomes a client.

The Z80 playground with the FPGA or re-doing the Z80 CPU core board to do simple memory paging would be a brilliant pre-cursor learning venture before tackling some more involved with TLBs and page faults.

I'm tempted to ask about the PIC, but then I decided.... I don't want to know.  No spoilers!

[Analog Kid]

The Z80 may have served it's purpose, a stepping stone, I have a few more things with it, then I might move onward to the 68000.

I felt the same a while back, so I built this:
https://hackaday.io/project/177988-68k-mbc-a-3-ics-68008-homebrew-computer

Took a look at that. Very interesting.
But weird: why a PIC in addition to the 68000?
Couldn't the guy figure out how to make a computer with just the 68K and some glue logic?
Or would the extra chips have been too much of a hassle?

[brucehoult]

The more annoying and harder to fix issue is UART overruns.  Oh I hear you fancy modern day MCU devs nodding, but serious, you have DMA and a FIFO you have NO EXCUSE.  I have one byte,  If I dont read it before the next frame starts I loose one.

The irony ... if I don't look at the OVRN flag I only drop 1 byte out of 10,000s.  If I check the OVRN flag I drop 1 in a 100.

Can you generate an interrupt when a byte has been received?  Then the ISR could read in the byte and store it in a circular buffer.

At 9600 baud and 1.8Mhz I doubt it.  That's the catch.  You would think, lets use a FIFO, but the time and instructions pushing and popping the FIFO are in the critical path as well.

Really?

I guess the interrupt routine will be something like:

Code: [Select]

PUSH AF               ; 11 T-states
PUSH HL               ; 11 T-states
IN A, (UART_STATUS)   ; 11 T-states
BIT 0, A              ; 8 T-states
JR Z, INT_EXIT        ; 7 T-states (not taken, data present)
IN A, (UART_DATA)     ; 11 T-states
LD HL, (BUFFER_PTR)   ; 16 T-states
LD (HL), A            ; 7 T-states
INC HL                ; 6 T-states
LD (BUFFER_PTR), HL   ; 16 T-states
POP HL                ; 10 T-states
POP AF                ; 10 T-states
EI                    ; 4 T-states
RETI                  ; 14 T-states

That's 125 clocks. Plus 13 to enter the interrupt handler (mode 0 or 1) and 4-23 to wait for the previous instruction to finish. So 161 total.

But that's not wrapping the buffer pointer. Ok, Make the buffer 256 bytes, aligned. So just use INC L instead of INC HL. It's even 2 cycles quicker. LD (BUFFER_PTR), L would knock 6 more cycles. So 153 total.  38.25µs at 4 MHz.

At 9600 BPS the bit time is 104µs and the byte time (8N1) is 1042µs.

The thing is more than fast enough to use a simple GPIO and interrupt on every bit!!  Let alone using a UART that buffers a character while it's receiving the next character. Even a '595 shift register can do that if someone tells it when to clock the just-received character into the output latches.

In fast it looks fast enough (just!) to do the 230.4 kbit/s speed Apple used with the Zilog SCC, purely interrupt-driven.

Oh wait, you said 1.8 MHz not 4 MHz. So the 38.25µS becomes 85µs. It's still fine.

[paulca]

Code: [Select]

; Expects str_buffer write pointer in HL
; Expect max len in BC
; Returns   HL = next free
;           BC = buffer remaining
; On error or empty read:
;           HL = unchanged
;           BC = ignore
dart_a_read_str:
    PUSH HL ; save the buffer start addr
.loop:
    ; char = getChar();
    CALL dart_a_rx_byte_blocking
    ; immediately check error flags
    ; CALL dart_a_rx_error_flags
    ; JR NZ, .bad_entry
   
    ; *write_ptr = char;
    LD (HL),A
    ; if ( char=='\n' || char =='\r' ){ end_entry(); }
    CP 0x0D ; carriage return
    JR Z, .end_entry
    CP 0x0A ; carriage new line
    JR Z, .end_entry

    ; save A
    LD D,A
    ; if length2go == 0
    DEC BC   
    LD A,B
    OR C
    JR Z, .bad_entry

    ; next char, restore A (the input), echo
    ; This is down here so we don't echo the CR/LF nor increment past it
    INC HL
    LD A,D
    CALL dart_a_tx_byte_blocking
    JR .loop ; while(true)

And the inner wait loop:

Code: [Select]

dart_a_rx_byte_blocking:
.wait_rx_ready:
    IN A, (DART_COMM_CHANNELA)      ; status register
    BIT DART_REG_RD0_RXCHAR_AVAILABLE_BIT, A
    JR Z, .wait_rx_ready   
    IN A, (DART_DATA_CHANNELA)
    RET   

I expect it might be more to do with signal integ maybe.  While everything else is clean the UART lines have a wide blur of noise on
them on the scope.

The other part which isn't uS long is

STOP->START

With a 2020 FTDI Quad UART blasting at it, it will not pause a microsecond between frames.  The DART chip is not expecting that.

So I might try adding a second stop bit.

https://gitlab.com/paulcz80/z80_asm/-/blob/master/os_design/dart_driver.asm?ref_type=heads

EDIT:
I see it. 

    CALL dart_a_tx_byte_blocking


Tx and Rx run at the same baud clock.  So the fact I enter into a "TX Wait loop" for the echo is spending ALL my budget.

This could possible by converted from:
    CALL dart_a_tx_byte_blocking
To
    CALL dart_a_tx_byte_unblocking

Where the byte to Tx is placed somewhere, interrupts turn on and an ISR actually sends it when it can.

Further edit:  There was a "I'll do that later" with this code, "ECHO" is meant to be configurable and DISABLED during a data transfer.

If I had implemented that, there wouldn't have been an issue.  With echo still enabled, it causes a lot of waste with TX wait loops and the other end (Python) has to keep discarding the echos.

[paulca]

As always though.  The "gotcha" with ISRs.  I have been afraid to do this for example:

isr:
   IN A,(SOME_REGISTER)
   ; some more processing

I have written code like that, but it has never survived the first debugging pass of non-working code before being the first to get removed and split between ISR and main thread via signalling.

It is especially dangerous as while you are in the ISR and until you call "RETI" the mode 2 peripherals will still consider themselves "in interrupt service".  I expect they are not fully functional in that state and reading writing registers on them may be unwelcome.  You are also holding the daisy chain state for other peripherals to interrupt... and if they are up the queue they are permitted to nest.

To split Rx and Tx into interrupts that means the main thread has to fork based on the interrupt signal.

; consider DJNZ for timeout in BC
.loop:
   ; check Rx ready ISR signal
   JP .NZ rx_routine
   ; check Tx ready ISR signal
   JP .NZ tx_routine
   JR .loop

[DC1MC]

I'll just leave this stuff here:

"UZIX is a UNIX Implementation for MSX machines, based on UZI written by Douglas Braun. It is a multiuser, multitask operating system, implementing almost all 7th Edition AT&T UNIX kernel functionality."
https://sourceforge.net/projects/uzix/


https://www.hartetechnologies.com/manuals/Cromemco/Cromemco%20Cromix%20System%20Administrators%20Manual.pdf
https://github.com/udo-munk/z80pack/blob/master/cromemcosim/cromix
https://manx-docs.org/mirror/harte/Cromemco/


Cheers,
DC1MC

[brucehoult]

To split Rx and Tx into interrupts that means the main thread has to fork based on the interrupt signal.

; consider DJNZ for timeout in BC
.loop:
   ; check Rx ready ISR signal
   JP .NZ rx_routine
   ; check Tx ready ISR signal
   JP .NZ tx_routine
   JR .loop

This is not interrupts, it is polling. That is how you lose input characters if the main loop does any significant additional work.

[paulca]

To split Rx and Tx into interrupts that means the main thread has to fork based on the interrupt signal.

; consider DJNZ for timeout in BC
.loop:
   ; check Rx ready ISR signal
   JP .NZ rx_routine
   ; check Tx ready ISR signal
   JP .NZ tx_routine
   JR .loop

This is not interrupts, it is polling. That is how you lose input characters if the main loop does any significant additional work.

The main loop is doing nothing.  That IS the main loop.

It could be rewritten without interrupts as:

READ status register
IF rx_ready:
   do_rx
IF tx_ready:
  do_tx
CALL do_something_else
LOOP

do_something_else: 
// There is little to no point in being here, we have sweat FA time to do anything useful.

Right or wrong, I fail to see the point of interrupts for tightly coupled high flow streams.  There just isn't enough time to do anything useful other than track the flow state and handle errors and terminators.

Interrupts are for when waiting would be stupid.  Like waiting on human input, if you have something else to do, be it flash the cursor.

Or updating the SYS_TICK if you want that luxury, then it makes sense to be "interrupted" as you can't tell when it will be only the timer knows.

If you look at similar routines in ZXSpectrum land or in CPM.  The most you will get is fancy borders updated or maybe a prompt flashing after you execute a large IO op.  eg LOAD ""  Your interaction options are "BREAK" or wait.

[tggzzz]

To split Rx and Tx into interrupts that means the main thread has to fork based on the interrupt signal.

; consider DJNZ for timeout in BC
.loop:
   ; check Rx ready ISR signal
   JP .NZ rx_routine
   ; check Tx ready ISR signal
   JP .NZ tx_routine
   JR .loop

This is not interrupts, it is polling. That is how you lose input characters if the main loop does any significant additional work.

Precisely.

As you say, the sane way to do it is the ISR puts the character into a FIFO (e.g. implemented as a circular buffer), and the main loop reads the buffer when it gets around to it - and then processes the character.

That works well for bursts of characters. Naturally if the average time to process a character is longer than the inter-character arrival time, you are screwed and there is nothing you can do about it.

For transmitting characters, that ISR tells the main loop another character can be sent either via a different FIFO or by putting a magic flag into the FIFO. Alternatively the main loop simply polls the UART to see if its tx buffer is empty.

[paulca]

In my case.  I receive 3 bytes. Parse them, receive 3 more.

If the other end is not waiting, I will miss because it takes me longer to parse 2 bytes of ASCII into hex than it does to receive a character, apparently.

I will try the FIFO.  However, I am 100% sure I cannot parse ASCII Hex as fast as an FTDI can send it.

EDIT:
It does raise the question, why not just send raw bytes.  The only answer I have is because it's fairly standard to send ASCII hex as it's "safe" and you can intervene or manually type it.

If I made the data transfer more specific, or used the other UART channel dedicated to it, that might help me remove that need to parse entirely and I may possibly be able to keep up with the FIFO.

[paulca]

To spice it up even more.  I don't think I even want the UART loader.  I already have a functioning (or had until I dismantle the breadboard)  "parallel IO" loaded with an STM32, using the PIO dedicated signallying READY/STROBE.

It was significantly faster and easier.

[tggzzz]

I will try the FIFO.  However, I am 100% sure I cannot parse ASCII Hex as fast as an FTDI can

Average or peak?

A FIFO can help with one but not the other.

[paulca]

I will try the FIFO.  However, I am 100% sure I cannot parse ASCII Hex as fast as an FTDI can

Average or peak?

A FIFO can help with one but not the other.

Average.  It's a loader, which is why, if you think about it, a FIFO or any buffer isn't really going to help, except with the parsing.  It's job is to copy things to memory anyway.

A FIFO into the parsing routine of a fixed length.
Upstream can then send "FIFO.length / 2" burst.
Interrupt clocks bytes into FIFO.
When FIFO.not_empty it can signal the parsing routine to begin.
It can also tell the up steam to send another burst.
Then we ping pong around the half FIFO, consume half, ask for another burst until done.

[paulca]

Even if I don't use the UART for "loading programs", it's other task I had planned was to integration with an ESP32 Comms module in default "AT command mode".

If I ask it to make a REST or Web request for me, it's going to dump whatever comes back over UART.  I don't believe it has a back signal.  It will send 2048 byte "lines" of content until it's done.

So not large requests from the Z80.  Even at that, it's going to arrive in a burst.  I need the code to be able to handle a sensible burst size that isn't 3 bytes.

[tggzzz]

I will try the FIFO.  However, I am 100% sure I cannot parse ASCII Hex as fast as an FTDI can

Average or peak?

A FIFO can help with one but not the other.

Average.  It's a loader, which is why, if you think about it, a FIFO or any buffer isn't really going to help, except with the parsing.  It's job is to copy things to memory anyway.

Then you have to reduce the amount of data transmitted, or to implement flow control either with hardware (UARTs CTS RTS) or software (XON XOFF)

[Peabody]

In my case.  I receive 3 bytes. Parse them, receive 3 more.

If the other end is not waiting, I will miss because it takes me longer to parse 2 bytes of ASCII into hex than it does to receive a character, apparently.

I will try the FIFO.  However, I am 100% sure I cannot parse ASCII Hex as fast as an FTDI can send it.

EDIT:
It does raise the question, why not just send raw bytes.  The only answer I have is because it's fairly standard to send ASCII hex as it's "safe" and you can intervene or manually type it.

If I made the data transfer more specific, or used the other UART channel dedicated to it, that might help me remove that need to parse entirely and I may possibly be able to keep up with the FIFO.

ASCII hex is safer only if it doesn't lead to dropping bytes.  A while back I did an SD card bootloader for MSP430, and ended up reading in binary from the card, which had been prepared from hex input by a Windows console program.  It saved a lot of resources on the MCU to be able to just receive a byte and write it to flash untouched.  Checksum at the end, of course.  But that's SPI, with a CLK line you can control if necessary.  Does your system need to use UART?  If it does, then +1 for flow control, slower baud rate, two stop bits, etc.

By the way, the Z80 DART datasheet brags that the input is "quadruple buffered". What's that all about?

And on ASCII conversion, would a couple 16-byte lookup tables speed things up?

[paulca]

I have been collecting ways to program it since I started.  I expect I'll keep doing it LOL

I can pull the flash chip and program it with a custom STM32 programmer.
I can use an STM32 and level shifters to use traditional parallel handshaked IO
I can use UART.
I even want to stop having to remove the ROM and relocate one of the above to RAM and allow it to write the flash.

All are possible.  But all are in some varied state of working/complete/not yet.

The PIO loader is fast and accurate.  The worst bug it had was a startup deadlock that was fixed with a retry and just needs both ends to force the other to reset.

However it requires a breadboard with level shifters.

I think this is why I wrote the UART loader.  It is a direct connect to the USB UART at 5V.... but it's slow and the more reliable seems to make it slower.

Too many options, first world problems.  Nothing wrong.

[paulca]

By the way, the Z80 DART datasheet brags that the input is "quadruple buffered". What's that all about?

I'm not sure I remember that, but it is, by any modern standards a pile of shit.  I thought I had a bug or a broken chip the first time I powered it up.  It took a bit of research to find that ... no... thats a period NMOS DART at it's finest.

2mA drive current at 2.4V output.  And it considers that a "valiant high".   Even without any load on it's output, save a 1M scope input, I get 4.1V and I feel lucky.

And with such little output drive for something like a UART you would assume it's low power, right?

40-50mA IDLE.   Over 100mA (apparently up to 200mA) flat out.

It does get warm lets say.

I may however play this to my advantage.  A small series resistor to load the PIO (of similar output abilities) just enough and I will get 3.3V interface voltages without level shifters.

[tggzzz]

2mA drive current at 2.4V output.  And it considers that a "valiant high".   Even without any load on it's output, save a 1M scope input, I get 4.1V and I feel lucky.

That voltage is more than sufficient. Provided it is above 2V, a TTL input will consider it a high.

The principal value to a higher drive current is to get the output from 0.8V to 2.0V faster.