/rant.

[AVI-crak]

Sorry to put it; but thats Bullshit.

The size of the data block in 99% of cases is not known to the receiving party. In the mode of full compatibility of different equipment - the block size is always unknown!!! You are passing a binary stream where the control characters are not unique!!! A control character in the wrong place can cause a data transfer error, and error correction will not help you. Just as waiting for the start control character after a crash doesn't help - because it might turn out to be false.
UARTs are designed to transmit textual information, which literally means 1 character for every 4 bits of arbitrary data. The screams and screams of horror about the difficulty of deciphering the data are just childish babble. If it doesn't work, use ready-made libraries, there are a lot of them!!!

If you want to send a binary stream through UARTs, then you need to use 9 bits of transmission. The ninth bit will act as a command/data switch, or an address/data switch.

[paulca]

I fear "we" (royal) are confusing two (or more) protocol layers.  I believe a lot of the issues described, IMHO reside not in the physical "link" layer UART provides, but in the upper layers surrounding what would be traditionally known as "Terminals", teletype or emulators.  The VAST majority of differences occur there.  I am thankfully below that layer.  So if I want to be a cheap skate and funnel memory over it with no protocol in a loop, it should work.

EDIT:  Nobody mentioned the FRAME_SYNC.  I'm both proud of that and ponderous.  I'm not 100% on that, but the theory was.. it's a fixed length transmission, spin it in a circular array and update it as and when you can.  Similarlly on the receiver.  Effectively combining two MCU memory copies of that struct in a uni-directional link.  (Oh... I want birectional later, but shit this uni-d should be easy?).

The FRAME_SYNC is an "unlikely byte" such that alignment can be verified by checking the struct has indeed got 0xF8 as it's FRAME_SYNC.  It also allows either synchronisation with another circular DMA Rx or single byte read until 0F8 and then read Frame length mechanic.

I have simplified that on the receiver side to a single shot.  Read 70 bytes.  Find the 0xF8 before proceeding.  It never finds a 0xF8.  It's all 0's.

I will pick the later up tomorrow and try and get it at least filling it's RDR register and DMA verified.

[AVI-crak]

I have seen many attempts by users to pass raw binary data over the UART, all of which failed.
UART is for text.

[langwadt]

I have seen many attempts by users to pass raw binary data over the UART, all of which failed.
UART is for text.

nonsense

[voltsandjolts]

Holy cow, so much rant about something so simple!
UART works just fine for binary.
My preferred framing method is COBS.

[SiliconWizard]

I have seen many attempts by users to pass raw binary data over the UART, all of which failed.
UART is for text.

 

[agehall]

Yes, everyone knows you must use pigeons for binary.

[SiliconWizard]

Well, I for one have tried learning the trumpet but I was never any good at it. Impossible to play any pleasant sequence of notes, so I decided that trumpet was an instrument just made to make noise and wake up soldiers.

[paulca]

Well, I for one have tried learning the trumpet but I was never any good at it. Impossible to play any pleasant sequence of notes, so I decided that trumpet was an instrument just made to make noise and wake up soldiers.

1*/s offset tangent...  Very much like Python in my opinion.   Where someone invents a technology to solve problem X for themselves (they felt, shit like this (post/thread), sucked doing it in C so they wrote a script language to call C for them).  Everyone jumps on it because it's f'ing excellent.. Brilliant for beginners as it's so simple.  For small scale RAD "rapid application development" - favourite of pre-sales and RandD teams - its perfect.   Fast forward 10 years and people are trying to use it to solve stupid problems via 15 deep in libraries) for which it wasn't only never intended for , but it a significant liability for and actually shit at.  And the language itself is so full of bugs and WTFs? that they simply cannot fix now because so many newbies learnt their entire career understanding how Python does things and that doesn't align with reality in so many cases....

Aka.  Over or extreme reuse/recycle.  Surprisingly the hardware folks get that far more often than in software.  We tend to get it the other way. 

C is just as bad, but at least C had the argument that it gives a pretty clear representation of what actually happens in the hardware.  Python?   It takes an RP2020 2 seconds to count to 10 million in python.  I have seen people make python code 10^3 times faster by simply aligning the function calls to their C counter parts underneath and avoiding python.  It still ran 10^16 times faster in C and resulted in only twice as much code.

I watched a guy explain a particular audio filter for an hour.  When it got to the interesting part I was hoping for a sample by sample or an 8-16 byte window, maybe a millisecond, but... he produced a python script with took 36 seconds to process a 3.2second mono guitar clip.  What the actual?  < Picard Face Palm Meme

Another /rant.

[langwadt]

I watched a guy explain a particular audio filter for an hour.  When it got to the interesting part I was hoping for a sample by sample or an 8-16 byte window, maybe a millisecond, but... he produced a python script with took 36 seconds to process a 3.2second mono guitar clip.  What the actual?  < Picard Face Palm Meme
Another /rant.

he forgot the part about, once you have verified and tuned your filter in python/matlab/etc. you rewrite it all in c,assembler, or similar so it will actually work in real time

[SiliconWizard]

I watched a guy explain a particular audio filter for an hour.  When it got to the interesting part I was hoping for a sample by sample or an 8-16 byte window, maybe a millisecond, but... he produced a python script with took 36 seconds to process a 3.2second mono guitar clip.  What the actual?  < Picard Face Palm Meme
Another /rant.

he forgot the part about, once you have verified and tuned your filter in python/matlab/etc. you rewrite it all in c,assembler, or similar so it will actually work in real time

Or, you don't, and it doesn't work, and then you start playing the trumpet instead.

[Siwastaja]

I think this highlights a point we agree on.  It's the bloody users.  If you dare to make an ASCII protocol, you absolutely must be prepared to handle ALL the human cases that the users will throw at it.

Of course, and you should.

Because literally the only sensible argument for the ASCII is the human interoperability. But in the RealWorld^tm, you mostly come across UARTs parsers that fail exactly at that. Some fail more (to the point of being colossal pain), some fail less, but almost all fail.

Typical experience with an ASCII-based embedded device (some examples that come into my mind: a super expensive IMU I interfaced with a year ago; or a hard disk debug interface; or a HP programmable power supply interface; CNC glue dispenser) is this:
* You enter commands and nothing happens
* Or maybe after you have entered two commands (wondering why the first did not work), then first of the two is processed
* You Google for hours what terminal emulator programs to download and how to configure them to use one of the at least 4^2=16 line feed combinations (CR, LF, CRLF, LFCR, and yes, those can be different depending on direction)
* When you finally get the thing to accept a simple command at all, you start using the thing
* Because it's a terminal, meant to accessed by user, you type, use backspace, arrow keys, DEL - nothing of that works either at all, or worse, characters at wrong places get deleted or even more usually, what you see on the screen does not match what is internally in the buffer
* Then you again Google for hours what terminal emulator programs to use and how to configure them to use the correct control codes or escape sequences. The number of combinations explodes to millions.
* With the IMU mentioned, I had to find a specific version of a specific terminal program and compile it from sources to get the desired conversions; just to configure that IMU to enable it in BINARY MODE, after which it is usable.
* You think you have finished a command and press enter and the thing does something completely different or crashes because the internal buffer did not match with your screen, because you pressing backspace was processed differently at the two ends.

You can of course make a decision that the UART interface is not to be used by humans. But there goes the #1 argument for the thing. If you have to generate and parse responses using a Python script for example (which is a great idea to be honest), then you could have used a binary protocol just as well, and have the Python script make it human readable!

The ASCII-controlled RS232 devices of the 1980's have all but disappeared. What remains are simple application-specific UART links, often in binary because it's more efficient use of tight resources, and easier to deal with.

[westfw]

I have seen many attempts by users to pass raw binary data over the UART, all of which failed.

Kids today don't remember dial-up Internet, or the other protocols that did binary just fine.
AX.25.  SLIP.  PPP.  XRemote.  ARAP.  XMODEM, KERMIT, UUCP.  even smart and graphics terminals transmitted stuff in binary.

[AVI-crak]

Physical layer protocols have a precise standard, violating publicly (to the outside world) is like screaming about your homosexuality. Data transfer protocols also have their own standard and precise description. You have the right to write your own new data transfer protocol, but you won’t be able to use it everywhere - they simply won’t understand you.
Within one circuit board (at a short distance) - you can have any kind of sex. It does not matter how and where, the main thing is that there is mutual agreement.
At the expense of UART, the transfer of 0x00 - for the terminal literally means the end of the communication session. Everything, now you can not send data to an external device, it no longer wants to communicate. Sending raw data will inevitably send 0x00. A normal terminal will still receive data, but it can no longer guarantee correctness. Because it expects 0xAA or 0x55.

[paulca]

So debugging first the scope and then the receiver MCU and I believe the UART does not like the DMA loop.  Neither scope nor reciever can stay locked onto the bit sync for some reason.

Rx throws constant Overrun errors and occasionally Frame errors and start/stop errors, but doesn't receive a single byte.

I think I've figure it out.  The circular DMA is spamming the UART and giving no time to be idle.  That makes the "START" conditions ... well... there aren't any.

Doing a single capture on the scope "from boot" and the struct is correctly decoded.  All 0s and a single 0xF8.  Hitting "Single" again though while it's running and I get the weirdly rotated stuff again.

I'll briefly look to see if there is a solution to that, or I'll just use a single shot DMA transfer instead of a circular array.

I think I need idle line detection as well as inserting idle time into the TX stream.

[newbrain]

Doing a single capture on the scope "from boot" and the struct is correctly decoded.  All 0s and a single 0xF8.  Hitting "Single" again though while it's running and I get the weirdly rotated stuff again.

With one stop bit there is no way for the decoder to tell apart a stop+start situation from any set bit followed by a reset bit in the actual data..
With more stop bits, the probability of getting it wrong goes slightly down, but it is still quite possible to have false matches.

This is probably what the Rx part also experiences, leading to the errors you see, and it's (almost) completely independent from the choice of sending binary or text.

It the receiver is enabled when the stream has already started, how is it supposed to synchronize?
You need to insert an idle state for at least one character (10 bits for 8N1) duration every now and then to give it a chance to clear the previous errors and re-synchronize.

Sending raw data will inevitably send 0x00. A normal terminal will still receive data, but it can no longer guarantee correctness. Because it expects 0xAA or 0x55.

What? Where does this come from? Certainly not from the ANSI (~DEC VT) terminal standards?

[Siwastaja]

With one stop bit there is no way for the decoder to tell apart a stop+start situation from any set bit followed by a reset bit in the actual data..
With more stop bits, the probability of getting it wrong goes slightly down, but it is still quite possible to have false matches.

A very good point; UART is not really designed for non-stop datastream (text or binary, doesn't matter). You need to add idle times that act as synchronization points. For a perfect case with no errors, one synchronization point (e.g., at the beginning) would be enough, but in practice, you will want them "every now and then".

Using two stop bits instead of one is equal to adding one byte-time of idle time after every transmitted byte, and significantly improves the probability of staying in sync or getting back in sync even if you do not add proper idle periods. What is proper? I don't know exactly, never thought about it deeply, but I'd think ten bit times should do it.

[paulca]

Pfff...

I made the transmission a one shot.  It still writes 2 structs.

When that completes it waits for 10ms before firing it again.

The scope still decodes wrong.  Although I can see my idle time. 

For the Rx code I tried using HAL_UARTEx_ReceiveToIdle_DMA(), expecting to get a partial or an error the first time, then have it restarted during the idle time and it should sync.... right?

Well, The UART throws a set of random errors back so the ReceiveToIdle won't start.  Grrr..  I got angry and put it in a while( broke ) loop and... it would not lock onto the stream.

Stepping into the HAL code.  The DMA aborts because having set the UART to Receive to idle mode and starting it, it throughs back and OverRun error or START/STOP error etc.  Which immediately causes the DMA to abort.

However, it doesn't clear the UART status.  I have found this twice now.  A failed UART does not stop the DMA and a failed DMA does not stop the UART, it just leave the other part hanging in LOCKED and BUSY state.

So... I put HAL_UART_Abort()  HAL_DMA_Abort() all over the place (angry code) and FINALLY it will lock onto the stream once in a while.

I'm not even going further to check what it receives it's disgusted me now.

I need to redo the angry code with something less like a hammer over the head.

For example, surely there is a way to start the UART with HAL in "Wiat for idle", then in that interrupt reset the UART and DMA state and start the stream whlie the line is idle.

[Siwastaja]

STM32 peripherals and their interaction with DMA is poorly designed and difficult to use. Poor, unhelpful libraries do not make that any better, as now you have to understand one more layer of crap.

Compare this to nRF52 where you only have the DMA mode, with just pointer register and byte count register in the peripheral itself (no mapping with a separate DMA), and a START triggering register where you write '1' to start. Could not get any simpler than that.

[tooki]

UARTs are designed to transmit textual information, which literally means 1 character for every 4 bits of arbitrary data.

I’m sorry… what?!?

As far as I know, the text encoding with the smallest word is 5-bit Baudot, but we don’t use that any more. Everything that’s not Unicode (which is 8, 16, or 32 bits) is some variant of 8-bit ASCII, with some EBCDIC and 7-bit ASCII lingering around.

If you reeeaaaalllly stretch the definition, one could call Morse code a 4-bit encoding, since the longest codes use 4 dots and dashes. But that doesn’t translate cleanly to bits.

[coppice]

If you want to send a binary stream through UARTs, then you need to use 9 bits of transmission. The ninth bit will act as a command/data switch, or an address/data switch.

Very few people do that. Most people send 8 bit data with stuffing. Bit stuffing or byte stuffing is how the vast majority of digital communications, async or sync, deals with the need to distinguish data from control bits.

[paulca]

Meh.  HAL is a ballix.

I started disabling the error interrupts I didn't want it to abort on UNTIL IDLE!  and it sent HAL into an infinite interrupt loop.  0 main execution.

Looks like I'll have to do the "Wait for idle" myself with the registers.  Surely I can just blank the ISR, enable IDLE start it and wait for the interrupt.  It's not going to be that simple, I know it.

[rhodges]

Sorry if you wrote which processor you are using, but I seemed to have missed it.

And just to get past two silly questions...

Does your CPU have CCM, and can you verify you aren't trying to use it for DMA?
Does your CPU have data cache that needs to be managed in software (like some MIPS)?

[NorthGuy]

@paulca You're approaching this very inefficiently.

Imagine you would want to pick a digital lock - 4 digits - 10000 combinations - very time cosuming to try them all. That's, sort of, what you're doing - you wrote a program which uses many things at one - use floating numbers (with a little bit cryptic IEEE-754 format), pack them into the structure, use DMA to transmit, use your own receiver. Some of these things fail, but you don't know what it is, so you need to guess which part went wrong. If many different parts are failing, fixing one part may not solve the problem. This keeps you guessing and consequently you need lots of time (or lots of experience) to guess what is the exact problem.

Now imagine you can pick the lock digit-by-digit - pick one digit, then move to the next. This would me much easier - only 40 possibilities instead of 10000. Similarly, you can approach programming software gradually, step-by-step. This is true for any software, but especially beneficial in embedded programming.

So, if you would move step-by-step:

- You first send a byte, watch it on the scope. Make sure the baud rate is ok, send different values, make sure they come out right.

- Switch to the receiver. Write a simple program which receives characters and dump them on the screen. Receive different characters, make sure you receive what you send.

- Go to the sender, build your structure, dump the binary content on the screen, make sure you understand it.

- Send the structure to the receiver, make sure the receiver gets the binary content ok.

- Fill in the structure on the receiver with binary content received from the sender. printf the structure and make sure it's identical to what you have sent.

- Go to the sender, change to DMA. Make sure the receiver still receives everything fine

- Change the receiver to DMA. Make sure everything is working.

- Produce a stream of structures with pre-defined values. Add a checker to the receiver which can verify the stream of structures. Run it for an hour and make sure there's no errors.

You can skip over steps if you feel confident, that is make steps bigger, but not bigger than you can handle. When you do a thing which you have done many times already, making bigger steps saves time. But if you're just starting with something new, try to take very very small baby steps ...

Also, you probably need to change the design a bit - use a preamble (or a pause) to make sure that the receiver can find the beginning of the structure, add CRC if needed, add error handling (what to do with lost packets etc). This will require extra steps to the implementation.

[paulca]

I tend to take a different approach.  Especially as I started with the auto-generator and HAL.  When one calls a function to send data over a UART, one expects data sent over UART, not a tour of the camp and playing whack-a-mole with undocumented features.

My general tact is to rapidly iterate.  Start with the meat of the problem, the hardest bit.  Assume it will work first time.  Test that theory with a rapid disposable prototype.

If it passes sanity tests, move on.  If it fails, iterate in on it, peel back a layer or two and dig more deeply.  If it still fails, try something else.

When you have a basic straw man working, even if it's in parts, sometimes better if it is.  You can loop backaround and refine.

It's sometimes referred to as "fail fast", "learn fast" approach and it also minimises un-necessary effort by identifying possibly un-necessary rigor and skip it.

I personally find it annoying, say in a screen share zoom meeting, when the person in control will diligently go through the process step by step each time.  Did the request file get written?  Yes.  Open it and check.  It's fine.  Run the batch.  Did it run.  Did ... and so on and so on, all the way until 5 minutes later they fail due to a typo in the config of the last step.  Do they go back and start again. Yep.  Do they just run everything up to that point and start to debug there?  Nope.  They go back through the rigor of testing each individual step that led up to it.  It's just wasteful.  You have to work smarter than that.  Be able to jump around, only dig holes you need to go down.

As things firm up, you reiterate into the finer refinements and testing.  There is no point refining, optimising or full testing something if it's not finished and you're not 100% sure what "finished" means anymore.