Converting struct in to array changing byte order

[sangar]

Hello all,

I am working on PIC18 (XC8 compiler)  based Project which acquires 7 sensors values and send them to PC. Each sensor value is  2 bytes long basically 16-bit integer.so,totally 14 bytes.To frame all the sensor values in one packet, I use structure and convert it to array while send to PC.

Problem:
In the packet received at PC side, I am getting LSByte followed by MSByte instead MSByte followed by LSByte.

EX:

Actual value of Sensor_1  =  0x0155 ,  but in the RX side I am getting  0x5501.
Here is the code:

Code: [Select]

typedef struct
{
 
    uint16_t CR_sensor;   //sensor 1
    uint16_t MR_sensor;   //sensor 2
    uint16_t BC_sensor;   //sensor 3
    uint16_t AR_sensor;   //sensor 4
    uint16_t BP_sensor;   //sensor 5
    uint16_t LVDT_1;      //sensor 6
    uint16_t LVDT_2;      //sensor 7
   
}Packet_t;

extern Packet_t packet;

Uart_transmission:

Code: [Select]

UART_Write_Packet((uint8_t *)&packet,sizeof(Packet_t));
Compiler: XC8 V1.36

I read that this is the problem due to big endian format used by Compiler.
Please suggest a solution.


Thanks,
Sangar





 

[Mechatrommer]

correct the endianess in PC.. problem solved. or val = ((val & 255) << 8 ) + ((val >> 8 ) & 255) in mcu... edit: damned smiley code..

[cv007]

due to big endian format used by Compiler.

The data is actually stored in little endian - the lower address has the lower byte. A uint16_t is stored LL HH (LL in the lower address, HH in the higher address), which means if sending out as bytes the lower address goes out first.

If you want to send out HH LL, you have to reverse it somewhere.

for( uint8_t i = 0; i < sizeof(Packet_t); i += 2){
    UART_Write_Packet((uint8_t *)&packet[i+1],1);//HH
    UART_Write_Packet((uint8_t *)&packet,1);//LL
}

or rewrite function

UART_Write_Packet(uint16_t* pkt, uint8_t sz){
    for( uint8_t i = 0; i < sz; i++){
        UART_write_byte(pkt>>8 );//HH
        UART_write_byte(pkt);//LL
    }
}

or many other options

[agehall]

Always use htons() and ntohs() for things like this!

[AG6QR]

correct the endianess in PC.. problem solved.

At first I misread this as, "correct the endianness of PC".  IOW, avoid using anything with an Intel architecture or one of the other little-endian machines.  I was going to shout out a big "Bravo!"

But then I realized you probably meant to write the PC software such that it would flip the byte order.  I suppose that's the most practical solution, and I will (somewhat reluctantly) endorse that.

But something about fixing the PC to be big-endian still appeals to me.

[Bruce Abbott]

I read that this is the problem due to big endian format used by Compiler.

XC8 user guide page 149:-

All integer values are represented in little endian format with the Least Significant Byte (LSB) at the lower address.

In the packet received at PC side, I am getting LSByte followed by MSByte

And that is a problem because?

[IanB]

I read that this is the problem due to big endian format used by Compiler.
Please suggest a solution.

Either correct the byte order at the sending end or the receiving end. This question is a little bit like asking how to tie a shoelace, is it not?

Another solution is to send the data in hex format, by literally sending "0155" over the link. A good reason for sending human readable data over the wire is that you can more readily sniff the messages and see what is being sent and received for diagnostic purposes.

[Nusa]

For portable code, the ideal solution is to avoid endianness issues entirely. I've written a lot of portable code in my day.

One solution is to transmit data in text format. As a bonus, it's a human-readable data stream. If you're transmitting floating point data between unknown systems, I'd recommend this for generic situations. Native formats for floating point can be VERY different.

Otherwise, my recommendation is properly define your transmission packets at the BYTE level. Then you know EXACTLY what to pack and unpack on each end, and the enddianness of the platform is irrelevant and doesn't byte you (pun intended).

So instead of what you have, something like:
typedef struct
{
    uint8_t CR_sensor_msb;   //sensor 1
    uint8_t CR_sensor_lsb;   //sensor 1
    uint8_t MR_sensor_msb;   //sensor 2
    uint8_t MR_sensor_lsb;   //sensor 2
    uint8_t BC_sensor_msb;   //sensor 3
    uint8_t BC_sensor_lsb;   //sensor 3
// etc...you get the idea   
}Packet_t;

[alm]

While I agree that ASCII can be a good solution, I do not see the point of the latter suggestion. If you want to send binary integers > 8 bits, just define a byte order for the protocol and use something like htons() to convert. To split up the bytes in a struct still requires pretty much the same code as converting, and results in code that is harder to read.

How is:

Code: [Select]

packet->CR_sensor_msb = cr_sensor >> 8;
packet->CR_sensor_lsb = (uint8_t)cr_sensor;
[...]
UART_Write_Packet(packet, sizeof(packet));
[...]
cr_sensor = packet->CR_sensor_msb << 8 & packet->CR_sensor_lsb;

More readable than:

Code: [Select]

packet->CR_sensor = htons(cr_sensor);
[...]
UART_Write_Packet(packet, sizeof(packet));
[...]
cr_sensor = ntohs(packet->CR_sensor);


[NorthGuy]

How do you acquire data from sensors? The easiest way is  to store the acquired bytes in the correct order (opposite to what you do now).

[NivagSwerdna]

correct the endianess in PC

+1

These endian-ness issue crops up a lot when converting from internal representations to on-the-wire forms.  In particular you find it when writing network protocols (if that's what you spent some of your life doing many decades ago)!  The Berkley Socket functions htons (host to network short) functions were created to help out with this respect.  On the plus side having a well defined network order at least allows the recipients to convert to their own host order which in a heterogeneous environment may differ (certainly did in the environment I was working.... but that's a longer story).

[ealex]

Treating structure as byte arrays and sending them over between system with different endianes is a bad idea.
It will work if you structure is homogeneous - same data type and it will break and make you waste hours when you change something in the structure - different data types or just structure packing.

Specific case : 3 mcu's , different endianes, very little available resources ( 1K of RAM on one of them )
complex structures had to go from 1'st to 3'rd where the 2'nd acted as a router + data modifier.
The chain was : MCU 1 gets CAN frame, populates specific frame structure and sends it to MCU 2
MCU 2 shifts endianess, changes something, shifts endianess again and sends it to MCU3

now imagine what will happens if a compiler option changed on one of the mcu's to pack structures or when CAN structures change ...
=> hours and hours and hours of comparisons, hunting bugs, etc


=> the solution was to have common macros to shift / pack / unpack each structure to the serial buffer
added bonus : we could make a PC decoder and have some real-time debug

[NorthGuy]

Treating structure as byte arrays and sending them over between system with different endianes is a bad idea.

FYI: That is how Internet works

[sokoloff]

Treating structure as byte arrays and sending them over between system with different endianes is a bad idea.

FYI: That is how Internet works

Not really. It treats structures as structures and uses hton and ntoh (host to network and network to host) convertors to put those structs into bytes on the network in network-byte-order.

[NorthGuy]

Not really. It treats structures as structures and uses hton and ntoh (host to network and network to host) convertors to put those structs into bytes on the network in network-byte-order.

PHY operates on the byte level. Drivers treat the sequences of bytes as packets (structures). It is up to the user program to apply hton/ntoh functions as necessary. In the end, "it" does all the job for you.

When you're considering communications from MCU to PC through a serial line, you need to it all by yourself.

[sokoloff]

PHY operates on the byte level. Drivers treat the sequences of bytes as packets (structures). It is up to the user program to apply hton/ntoh functions as necessary. In the end, "it" does all the job for you.

Agreed, but surely you don't consider PHY/layer 1 on a local LAN to be the same as "the Internet"

[NorthGuy]

PHY operates on the byte level. Drivers treat the sequences of bytes as packets (structures). It is up to the user program to apply hton/ntoh functions as necessary. In the end, "it" does all the job for you.

Agreed, but surely you don't consider PHY/layer 1 on a local LAN to be the same as "the Internet"

Of course not. Internet is all of the above and more ...

[Bruce Abbott]

Always use htons() and ntohs() for things like this!

But XC8 doesn't have those functions built in so you have to write them yourself, and you are back to square one because the implementation is platform-specific.

Luckily Microchip have a TCPIP library, which includes the following (in network.h):-

Code: [Select]

#define byteSwap16(a) ((((uint16_t)a & (uint16_t)0xFF00) >> 8) | (((uint16_t)a & (uint16_t)0x00FF) << 8))
#define byteReverse32(a) ((((uint32_t)a&(uint32_t)0xff000000) >> 24) | \
                          (((uint32_t)a&(uint32_t)0x00ff0000) >>  8) | \
                          (((uint32_t)a&(uint32_t)0x0000ff00) <<  8) | \
                          (((uint32_t)a&(uint32_t)0x000000ff) << 24) )

// host to network & network to host macros
#define htons(a) byteSwap16(a)
#define ntohs(a) byteSwap16(a)
#define htonl(a) byteReverse32(a)
#define ntohl(a) byteReverse32(a)


 

[Ian.M]

Ditch the struct and code it in pure 'vanilla' ANSI C with *NO* assumptions about the endianness of the underlying architecture or compiler implementation.

To do this store the sensor values in a uint16_t array so  the transmit routine can loop through them as uint16_t values without type punning.   Use an enum for the fixed index of each named sensor in the array - the sequence of the enumed names sets the sequence of the fields in the packet.  Make the transmit routine responsible for the TX byte order of each uint16_t value, extracting the bytes with shifts and masks.

[madires]

As we got a network byte order define a byte order for your protocol and convert to/from based on the endianness of the MCU or PC. For performance reasons I'd choose the MCU's endianness as byte order for my protocol.

[Nusa]

As we got a network byte order define a byte order for your protocol and convert to/from based on the endianness of the MCU or PC. For performance reasons I'd choose the MCU's endianness as byte order for my protocol.

Which is fine between KNOWN systems that are explicitly aware of underlying assumptions. Which are hopefully boldly documented in the code. There are so many programmers that aren't even AWARE of what endieness the current processor is using.  It varies among micro families as well; it's not just a PC and intel-based Mac thing.

My point is that the problem came up in the first place because the transmission packet was poorly defined, to the point of being different depending on the platform it was executing on. My example above was for illustration of a simple case. There are many variations on the theme. And yes, using the network byte order macros is one of those. Much of my career predates the existance of that standard, nor are they a total solution.

[Mechatrommer]

correct the endianess in PC.. problem solved.

But something about fixing the PC to be big-endian still appeals to me.

we cannot say either big or small endian is right or wrong, its inherent in the design of the hardware... btw, in avr chip there is register setting to store 10/16bits adc value as right or left aligned in adc register, setting this appropriately, and you dont have to change your code a thing. i'm not sure about pic18... otoh sending in ascii (of hex ascii) format have some good points as mentioned, but also bad points that needs to be aware of, the transmitted data will double the size (create some issue in large data transmission, slower etc) and have translation processing (binary to ascii) penalty, so not suited if higher speed acquisition application is required. ymmv...

[legacy]

OT:

Code: [Select]

       uint32_t data=0x01234567;

       Little endian             Big endian
       +----+----+----+----+    +----+----+----+----+
       |0x67|0x45|0x23|0x01|    |0x01|0x23|0x45|0x67|
       +----+----+----+----+    +----+----+----+----+
        byte:    0    1    2    3         0    1    2    3

Little Endian (e.g. how it appears on x86) vs Big Endian (e.g. how it appears on m68k and PowerPC)

Recently a friend wanted to realize a 68K clone in Little Endian

Endianess is a little endian machine. The least significant bytes are stored at the lowest address. This differs from the 68k which a big endian machine.This impacts the order of data stored in the system and specification of immediate constants.

And people claimed that

Oh, WOW, a little-endian 68K?!! I love it!! So much more elegant and sensible.

more elegant? more sensible? more-UAT-W-h-a-t?

My mind goes back when people programmed directly in machine codes and their program-lists where just a tabs of hex numbers. Oh, it's so clear to me WHY Motorola and IBM have always enjoyed the BigEndian format: because it makes it natural, since for human beings, numbers usually start with the higher part on the left.

FSCKOFF Intel!!!

And their silly decision to "make it different", reinventing the wheel in Little Endian.

btw, now why on the why ARM wants to be LE(1)?!?!?
Business decision? Marketing? Intel alliance?
(as initially intel wanted to produce arm chips)

Probably technical decisions are like blowing in the wind, but for sure we are still (and we forever will be) annoyed by their choice 

edit:
(1) yes, ARM-chips have a trick somewhere in the chip-configuration which allows them to switch from LE to BE. Some MIPS modern chips (e.g. MIPS32/R2) have a similar trick as well as the old MIPS-R3K used by SONY for their PlayStation1. The chip is configured as LE/BE (firmware choice) as the first instruction executed when the power on. But it's not the point.

[nfmax]

It seems people are missing the point here. The byte endian-ness is a property that must be specified in the communication protocol between the two systems, and the implementation of the protocol in each system must be responsible for converting to & from the native machine format. Just using 'text format' doesn't get round the problem - the native text encoding might be different in each system: 7 bit ASCII, UTF-8, UTF-16 or even EBCDIC! Even things like the bit order (for serial protocols) and polarity have to be specified.

No-one seems to have picked up that when sending structs, the member alignment also has to be specified by the protocol.

[helius]

The reason probably has to do with little-endian I/O devices. Your processor can have the capability to switch between different endian modes for each I/O device and memory (every PPC since the 601 has had it), but it needs to be supported by software with the appropriate instructions. If device DMA goes straight to memory, you have additional problems.

There are some algorithmic differences between big and little-endian. Certain algorithms (for searching or parsing) prefer a particular byte order.