memcpy() and volatile

[brucehoult]

It's the principle! remember, you do it today because although not a scalable solution it's a small problem. Then in the future the problem scales, the solution cannot and you have to reinvent the wheel. Tell me, how do J1939 auto ECU's manage this problem? they have several hundred messages of no same format, you are telling me that someone wrote a message handler for each one?

Only hundreds? They may well have written them by hand.

Or, they might have written a small program to parse some description of the messages and generate the code that would otherwise have been written by hand.

Premature generalisation is a sin. "YAGNI" as the eXtreme Programming people say. People such as John Carmack agree: "It is hard for less experienced developers to appreciate how rarely architecting for future requirements / applications turns out net-positive."

[Simon]

Premature generalisation is a sin. "YAGNI" as the eXtreme Programming people say. People such as John Carmack agree: "It is hard for less experienced developers to appreciate how rarely architecting for future requirements / applications turns out net-positive."

In my experience of both hardware and software, as soon as you have a working solution it becomes the way it is done, the first time you get it to work is the way you will go on to make it work. You cannot turn around to someone in management or sales and say that this thing that works right now, we have to scrap it for the next project and start again. They don't understand, they have no idea of the many ways of doing things and how just a small change in requirement in their opinion is a massive shift in what they are now asking for.

At my last job I worked with a very good contracting engineer, we had 5 hour straight meetings and thoroughly hammered out everything, we were always grateful we did as come the next project we already had solutions that had already been used and simply needed to add to our arsenal, again, with consideration for how to generalize the solution slightly.

[Simon]

It's the principle! remember, you do it today because although not a scalable solution it's a small problem. Then in the future the problem scales, the solution cannot and you have to reinvent the wheel. Tell me, how do J1939 auto ECU's manage this problem? they have several hundred messages of no same format, you are telling me that someone wrote a message handler for each one?

In a sense, maybe yes.

In at the hardware level, on a message bus, each device will only pick out and process messages it is interested in. It will look at the PGN and SPN to decide this, and if it doesn't care it will ignore the message. So it only locally has to handle a few kinds of message.

On the other hand, if someone is writing a debugger or bus sniffer that needs to decode and display all kinds of messages, then it will indeed have to handle dozens or hundreds of message types. But it won't do this with huge amounts of code, it will use a database where it looks up the message type and gets the decode information from the database, so it can automate the process. It will use this database to figure out how to unpack and decode the contents of each message.

Yea sure, a temperature sensor can do what it damn well likes, the software in it is definitely limited in scope. I am talking about an ECU on which I already run CAN open. What you are saying is that a J1939 ECU is basically running each message like a CAN open PDO message, just with different message ID's. I already handle PDO messages as you describe so yes I can effectively convert the 3 messages in question to run on the PDO handling code.

But the fact still remains that hardware registers in micro controllers are mapped with structures. As they are hardware registers it is not a question of arguing about how the struct memory is allocated. It is already allocated by hardware and these systems work. I understand that maybe RAM allocated structures will be created differently but then I want to know why and how. I can't just keep followring recipes.

If you want an example of my frustration, here I have been told that I should not define variables as static unless inside a function. In another thread months ago I was told I absolutely should declare anything static that I did not want to be global, you see my problem? The language was created in the 70's and not formally standardized until 1989, and then standardized about ever 5 years. People tend to stick with what they learnt when they were young and then don't change, so opinions are always mixed.

[brucehoult]

If you want an example of my frustration, here I have been told that I should not define variables as static unless inside a function.

I can't find any such thing in this thread.

In another thread months ago I was told I absolutely should declare anything static that I did not want to be global

If you have code that uses global variables with the same name in different files and you don't want them to actually be the same bytes in memory then, yes, all of them (or all but one) need to be static. Exporting the name, and the linker merging all global variables with the same name, is the default.

[IanB]

But the fact still remains that hardware registers in micro controllers are mapped with structures. As they are hardware registers it is not a question of arguing about how the struct memory is allocated. It is already allocated by hardware and these systems work. I understand that maybe RAM allocated structures will be created differently but then I want to know why and how. I can't just keep followring recipes.

Well, yes, in many systems, hardware I/O registers are mapped to particular memory addresses, and then it is possible to overlay a structure on that section of memory to make it easy to access the registers by name. However, it is still essential to define the struct carefully to ensure the structure elements align properly. When these structure definitions are provided in a header file, someone has gone to great care to make sure this works.

If you want an example of my frustration, here I have been told that I should not define variables as static unless inside a function. In another thread months ago I was told I absolutely should declare anything static that I did not want to be global, you see my problem? The language was created in the 70's and not formally standardized until 1989, and then standardized about ever 5 years. People tend to stick with what they learnt when they were young and then don't change, so opinions are always mixed.

Well "should" or "should not" are much too broad and general to be treated as good advice. If someone tells me I should not eat red meat, the first question I am going to ask is "why not?"

It is an unfortunate cock-up in C that "static" at file scope has a different meaning from "static" at function scope.

Declaring a variable static at file scope causes it to be local to the file and invisible outside of it. Whereas a variable not declared static at file scope is still static in the sense of maintaining its value, but it is also visible to other files in the linkage unit (it can be referenced with the extern keyword). This means all variables at file scope should be declared static unless you want them to be visible externally.

On the other hand, static inside a function has nothing to do with visibility, and just causes the value to be persistent between function calls. There are a few cases where this is needed, but not many. More often than not it is a really bad idea and should be avoided as it can prevent recursion and re-entrant behaviour.

[Simon]

If you have code that uses global variables with the same name in different files and you don't want them to actually be the same bytes in memory then, yes, all of them (or all but one) need to be static.

So I won't call that a contradiction but from what you say you assume that all variables not defined in a function are normally accepted to be global at inception unless qualified with static, when I do use global variables unless I declare them as extern in a header file that is included where ever they are used or I end up with issues with the link you describe not necessarily happening and strange stuff or the compiler outright complaining (hopefully). so the compiler has to be explicitly told either way or you get undefined behavior, unless of course you are setting up a variable that will be interpreted as global but that is used in that file alone and there is no other named variable.

[Simon]

On the other hand, static inside a function has nothing to do with visibility, and just causes the value to be persistent between function calls. There are a few cases where this is needed, but not many. More often than not it is a really bad idea and should be avoided as it can prevent recursion and re-entrant behaviour.

On the other hand I find static variables in functions incredibly useful being able to put variables that need to be remembered between function calls but that are used only by that function in that function so that I have a single unit of code with everything it needs. Classic example in my project is the last_run variable that any function called in main has. It would be a nightmare to have many of these defined outside of the function. I believe that this is a good use of a static variable in a function and it was an eye opener when I found out about it, as you say as the word has two meanings I was initially only aware of the file level one.

[Simon]

But the fact still remains that hardware registers in micro controllers are mapped with structures. As they are hardware registers it is not a question of arguing about how the struct memory is allocated. It is already allocated by hardware and these systems work. I understand that maybe RAM allocated structures will be created differently but then I want to know why and how. I can't just keep followring recipes.

Well, yes, in many systems, hardware I/O registers are mapped to particular memory addresses, and then it is possible to overlay a structure on that section of memory to make it easy to access the registers by name. However, it is still essential to define the struct carefully to ensure the structure elements align properly. When these structure definitions are provided in a header file, someone has gone to great care to make sure this works.

And I was equally careful in creating my structures to send CAN data but as things will scale and this is an error prone way to do it at scale I will convert to PDO handling as that system is already written.

[Mechatrommer]

Premature generalisation is a sin. "YAGNI" as the eXtreme Programming people say. People such as John Carmack agree: "It is hard for less experienced developers to appreciate how rarely architecting for future requirements / applications turns out net-positive."

things like "refactoring" is imho one of "you are going to need it in the future" in mind. following YAGNI, whats the point of refactoring? as long as it work? but its very well accepted concept. experienced programmers know how to refactor well since they can anticipate which one is more reusable than the others, there is no way i know of describing this in words (experience) you need to look at the code wholly and you know what to do. this is important when laying out a programming from scratch, usually on first stage version 1.0.0, very well generalized / refactored code to start with can ease scalability (upgrade, expansion or bugs fix) process later and sometime done by another new programmer, and can enforce or as a guideline example on how good coding practice is/should be done in that particular project, if uniformity of style can be maintained throughout the process it can help new maintainer to understand the code and paradigm quickly. let alone basic bugs like incorrect casting or memory overrun/leak issue, inconsistent declaration etc, coupled with unorganized spaghetti code will be a nightmare to fix. ymmv.
https://www.freecodecamp.org/news/what-exactly-is-a-programming-paradigm/

[Simon]

This program as it is has been through much rewriting, simply because I have had to learn about CAN Open the hard way. Every time I learnt something else that was needed to make the system work I had to go back and rewrite functionality to support the improvement. There was always the temptation to just get it to work but I always saw that this would just create even more convoluted code that I would be stuck with and it would actually take as long as the rewrite, but some one could point at what I had and say why change?

We had this argument about the whole machine. it has 2 analogue inputs that can be used to run the motors, but I did not like this, but I was told, yet but it works.... I said no. This is a very bad way of doing it in ways that you don't understand because you are not an electronics engineer. We must take on this CAN open thing because sending a voltage that has to survive interference and be interpreted the same by multiple motors is a fools way when I can send a number, and that number will always arrive as sent.

The previous iteration of the machine used PWM outputs, put through digital isolators, low pass filtered and then sent to the motor driver analogue inputs. A very sorry state of affairs. I put my foot down and here I am 2 10 weeks later with a barely functioning CAN Open implementation that I am still ironing the bugs out of but not for a moment would I give it up and go back to what we had.

[Psi]

Why CAN and not something much simpler like RS485 where you roll your own small frame-format/protocol to only do what you need for your application. Instead of learning tons of CAN stuff

I'm not saying you shouldn't use CAN or you should use RS485. I don't know enough about your application. Just curious why CAN.

[DiTBho]

Why CAN and not something much simpler like RS485 where you roll your own small frame-format/protocol to only do what you need for your application. Instead of learning tons of CAN stuff

Think, I worked on a quadcopter project for semi-professional aerial 4K shots + FPV, and we used canBUS for the ESCs that drive the motors as well as for the gimbal that positions the camera

Same question - Why?  - I asked the project manager - because I tell you, and because the customer wants it that way - he replied

So 
 
RS485 seemed much, much simpler to me, both on the hw, fw and sw side. But nothing to do. No dice.
And years later... I found myself in exactly the same situation with electric actuators on bicycle derailleurs.

but in this case, for my project, I will not use canBUS, even because I still have the PCI32-CAN card HBA I used for the quadroter and its kernel drivers stuck in kernel v3, which no one has ever wanted to update and... they are so complicated that I don't even want to.

[DiTBho]

Premature generalisation is a sin. "YAGNI" as the eXtreme Programming people say. People such as John Carmack agree: "It is hard for less experienced developers to appreciate how rarely architecting for future requirements / applications turns out net-positive."

I do agree, too 

[Simon]

Why CAN and not something much simpler like RS485 where you roll your own small frame-format/protocol to only do what you need for your application. Instead of learning tons of CAN stuff

I'm not saying you shouldn't use CAN or you should use RS485. I don't know enough about your application. Just curious why CAN.

The motor drivers use CAN Open and that is the end of that. I can't change that and anything motor wise off the shelf is much more likely to be CAN Open than anything else.

As for the implementation of CAN bus itself at a low level you have to have a good long think (hours not minutes) at how you solve all the problems the CAN bus controller inherently solves in hardware that you have to work out for yourself and code in software with RS485 or any other USART type port bus.

Yes the CAN bus controller peripheral on the SAMC is complex, but good news, it was designed by BOSCH and is licensed into many micro controllers and PC hardware. So having a working knowledge of that particular controller is very portable.

You are of course confusing CAN bus with CAN Open in your comparison to RS485. RS485 is simple enough in principle but the whole reason you use serial communication is so that you have an easily scalable system to create a network that can increase in size by a sensible amount. Even with just two devices you will have lots of fun with RS485. Unlike CAN bus RS485 is not collision tolerant, there is no bus arbitration in hardware that resolves conflicts automatically so that the winner transmits successfully and the looser tries again automatically. All that CAN bus handles automatically in hardware with no CPU cycles used is absent in RS485.

Once you try to solve the problems of bus contention you soon start scheming up quite a complex system of master slave messaging, one that may not rival CAN open in size but certainly means you are not just writing code to follow some rules, you have to make the rules up yourself, make sure they are fool proof and then implement them whilst wishing you were not spending so much of you CPU time on communication rather than your application. You soon start to miss that co processor that is the CAN bus controller.

If I used RS485 then rather than CAN Open I would be implementing something like MODBUS on RS485 as that is what any motor driver would have. I don't know how complicated it is but once again I'd be writing as much code nearly as I have on CAN Open. I also believe that CAN Open is also implemented on RS485 too....

No matter how you try to solve these systems the problems are universal and the total solution will always amount to about the same, but CAN bus does more in hardware for you alleviating the load.

Lin bus was invented as a companion to CAN bus, the reason? to save money in cars, first cars saved money in wiring looms and connectors by implementing CAN bus, then they realized that this was a lot of hardware to put into a window motor and every single micro controller has a UART but not a CAN controller as a UART is really simple, and if it is just a window motor or a sensor then it's OK to spend most of your CPU time on processing. But Lin bus networks are just small networks that connect to a CAN bus enabled Lin master that uses CAN bus to talk to the rest of the car but now only one sub system has to have the "expensive" CAN controller. it was worth the mammoth software task of writing the Lin stack because if you add up all the little 8 bit micros in a car and multiply them by the amount of cars made each year it is a lot of money and a totally different effort/reward analysis.

[Simon]

And just to say, I am updating my motor drivers every 7ms at the moment, try that on RS485.....

[langwadt]

And just to say, I am updating my motor drivers every 7ms at the moment, try that on RS485.....

what do you imagine would be the issue?

[DiTBho]

@Simon
what do you use on your GNU/Linux computer as a canBUS HBA?
USB-to-CAN? 

[Simon]

And just to say, I am updating my motor drivers every 7ms at the moment, try that on RS485.....

what do you imagine would be the issue?

Well I don't know that there would be any for sure but I do know that RS485 would come with extra overheads for the CPU compared to CAN bus, so I would be careful about going mad like that.

So there are 2 messages being sent to each driver and the driver responds with 3 messages, so every time I send a sync message onto the bus that is a total of 11 messages, there is also a heartbeat message x3 being exchanged every 500ms and some background SDO communication once per second. Decoding an SDO message is a lot of work, it takes longer than a message to be sent.

On top of that the micro controller CPU has to do all of the running of the system, working out what to send to the motor drivers next and keep the HMI on it's own CAN bus updated.

If I were doing this on RS485 I would have to do the error checking calculations on both sending and receiving. This would be pretty simple algorithms simply because I know no better. CRC is not something I have done. The CAN controllers do CRC for themselves. I would also have to totally manage the bus access of every message from the ECU program. So with CAN open I just chuck 4 messages at the CAN controller and leave it to transmit each one and then send a single sync message which again I just send without having to care about bus contention.

Once the sync message is received 4 responses come back, with an RS485 system these could not just respond like that, each one would have to be triggered, so 4 request messages instead of one sync message.

Remember, we are only talking about 2 drives here, CAN open supports up to 127 slaves, and the PDO messages do not have to be sent every sync cycle. Now could an RS485 based system where the application code has to include all that the CAN controller does, manage so many slaves at speed? I would be looking carefully at an RS485 system that had to do this as I would have one CPU, not the equivalent of 2

Of course, you forget one important aspect, I can't make this decision. Like all drive manufacturers, these are designed to work on CAN Open that sits on a CAN bus layer, I could not use RS485 if I wanted to, and I can see why by default RS485 is not used.

The other advantage of CAN bus is that it is easier to add more nodes. Because on a UART based system you MUST have a strict master slave scheme or there will be bus clashes. Many drivers these days will cope with this electrically, but if it is a normal condition then that is more code writing, more overhead, more development time to replace hardware with software. So if you use RS485 either you need to update the master code to include any new device or you need a system of scanning the bus at start up to see who responds, another mountain of code. If you were to use predefined ID's to represent devices you soon end up with more ID's than the system will ever have as you must include scope to scale the system or the whole point of the system is defeated by constant rewrites.

You see I am not joking when I say that I have given all of this many hours of thought, over the last few years days of thought in total. But when the day came that I had to argue about using CAN bus / CAN Open and upset all of my management right up to the boss of our billion dollar group as I committed 2 months of full time work to what was thought to be a folly, I knew that I was right, even though, I still had no choice as there was no RS485 option.

Well I think the drive manufacturer has an RS485 addon but looking around it is clear to me that CAN bus is at least as prevalent if not more preferred than RS485 by all motor driver manufacturers and in my line of work it is motors all the way. At the end of the day it is not my choice, but I am happy with the choice.

The CAN bus layer took 2 weeks to develop. If this was an RS485 system there would have been more development time to deal with the stuff that a CAN bus controller does in hardware. Then and only then I would have started to tackle any of the actual higher level protocol stuff for I assume MOD bus.

Oh, another CAN bus advantage, it has it's own dedicated DMA with the system RAM, pretty neat, so not only a coprocessor but dual channel RAM access....

[Simon]

@Simon
what do you use on your GNU/Linux computer as a canBUS HBA?
USB-to-CAN? 

I have not got to the Linux stuff yet.

Before you go on to make a clever point, remember, it is the motor driver manufacturer that chose the CAN bus option for me.

On a linux machine i would have a who SBC of resources at my disposal, I would also not be writing low level code, so the choices are about entirely different issues. it's about what is natively supported.... ooh, I hear there is this thing where CAN bus controller IC's on SPI have a driver baked right into the linux kernel and there is something about socket CAN. But this is all for another day so don't worry, tons of threads on that when I actually gfet to it right after this mess

[ejeffrey]

You can either offer a proper answer and stop telling me I am ranting for weeks or continue to belittle me.

I don't know what your question is and you haven't provided a clear *example* (that means code) of what you are doing.  You verbally describe 1-2 lines of code at a time without context and I for one am completely lost at what you are actually doing.

[Simon]

You can either offer a proper answer and stop telling me I am ranting for weeks or continue to belittle me.

I don't know what your question is and you haven't provided a clear *example* (that means code) of what you are doing.  You verbally describe 1-2 lines of code at a time without context and I for one am completely lost at what you are actually doing.

I attached the entire project. Then I was told that this was also not good as it is too much to go through. Funnily enough that is what I tried to explain. I explained the bare bones of what was involved with the mysterious issue but apparently that is too complicated to understand. Sure if I spend time recreating a project with just that small piece the issue will likely go away because as we have established there is a wider problem somewhere.

[Simon]

I have found another place where too much data is being copied by memcpy, that is likely the culprit.

[brucehoult]

Premature generalisation is a sin. "YAGNI" as the eXtreme Programming people say. People such as John Carmack agree: "It is hard for less experienced developers to appreciate how rarely architecting for future requirements / applications turns out net-positive."

things like "refactoring" is imho one of "you are going to need it in the future" in mind. following YAGNI, whats the point of refactoring? as long as it work?

Refactoring is what you do later on when you discover an ACTUAL requirement that can not be easily implemented in the current design.

You refactor the code -- that is, change its structure without changing the functionality AT ALL -- until it is in a form where the now known to be needed functionality is easy to implement.

And you do and commit that in TWO steps, not one. In fact preferably three:

- refactor, verify that all tests still pass

- add tests for new desired functionality, verify that they fail

- add new functionality, verify that the new tests now pass, and that old ones still do.

[DiTBho]

And just to say, I am updating my motor drivers every 7ms at the moment, try that on RS485.....

what do you imagine would be the issue?

ehehe, I worked on some industrial cutters connected to reels-winding, machines of several tons, as big as a room that make shopping bags for supermarkets, unroll a reel of thin plastic film, cut it, and seal it hot, also making 16 bags in parallel.

I happened to have to design a perimeter Lidar 2D, and a part of the unit that manages both the pneumatic part and the blades array, and then have to interface them with the "computer" that manages the machine, which I have no idea what it is (I couldn't take a peek (1)), but I found EtherCAT, which I also found in avionics, as well as AFDX, in UDP/IP based applications with custom go-back-n algorithms.

They { EtherCAT, AFDX } are both ethernet based, but have been modified to be not only deterministic, but also lightweight, and in the work I have done, the response time was within 5 msec.

I found Tulip chips (ex Digital tecnology), which, in Linux and VxWorks terms, they "offload the CPU" - written like this in the manuals -  by performing lowlevel (LLC) checksums and performing networking operations autonomously, so the CPU, doesn't matter if it's a superscalar PPC460 with 4 cores and 32MByte of L2 cache!!!, rather than a non-superscalar MIPS-I wheelbarrow, with neither cache nor pipeline, the CPU always sees "the network coprocessor" as interrupt-driven and DMA-driven partner.

Similarly, in the world of perimeter security (both civil and military, from avionics to naval), I have seen engineers with 30 years of experience, i.e. people who started working in 1990, developing solutions based on RS485.

They know how to do it because they have been doing it for a long time, and they are solutions that couple the RS485 with ad-hoc intelligent cards that practically do "offload" networking functionalities.

Pretty like Tulips (used in { EtherCAT, AFDX, ... }) and CanBUS.

- - -

So, in my opinion today there is a lot of emphasis on canBUS because it presents itself as a ready-made package, as well as Bosch is a very solid and serious company that offers solid guarantees, while these 90s engineers are close to retirement age, so it would cost more to train new engineers, also considering that it is a rather old technology.

It costs much less, and that's not something to underestimate 


(1) the security guy was watching me closely, and he was much bigger than me - "don't touch that panel!!!" - I risked getting slapped several times!
I was abroad on a business, so … somehow I managed to keep my curiosity in my pockets.

[Simon]

The simple fact is that if you have onboard CAN bus on your micro controller you effectively have a comms co-processor. One thing I forgot to mention earlier about Lin was that it runs at a mere 20kbps with a fast option at 80kbps. Lin also does not need a crystal for the clock as it can self synchronize at the start of every message. So another huge cost taken out. at such a low speed getting the system RC oscillator to resync is probably easy as there is lots of resolution between a few MHz and the 20-80kHz. It was designed for a specific task and has the trade off of those tasks.

As I said I have thought long and hard about how to implement an entire network off a UART and it is not easy. At my last job we did it or rather the subcontractor did it for us after using CAN bus on previous projects. But those were customer specific systems that would not change and we knew what the limit devices were on the bus so the ID space was simple enough.

In this job I work on any motor driver that I can lay my hands on and CAN Open is very much the industrial standard to the point that it works on Ethernet and RS485. I don't know how they do this but the manual for the Drive covers both ethercat and CAN Open at the same time.

I am guessing that if you want even more speed and more devices ethernet is the physical layer designed for that, CAN FD is still in it's infancy with CAN Open but that does increase the speed 10 times and I think you can implement the extended ID system if you really want to. If anything CAN bus as a physical layer is designed to run both standard and extended ID's and I beleive CAN and CAN FD can work together. I base this on the Bosch CAN controller have different CANS FD modes, one where is sends the header at normal speed but the data at the faster rate and one where it sends the lot at the higher speed.

I think these are all the things that keep CAN in the industry, it is so robust but also able to coexist whilst reducing development time. In my line of work if the micro controller with CAN costs twice the price it is still a no brainer, our volumes are so low that development time is more expensive. So I have spent 2 months on CAN Open simply because after two and a half years at this company it has become very clear that CAN Open dominates in the supply chain we use so I either get with the game or get left behind. There are so many projects I have to work on in the future that involve motors that having to design bespoke circuitry for each motor will just slow everything down. My current experience has been that I have tried 3 motors for this project, 2 of those spoke CAN Open, one of them I rejected at the time as clearly they just did not have decent alternative IO options and I did not have CAN Open as an option. Now I do, if I had been confident enough at the time to put a hold on the project and develop CAN Open at the outset, the job would be done by now. That is the power of standardization. The next project I will work on I will simply plug the same electronics in and get it working.