STM Cube: distributing a project where not all modules are supplied in .c source

[peter-h]

32F417

The way Cube normally works is that the source code tree is compiled into .o files into the DEBUG directory. This directory is totally cleaned out beforehand with
rm *.* or some such.

There is also a RELEASE directory but the distinction is purely artificial; you can build everything in DEBUG and just change the optimisation level under project properties.

I want to set it up so some people can develop code but I don't want to give them all the .c sources.

There is a whole load of complicated ways to do this. Perhaps the neatest is to wrap up all the .o files into a library (.a), but the script for that is not trivial. And it doesn't actually give you anything because anybody can list the library and get the .o modules. And you still have to distribute .h files etc. So you just create a lot of work for yourself.

I was thinking of simply deleting the applicable .c files, collecting up their corresponding .o files and saving them somewhere, and adding a path to the linker command line where the linker can look for .o modules.

I wonder if there is some neater way. Unfortunately Cube is a vast labyrinth of scripts and every time you install a new version you overwrite these. So this needs to be something which can run as a standalone batch file. I could even run that as a post build step in the primary project; I already have stuff in there to e.g. add a crc32 to the binary output file.

[GromBeestje]

I assume you mean the IDE with Cube. This uses afaik the eclipse build system. I am not that familiar with it.
The standard way to do what you indent is indeed to create an .a file.

I am working with a bunch of Makefiles myself, and I have created a way to create such libraries. You'll have to supply the corresponding .h files for the functions yourself.

The way I create a static library from my environment is like

See for example https://github.com/a-v-s/ucdev/blob/master/build/staticlib/rtt/Makefile

You have to set your target architecture. Eg. An STM32F0 series is a Cortex-M0,  STM32F1 series is a Cortex-M3, etc.

Code: [Select]

ARCH?=ARM
SUBARCH?=M3

Other then that, you add your source files and include directories required to build the library.

Code: [Select]

C_SOURCES  += $(RTT_DIR)/RTT/SEGGER_RTT.c
C_SOURCES  += $(RTT_DIR)/RTT/SEGGER_RTT_printf.c
C_INCLUDES += $(RTT_DIR)/RTT
C_INCLUDES += $(RTT_DIR)/Config

Please note, the header files required so *use* the library can be less then the ones you need to *build* it.

The makefiles assume the cross compiler is in the path, and for arm is prefixed by arm-none-eabi-
This can be overriden by defining PREFIX, etc one can set [code[PREFIX=/opt/gcc-arm-none-eabi-9-2020-q2-update/bin/arm-none-eabi-[/code]

I am not that at home in the eclipse build system, or what modifications the Cube IDE has wrt standard Eclipse, but it should have some place in the build configuration where you can specify libraries, where you specify the library. It will have a path and a library name (without the lib prefix and without the .a suffix)

In the raw makefile, you'd do
So, when you got your .a file
/some/path/to/library/libexample.a
you add the linker flags,

Code: [Select]

-L /some/path/to/library -l example


[artag]

I don't see why distributing a bunch of .o files is any different from a .a file (except it's less convenient for the user). A .a file is literally a bunch of .o files in one. It only differs from a tar or zip file in that the linker expects only .o files and can search them for the references they contain exactly as it does .o files. In a link script the difference is primarily that specified .o files will always be included whilst only those matching link references will be pulled from the .a file and included in the build.

If your aim is to obscure the names of functions .. you can't really do that and allow the end user to do a link. It's what a link needs.

I guess you could be trying to expose the API but not the internal parts of the build. In that case you could consider providing the API via a jump table and a separate binary. The jump table has the only symbols visible and implements some sort of bridge to the actual binary. I think a number of devices with soft fuinctionality such as the espressif chips, Nordic nrf51/52 and cypress ezusb-3 do this. It's a lot more maintenance for you though.

Another possibilty is to use  meaningless internal function names, perhaps supported by #defines so that it's not completely unmaintainable. To be honest, in this age of open source, it feels like you're just creating extra work and making your offering less attractive to customers.
 

[peter-h]

Thanks all.

Let me fill in more detail.

I have a Cube IDE project which comprises of

- some ex Cube MX (the STM "code generator") code, largely modded to remove bloat; the project was started years ago with this bit
- ST port of FreeRTOS (very few mods were needed)
- ST port of LWIP (lots of mods were needed)
- ST port of MbedTLS (lots of mods were needed)
- ST ETH code (loads of mods again)
- ST USB code (client only - CDC & MSC) (load of mods needed)
- ST port of FatFS (very few mods)
- interface code for SPI serial FLASH
- interface code for a number of serial ports
- a couple of LWIP-interfaced modules written by outside people (no doubt they will recognise this )
- a ton of code I wrote over the past 2 years of my life; stuff which defines what the product actually is, an API for a ton of stuff, and countless bug fixes to stuff above, revealed in extensive 24/7 testing

The project is in a directory tree:



and since I didn't set up the ST port stuff, I don't know where and how these directories are set up, but I don't need to know. I know how the stuff I've been writing last 2 years works

I am not posting what this thing does but I can say that in some application contexts there will be outsiders (with C expertise) writing their own modules. They will be given the whole Cube project, with an example file like hello.c. I will absolutely not be supporting ST Cube and creating a project from scratch; that is way beyond the call of duty.

And the idea is that their life will be far easier than if they were coding for some development board (these normally come with example code but usually it is crap for anything beyond trivia) or for some "hobby board" (which tend to be mechanically unsuitable for serious industrial use, as well as production-problematic long-term). For example LWIP and proper code examples will make the development of internet applications a lot easier.

And I don't want to give them all the source code. They can have the ex-ST code (even though the fixes to make it work probably amount to a man-year) because it is basically public-domain, but I don't want to give out the whole lot, especially most of the stuff which I wrote.

These other coders will write applications which run at base+32k (0x00008000) because the bottom 32k is a boot block which has various recovery features, etc. Obviously I won't publish the boot block code, too.

So I am looking for the simplest path to generate this "0x00008000" development kit.

The main difference will be the linker script, obviously. That's easy. No boot block section, start collecting TEXT sections at 0x00008000, etc.

Then I need to delete the .c files for all the stuff I don't want to hand out. But I need to provide the .o files. I also need to provide all the .h files for the entire project and I am hoping that I can use all the existing ones as-is.

So I need to copy the .o files to another directory e.g.

<projectdir>/O_FILES

and add a linker command line option to look in there.

The stuff should be a standalone batch file. Something obvious and self documenting so in 10 years' time it will be obvious what it does.

The batch file can be run as a post build step, so the above .o files are always current.

Out of maybe 1000 .c files there would be only about 30 that would be stripped out.

I am not worried about function names being visible. I just don't want to hand out nicely commented and valuable C source in some cases

Have I overlooked anything?

The "neat" way would be to produce one huge .a library file, but that is just "neat" and achieves nothing, given that Cube itself is about 4000 files! And I would have to do what I think would be a vast amount of work on the .h files. Also I have no idea how one would condense all that code in that directory tree into one library, and keep the .h files right. There are multiple pairs of src and inc directories so e.g. LWIP has one pair, FatFS has another pair, and so on.

So I am looking for the simplest way to be able to remove some .c files and supply just the .o files, and this needs to survive the rm *.* cleanup command which Cube does on the Debug directory at the start of each build of a project which has been cleaned-up.

[thm_w]

Get them to sign an NDA.

[peter-h]

That's really funny.

[artag]

That's really funny.

I'm inclined to agree.

But what is it about NDAs that makes corporate types happy ? Is it just an arse-covering thing that ensures they can tell their boss it isn't their fault the competitor knows their secrets ?

[Jeroen3]

ARM CMSIS Maths package is distributed via .a files and linked into stm32cube no problem. That should be the way to go.
Take the files you want to hide. Compile them into an .a, link this into your project.

Then you share the workspace, you omit this project that compiles the .a, but keep the output directory that the other projects refer to.
Note that you won't be able to use the "depends on' subproject options in eclipse if you intent to remove the project.

You can even apply this method to all your other lirbaries you use, compilation time must be horrible in this project because everything recompiles all the time.

[peter-h]

The recompilation is only for modules for which the "makefile generator" in Cube cannot find the .o file.

So if you do a Clean Project, you do indeed get the lot recompiled.

This means that you could give someone the whole project, having just deleted the "non-distributed" .c files, and it will work just fine. Until he does a Clean Project and then he will have to restore these .o files. The rm *.* command doesn't run on every build - sorry if I suggested that.

But why would someone do a Clean Project? Well... Cube doesn't always 100% notice that some file was externally edited and just dropped into the project directory. So when I am setting up the Cube project on a new machine (using the Import function - this is easy to do wrong but easy if properly documented step by step) I always do a Clean. Or someone could do a Clean by mistake. But that isn't fatal because they will have the .o files in an archive anyway.

I am just really confused at how a Cube project gets built up. For example where the ST Port of LWIP comes from and how it gets loaded into Cube. But luckily all this has been done.

[SiliconWizard]

That's really funny.

Well, funny. Dunno about that.
From what I got, you allow your customers to write custom firmware for your products, so you need to distribute the firmware in whatever form is most convenient for you while being usable for your customers.
While NDAs are OK for B2B, you don't force end-users to sign NDAs to use a product. I guess that's what you found funny, but thm_w may not have had the whole context.

So while a NDA per se may not make much sense in your case, surely your products and associated firmware are disitributed with some kind of license agreement?

[thm_w]

That's really funny.

Well, funny. Dunno about that.
From what I got, you allow your customers to write custom firmware for your products, so you need to distribute the firmware in whatever form is most convenient for you while being usable for your customers.
While NDAs are OK for B2B, you don't force end-users to sign NDAs to use a product. I guess that's what you found funny, but thm_w may not have had the whole context.

So while a NDA per se may not make much sense in your case, surely your products and associated firmware are disitributed with some kind of license agreement?

Ah I didn't realize this was for end customer use, all I see in the OP is "I want to set it up so some people can develop code but I don't want to give them all the .c sources.", and I assumed it was contract work being done.

[peter-h]

It's actually a bit of each.

My point was that NDAs are worthless because if somebody wants to rip off your product they can do so and your only remedy is a $10k/day lawyer. And if you want to take action across country borders, the 10k will just be wasted.

[Jeroen3]

You know you can read up on how Cube Eclipse works, right?
You can also see the makefiles that are generated and how they fit together.
Sure Eclipse is somewhat strange, because it's a bit older.

One thing you absolutely should not be doing is working against Eclipse. It will make your job hard, and it will not align with the expectations of your customers.

[peter-h]

You know you can read up on how Cube Eclipse works, right?

No doubt you can, but you would need a PhD

This is Cube

[John Coloccia]

Curious why you want to keep the source code private? FWIW, not distributing in source form makes it much less likely that I would consider using any third party library in an embedded project.

As a side note, I've yet to see Eclipse generated makefiles that work properly, and generally speaking the way they detect/handle dependencies is atrocious...and often just flat out wrong. It's not uncommon that I'll just write my own makefile and be done with it.

[peter-h]

The product is a hardware one. I described what modules are in it above, and while most of these are open source (albeit many containing bug fixes) there are a few which aren't O/S. Of these, a few are quite specific to the functionality and I don't want to hand out the source code to those.

Re Cube generating wrong dependencies, I found all kinds of weird stuff when we first started using it a few years ago, but nowadays it seems to work ok, even correctly detecting which files were edited externally (which didn't used to work, despite a config option for it).

But still an easy recommendation if somebody is having trouble is to do a Clean Project. That deletes everything in the DEBUG directory, but creates the described problem that for some of the .o files will not have had a .c file supplied.

[eutectique]

But still an easy recommendation if somebody is having trouble is to do a Clean Project. That deletes everything in the DEBUG directory, but creates the described problem that for some of the .o files will not have had a .c file supplied.

Then you need two projects:

• one for yourself, building: [ *.c + *.h ] --> *.o --> secret-lib.a --> link to binary
  
• another for distribution: *.h + secret-lib.a --> link to binary
  

This is essentially done for libc, you will find all your headers in include directory and corresponding libc.a in lib/arm-none-eabi/newlib/thumb/v7e-m+fp/hard directory (or whatever your architecture is).

[eutectique]

BTW, I always say make clean when I switch branches, or cherry-pick a commit, or merge my branch with master. Or start my day, for that matter. Consider this the normal practice.

[peter-h]

Then you need two projects:

Exactly!

I am just not sure how to collect stuff into a .a file. Under DEBUG I see a load of .o .d .su files.

This is essentially done for libc

I have already replaced libc.a with one which is a) weakened and b) has had the printf family removed (and I have a replacement printf which doesn't use the heap, etc - various previous threads).

I always say make clean when I switch branches, or cherry-pick a commit, or merge my branch with master. Or start my day, for that matter. Consider this the normal practice.

I concur, but during a day's development this is not needed.

It is obsolutely necessary when reverting to a previous version of a project.

[eutectique]

Then you need two projects:

Exactly!

I am just not sure how to collect stuff into a .a file. Under DEBUG I see a load of .o .d .su files.

-- In your in-house project the final binary should depend on secret-lib.a .
-- This library, in turn, should depend on, and be built from, the number of .o files.
-- These particular .o files should depend on .c files that you want to keep inside.

Forget .d and .su, these are text files with dependency and stack usage information. You only need object files to create the secret-lib.a .

[abyrvalg]

Google "stm32cubeide create library" returns ~101000 results, but you don't need a PhD to pick the first one 

[eutectique]

One more thing.

You would want your secret-lib.a distributed without any debug information, stripped and optimized to death with -O3 or -Os. For that, you should have a separate set of CFLAGS for the sources which make up the library.

Probably, consider creating several variants of the library, and name them appropriately, for your own confidence and peace of mind, like
-- secret-O3-release.a (for external use)
-- secret-Os-release.a (the same)
-- secret-Og-debug.a (for yourself)
-- etc


Another one more thing

Consider creating a file with version and build information, say secret-version.c which will be included into the library. It should merely contain one line:

Code: [Select]

__attribute__((used)) static const char secret_version[] = "(@) " __SECRET_VERSION__ " built on " __DATE__ " at " __TIME__;

Then, the version can easily be had with

Code: [Select]

$ strings secret-lib.a | grep \(@\)
(@) gimbal-2.7-RC-ga865e9142-dirty built on Mar  9 2023 at 15:14:38

Supply __SECRET_VERSION__ to the CFLAGS mentioned above.

[peter-h]

This is brilliant stuff - thank you very much!

Before I read the above, I did a little hack to see if there is an easy way to get the linker to pick up .o files from some directory. There doesn't appear to be; this is possible only with libraries (.a). However I found this bit



where you can create a directory and put .o files in there, but you have to specify all the .o files individually. Well, I have about 20 of these "no source provided" files. Making a library would be neater - especially as an automatic post build step in Cube.

So the next step will be to use the Archiver (ar) tool to make a library e.g.

http://www.delorie.com/djgpp/v2faq/faq8_22.html

Funny you mention debug level. I am currently running with Max. But varying this makes no difference to the binary size. It sounds like it varies the symbol information loaded into the .o file - correct?



I am not actually that bothered about symbols in these files - I don't have stuff like "TOP_SECRET_RSA_KEY" in there I belive M$ did that once in WinNT.

[abyrvalg]

Seriously, the first google result contains a detailed instruction how to configure CubeIDE to produce a static library: https://community.st.com/s/question/0D53W00000HNPWISA5/writing-stm32cubeide-libraries-what-is-the-best-practice

-O3/-Os and symbol stripping are to make decompilation of your *.o back to C harder.

[peter-h]

Sorry, but I am too thick to understand that post

Looks like my best option is to knock up a batch file which collects up all the .o files (for which I am not providing the source) into a .a library, and add that library to a linker search path.

Then I need to work out how to change the compiler debug-level just for those source files. I found that - it is under Properties for each file.