A rant about lack of support on open source software, or anything else actually

[peter-h]

I have a very small business and have no real money to spend on R&D.

So I've had a guy working here Mondays (realistically, just the afternoon) doing various stuff like server IT. He also has embedded experience, in areas which supplement mine.

3-4 years ago we decided to develop a new product, which has an ethernet interface. TCP/IP etc was well above my pay grade (which is mostly embedded/comms/control) plus I had not done any C for about 10 years. So he started on it, Monday afternoons. Started with one ST dev kit, and after a year I did the PCB, which worked 1st time. Another year later I realised progress was minimal and it would take another 10 years, perhaps 100 years, to get anywhere (obviously, half a day a week means you forget much of what you did a week before) so I got stuck in, went up the steep learning curve, and have done most of what needs to be done. Crucially I know what I have done, and it is all 100% solid code (solid except for the simple http server used for config ).

I quickly learnt that you end up working alone.

No support from ST (32F417). They have a forum which is full of desperate people, no ST presence (they are only a €14BN company, so they can't afford to support their products) and virtually no useful replies from anybody. Just one guy occassionally posts good stuff but only after he's told you that you are a complete moron. They do loads of videos which are barely legible (very poor English).

No support on LWIP. There is a mailing list which is dead. Vast amounts of bogus info posted everywhere.

No support on MbedTLS. There is a mailing list which is dead.

No support on FreeRTOS. There is a mailing list which is dead. Fortunately FR runs well.

No support on FatFS. There is a mailing list which is dead. Fortunately FF runs well.

There are also some forums for above (which are also dead).

Then there are many clickbait sites which copy stuff from other sites, in bulk, to get google hits.

93.7% of the internet is desperate people asking questions on the above and getting no answers, and sometimes getting duff answers. Then there is Github (almost everything gets lifted off Github, which you discover by googling on source code comments), and probably a similar % gets lifted from linux sourcecode if your target is 80x86.

There are some more general forums (notably this one) where you do get useful replies - if you pick the topic carefully.

The Monday guy spent literally years (of Monday afternoons) trawling the internet for bug fixes and patches for the above stuff. He won't post questions anywhere, which makes it a lot harder. But now his part is nearly finished. We have a good running board. We are tidying up the loose ends.

I have learnt a lot and now know how most of it hangs together. In 1-2 years I learnt basic C (still don't get pointers to pointers), LWIP, FreeRTOS, FatFS. A lot of 32F417 bare-metal. But it's been extremely stressful. Nobody to even ask about some bit of C syntax (except on Mondays).

And it is clear that there are many others in my position, trying to get info and eventually, after an absolutely vast amount of googling, eventually finding some tiny snippet of info which happens to solve their problem. Then onto the next problem, and more absolutely vast amount of googling. A few hours later, that bit is running.

How do other people manage?

If you are hourly paid then you don't need to care. Another 1000 page RM = bread on the table at home for another year. What is not to like? The company doesn't matter.

I have looked for people who can do specific portions on the product and found two. Both got boards sent and the Cube dev kit so they could develop properly. One worked out well and did two modules, fixed price. The 2nd I had to stop after he ran up a vast number of hours and went off on a complete tangent (I should have done that fixed price too but for some reason didn't). I paid him but cancelled a part of the project which he was going to do also (involving a server on the web, which he does a lot of). A 3rd chap, fixed price, wrote some code which due to me not doing a precise enough spec he wrote for one of the ST dev kits and I could not run any of it, but I was able to use some of the code. But I have been totally unsuccessful in finding anybody who can do the ETH end (ST ETH, LWIP, MbedTLS). I need someone who really understands that pipeline who can go over all the code, apply latest patches, etc. and be around long-term, and be generally contactable on days other than Mondays. Probably, there is nobody. Obviously only really good people want to work fixed-price

It's a really inefficient way to work.

And it was never like this in the past, when chips were simpler. I developed literally hundreds of products over 40 years, all working alone. Just did it from data books. No internet.

The problem is the new functionality. It is so massively more complex.

[Foxxz]

Pretend the open source code doesn't exist. Then take the actions you would have taken had the code not existed.

I don't get where "support" is expected for any open source code.

[pcprogrammer]

I can feel your pain.

Yes the open source world is not to good on documentation and also technical documentation of manufacturers are not top shelf anymore. It takes effort and good insight to get things done. And indeed when one chooses the "wrong" topic to play with there is not a lot of help around. This forum is quite good and certainly helpful with lots of people with different skills.

Your ethernet quest is one I have on my playlist too, just for hobby because I don't do paid work anymore. But the playlist is long.

My venture at the moment is reverse engineering a FPGA design. Not easy either and not a lot of info to be found. But that is the challenge.

Success with your venture,

Regards,
Peter

[peter-h]

Pretend the open source code doesn't exist. Then take the actions you would have taken had the code not existed.

Then, small electronics companies could not develop anything of the complexity needed today. The chinese would totally dominate, with 80x86 boxes running linux and Siemens etc would eat the rest, where people don't want to buy chinese junk, or it doesn't exist. Today they dominate almost totally Market segments accessible to small companies are very narrow now.

I don't get where "support" is expected for any open source code.

Fair point. One would think somebody would have set up a forum with specific sections for specific OS software. Mailing lists are a 1980s/90s thing

[Berni]

Support is boring work, nobody wants to do it.

The difference in the old days was that you would pay thousands of dollars to buy a library like a TCPIP stack. That price of the license also included some form of support. If you don't want to pay up then you had to develop it yourself, putting hundreds or thousands of man hours into it.

Over time some people did develop it themselves and offered it for free to the world. You already got the code for free, there is no obligation from them to provide support. Tho for some projects (like FreeRTOS) you can pay to get support.

Large chip manufacturers like ST also provide support trough there local offices, but you have to be a reasonably sized company for it to be worth there effort. They organize seminars where you get to talk to there engineers one on one (including free food at the hotel it takes place in). You can talk to the local sales office to get early engineering samples of new products..etc

I have used FatFS, FreeRTOS, LWIP... etc and they are awesome libraries with what they can do. Certainly some of the better examples of open source projects. But have i ever had a problem with them? Yep, plenty of problems where things didn't work. I just had to sit down and figure out exactly what is going on, but in a lot of cases it turned out to be partially my fault for not quite using the library in the way it was intended. A lot of the time i actually learned a fair bit in the process on how the library works. Modern chips have excellent in circuit debugging capability, so this helps hugely in tracking down bugs.

Open source libraries are not to be treated like a product that you just use. That's the point of it open source. You can look at the source and see how it works to find problems. Sometimes you might find an actual bug in it. In that case it is usually the fastest to fix the bug yourself and then send the code back to the project owner so that they can integrate the fix into the next version. This is my way of thanking them for providing me with the code that saved me from having to make it from scratch on my own.

[paf]

Very different perspectives on your issues.

Money: If you have no money, but you have time, then Open Source is for you. You will spend a lot of time studying and you can read the source code. Without Open Source, you would need lots of money to have access to software.

Layering/Scaffolding: All the stuff you want to do is complex, so you should start by first learning C, then embedded with the STM32, then networking.

Complexity: Yes, look into the 32F417 data sheet, and try to think how to do what the 32F417 does with Z80s...   

The right tools: Why not use a Linux SBC board, where many things that you need are already done?

Debugging: When developing something, you need the knowledge/tools not only to develop it, but also to debug it. Are you familiar with ethernet tools like wireshark?   


It may seem I am "beating you", but I am also an "old guy".

[JPortici]

dead

which is why me and some other users tend to share our code snippets on the microchip forum.
Documentation is incomplete and find a solution?
Open a thread so that others spend less time searching for solutions, there are almost only pros and no cons:
- posting your code and the following discussion may result in others finding hidden bugs you didn't think about, improving overall quality
- seeing that you contribute may also lead others to contribute (but it's almost always wishful thinking)
- you have no idea on how many times it happened to me that i was searching for a problem and found my own threads and solutions from months earlier.. d'oh!
- depending on the community, ster some time community may not be dead anymore. But there are places that can't be saved (anything stackexchange for example)

I expect free support from the manufacturer, after all i am buying the silicon and filing compiler/library bugs for them that eventually get solved
And if expect free support from the community as i try to exchange information with information

[madires]

In an idealistic world you would have wonderful documentation covering every possible aspect and forums with highly motivated users providing great support for open source. But we humans are so greedy, even demanding to get free support for free software to make money.

[peter-h]

I would definitely expect free support from the manufacturer. What they are doing is pretty incredible. They get away with it because they are all doing it.

Obviously nobody will support open source stuff for free but you could have a donation funded site (if it is actually good). I run a tech forum (not electronics related) which is donation funded and we get a few k a year. Scaling that up say 10x you have enough to pay some part-timers. But they would have to know their stuff. No adverts, but you could have adverts too. It would be an idea for EEVBLOG to set up these topics, perhaps? A lot of forums (with advertising) have paid moderators, after all. Those which have unpaid mods tend to end up being modded by weirdoes who hate people

Re "no money" I wouldn't say I have literally no money. But employing even one person on say 50k is extremely expensive.

I also didn't say "free support" There is the pretence of support (which is indeed free) but it is virtually useless.

I wonder if anyone here has gone the commercial route and paid a vendor for say a TCP/IP stack and an RTOS? What sort of numbers?

[DiTBho]

I quickly learnt that you end up working alone.

Yup.

[voltsandjolts]

They have a forum which is full of desperate people, no ST presence ...... Just one guy occassionally posts good stuff but only after he's told you that you are a complete moron.

Hehe, thanks for that line, made me LOL.

Regarding open stuff, someone on the forum here said "we can't afford the free stuff", which is often a very apt phrase. I wonder if shelling out on full Keil or Segger libraries would have got your project done in a fraction of the time, or would different problems crop up, who knows?  If the libraries from either of those vendors completely covered your requirements it would certainly be a big time saving over integrating various open stuff....and we'd likely have a lot less forum threads (sorry, couldn't resist!)

[nctnico]

They have a forum which is full of desperate people, no ST presence ...... Just one guy occassionally posts good stuff but only after he's told you that you are a complete moron.

Hehe, thanks for that line, made me LOL.

Regarding open stuff, someone on the forum here said "we can't afford the free stuff", which is often a very apt phrase. I wonder if shelling out on full Keil or Segger libraries would have got your project done in a fraction of the time, or would different problems crop up, who knows?  If the libraries from either of those vendors completely covered your requirements it would certainly be a big time saving over integrating various open stuff...

From what I've seen commercial libraries aren't without issues. Often the vendor oversells the production readiness and level of support while the buyer grossly overstates the number of units sold. As a developer you are left with a binary blob with unknown content you need to find workarounds for in order to get the project out of the door. With open source at least get the source so you can hack & fix showstoppers.

IOW: There is no perfect world.

[HwAoRrDk]

My experience with paying for proprietary solutions from vendors - not in the embedded world, I should add - is that it tends to go one of two ways: a) everything is wonderful, the product is great and meets all your needs, well documented, easy to get going with; or, b) the vendor's product is actually utter shite, doesn't do everything it claims to do, is horribly documented (despite perhaps having a sheen of goodness or greatness of quantity), support is severely lacking in answers if you ever stray one millimetre from the beaten path, and you regret your decision (or are bitter that such a thing was forced on you) and wish you had gone with an open-source solution.

Sometimes there is even an inverse correlation between price and quality - the more you pay, the worse it gets.

C'est la vie.

[peter-h]

I wonder if shelling out on full Keil or Segger libraries would have got your project done in a fraction of the time, or would different problems crop up, who knows?  If the libraries from either of those vendors completely covered your requirements it would certainly be a big time saving over integrating various open stuff....and we'd likely have a lot less forum threads

I have not seen much posted on these topics. Perhaps because the licensing is under NDA. If you get source, it definitely would be. For some mission critical stuff people use VxWorks and reportedly that is very expensive. One can dig up a load more e.g. https://www.lynx.com/embedded-systems-learning-center/most-popular-real-time-operating-systems-rtos. Some of the comments on support are not exactly flattering...

From what I've seen commercial libraries aren't without issues. Often the vendor oversells the production readiness and level of support while the buyer grossly overstates the number of units sold. As a developer you are left with a binary blob with unknown content you need to find workarounds for in order to get the project out of the door. With open source at least get the source so you can hack & fix showstoppers.

I think widely used OS code like LWIP / FreeRTOS can't have really obvious holes. MbedTLS, my feeling is a bit less so, if only because there is a lot less TLS in the embedded world (a lot of TLS type stuff is automatically done with 80x86 boards running linux - as many here have pointed out, while slagging off MbedTLS). And MbedTLS is huge - 150k of arm32 code, which is many times bigger than anything else. And given the lack of responses on their forum, there could be major holes there.

the more you pay, the worse it gets.

The more you pay the longer the NDA will definitely be, so nobody is going to find out

In general, I think the OS software scene could be done a lot better than it is now. Most of the "support channels" are dead, and have been dead for years. A lot of google hits on fairly common issues dig up posts from say 2005. Then you go to that forum/list and find it has been dead for last 10 years. Well, lots of posts (so it has great SEO, because google loves long techy prose) but no answers. Look at http://elm-chan.org/fsw/ff/bd/ for one of many.

As a result, googling turns up 90%+ total junk. I am on ~3-4 lists myself so I get a reminder every day, of people out there working on similar projects and nobody getting any input. I suspect most people who were on those lists in the good old days have either left or have blacklisted them so they go into junk. You get the same with forums and you see it as admin.

Some fresh air is badly needed. But not yet another new forum. It takes too much work to get a forum going (I did it 10 years ago but I had a great address list of about 5k). EEVBLOG could do it.

[HwAoRrDk]

I have heard it posited that the reason why some open-source community support sucks these days is because a lot of newer-generation users of these products are oblivious to, don't want to use, or don't expect to have to use more traditional forums of community support like mailing lists, forums, or IRC. They're expecting things like Discord, Slack, GitHub, Facebook, etc. So when these people end up using some open-source code from a decades-old project that is still on SourceForge, only has a mailing list, the primary project maintainers might hang out on some obscure IRC server once a week at random times, and only uses SVN for version control, then it's no surprise that they don't feel like getting involved in the community because it's all via means they're totally unfamiliar with. So therefore you get a dwindling or absent population in the community support channels that do exist.

There's something to be said in those cases for "get with the times, man!", but then again I've read a lot from people in the open-source community who hate that with every new project that's all "just hit me up on my Discord server" you lose public discoverability and permanence of record, because these channels are mostly invite-only, real-time, and have no (or limited-period) archiving.

[SiliconWizard]

Part of the problem is indeed the way "support" is spread out between many different media. Speaking of github, one funny thing is that it's supposed to have become the #1 place for open-source software, yet most of the "big" open-source projects such as Linux do NOT handle tickets/support/bug reports through github whatsoever - it's just basically used as a mirror. They still use mailing lists and such, but yes, people are now less and less used to using mailing lists. So many do not bother.

As to using open-source software in your own products: that's a decision that should not be taken lightly.

I've used open-source *tools* such as GCC extensively, but I've personally tried to limit the use of open-source *libraries* and otherwise code to the bare minimum possible.
One library I've used the most is actually libpng. It works well, it's stable, it's a "reference" implementation and I've never had any need for support or any problem with it. But that's almost an exception.

[brucehoult]

I've used open-source *tools* such as GCC extensively, but I've personally tried to limit the use of open-source *libraries* and otherwise code to the bare minimum possible.
One library I've used the most is actually libpng. It works well, it's stable, it's a "reference" implementation and I've never had any need for support or any problem with it. But that's almost an exception.

I don't know if I've used libpng. I know I've used libtiff. My most-used library might be zlib (which both of those depend on), or maybe even more likely Boehm GC (which I've helped enhance and port to new platforms).

[golden_labels]

I will not comment on ST, as in this case it’s effectively a part of their commercial offer and the lack of support is simply poor practice from the company. Open or not, doesn’t matter in this case. Happens with many businesses and is indeed telling something about how they treat customers.

As a soft dev I can tell that documentation is poor everywhere. If you are only a consumer of libraries, you may experience sampling bias: you get good documentation from some commercial products, because the documentation is a part of the offer. It itself is a product directed towards you, the client. But for everything developed for internal needs or deployed as a dependency only: good luck having any docs. Exceptions are really rare. And even then the information is rarely adequate: written from the perspective of the author, not someone learning the tool. The reason you see it more in open projects is because with them you are always on the inside.

Support in FOSS is universally poor, but there is also hardly a way to solve that. Providing a service costs money. There are no people queueing up to donate either their time and mental health,⁽¹⁾ or money. A hint: healthy relations with the developers help a lot, just like they in business. If you are perceived as a contributing member of the community, you are likely to get even quite extensive support. Unlike when you are just another rando demanding their problems to be solved.

As for the rest, I have a feeling that I see once more the old pattern. I wonder how much more time is needed for people to understand that in FOSS:

• “Free” stands for “free as in freedom, not free as in free beer.”
• “Open” stands for the philosophical concept of openess and not merely being able to peek inside.

… and that donating something — software in this case — doesn’t bind the donor with obligations.

Understand that, finally. Some people create a project and they share their work. With some exceptions,⁽²⁾ they don’t do it to promote it or offer a polished product. It’s made public, so others could benefit from it. But those “others” — that is YOU — are not customers. Your role is the same as the original authors: you join the community of people, who improve the work and now you are the producer. A company needs a feature: the company implements it. A company needs a bug fix or a particular optimization: the company applies it. A company wants a better documentation or support: the company provides those services. If a company can’t do it themselves, they hire other people to do the job. Just because you took something that was shared, doesn’t create a producer-customer relationship.

⁽¹⁾ Providing support is hitting you hard, if you really care.
⁽²⁾ To name a few: Linux, Gtk, Blender, PHP and Python.

[peter-h]

They're expecting things like Discord, Slack, GitHub, Facebook

Yes; dilution is the big problem with social media. Starting on Compu$erve in early 1990s (where you were forced to post under the name on your credit card, so people had to be extremely careful because even then it was easy to find out your employer) and moving through Usenet (which was good but got killed by spam and needing special apps, but still has some somewhat productive newsgroups e.g. sci.electronics.design and comp.arch.embedded) I saw the explosion of www forums (everybody who could do PHP and what became MariaDB etc, set up a forum, or a dating site), then a decade or two later things moved to the "instant gratification" media like

They're expecting things like Discord, Slack, GitHub, Facebook

to which I would add Reddit, and for "even lower IQ net users", Twatter, Instagram and now TicToc Reddit has good SEO but the content is mostly junk; the structure was set up to maximise SEO and clickbait income. FB has ~nil SEO (is closed) and is useless even if you are inside it. Discord is just another forum package (which I looked at for a project; another ££££/day maintenance nightmare since it is written in Ruby which was fashionable 15 years ago, so maintenance is like COBOL66).

So, there is clearly a demand for something better and more focused.

One cannot help those looking for 20-byte easy to eat stuff. These people have come to dominate software sites and everybody ignores them. "I have a dev board xxx, wrote code to flash an LED, but it doesn't flash, help me". Everybody will continue to ignore them.

But one could do a better job for serious coders. The original people are still out there; when I post on the LWIP mailing list (100% unproductive, and I do ask quite specific clear questions) I get a private email from a "big name" guy telling me to basically not bother doing it this way, so I reply to him with detail, and never hear back. If a new forum with clear sections was set up (say here) then you would fairly easily get people to come over.

[brucehoult]

They're expecting things like Discord, Slack, GitHub, Facebook

Yes; dilution is the big problem with social media. Starting on Compu$erve in early 1990s (where you were forced to post under the name on your credit card, so people had to be extremely careful because even then it was easy to find out your employer) and moving through Usenet

BIX was excellent from when I joined in 1986 until maybe the mid 90s. McGraw Hill eventually got bored and the software and data were passed to a group of the users and run privately for quite a bit longer as NLZ ("Noise Level Zero").

I didn't get on to usenet until 1989. It was better in some ways, with being full of academics. It was worse in some ways due to being full of ... academics.

to which I would add Reddit, and for "even lower IQ net users", Twatter, Instagram and now TicToc Reddit has good SEO but the content is mostly junk; the structure was set up to maximise SEO and clickbait income.

Some reddit groups are pretty good. I like to think we run /r/riscv pretty well. Most of the questions people ask are relatively basic, but I've been impressed at how highly technically knowledgable people pop up when a very technical question is asked.  I think r/asm is also not bad, and the level of homebrew electronics / homebrew CPU / retro computing content and discussion in /r/beneater is pretty amazing -- especially considering the host and only mod (until a couple of weeks ago) has been MIA for 10 months.

[peter-h]

Some reddit groups are pretty good. I like to think we run /r/riscv pretty well. Most of the questions people ask are relatively basic, but I've been impressed at how highly technically knowledgable people pop up when a very technical question is asked.  I think r/asm is also not bad, and the level of homebrew electronics / homebrew CPU / retro computing content and discussion in /r/beneater is pretty amazing -- especially considering the host and only mod (until a couple of weeks ago) has been MIA for 10 months.

The archival value is poor though. Lots if chitchat but few deep contributions. Like FB really but open to SEO.

[brucehoult]

Some reddit groups are pretty good. I like to think we run /r/riscv pretty well. Most of the questions people ask are relatively basic, but I've been impressed at how highly technically knowledgable people pop up when a very technical question is asked.  I think r/asm is also not bad, and the level of homebrew electronics / homebrew CPU / retro computing content and discussion in /r/beneater is pretty amazing -- especially considering the host and only mod (until a couple of weeks ago) has been MIA for 10 months.

The archival value is poor though. Lots if chitchat but few deep contributions. Like FB really but open to SEO.

I really can't agree with you there.

Very often someone posts a story that was already posted weeks or months before, or that is a follow-up to a story from a year or two previously e.g. someone announces a chip and then later ships it.

It seldom takes me more than a few seconds to find the previous post and add a link to it in a comment. Which the OP should have done, but that's laziness not difficulty.

[abquke]

I have a very small business and have no real money to spend on R&D.

R&D is expensive. Almost no way around it.

[Siwastaja]

Your key issue is having no experienced/capable people at all.

The guy working half of the Mondays does not count.

You need to work half-time (i.e., around 20 hours every week) bare absolute minimum, preferably full-time. Plus you need to know what you are doing.

One such person can be enough, but zero is not. Also trying to learn C from scratch while working on a large project, while trying to maintain a business, is also not workable, especially if you are not so young anymore. It might work when you are 25yo and energetic.

Yes, finding the right people is
A) difficult,
B) expensive.

There is no way around it. It's completely wrong expectations that "just using software libraries" would allow anyone to develop. It's not like that at all.

Experienced people know what libraries to use and when because they have used them (or similar) before.

Multiplexing time is problematic. I'm currently working on two totally different projects and it basically means one of them just stalls, me being unable to advance it even near the rate I would like to.

[peter-h]

I will soon prove you wrong on several of your points

But you are right on others e.g.

You need to work half-time (i.e., around 20 hours every week) bare absolute minimum, preferably full-time.

is true to get something complex finished.

Actually

especially if you are not so young anymore

is quite funny. I was never very bright (nowhere near as bright as the sort of people I would have working for me) but I haven't got any more stupid at 65 compared to when I was 30, and I am happy with that There is nothing special about C (which I have used before) so long as you avoid esoteric stuff.

It is also obvious that the vast majority of people using libs like LWIP have little or no idea what they do internally. There is a POV on this and other forums that you should not get involved in say TCP/IP unless you can write your own stack, but that just isn't the real world.