[CrowdLaborFunding] PCB CAD tool idea

[Bassman59]

I have no doubt that the Kicad developers would not say no to anyone offering to spec out and implement a full-up library system underpinned by a proper relational database. Nobody is arguing that the database system is a bad idea. I agree that it's the way to go.

But ... has anyone stepped up to offer to do all of that work? No. The silence is deafening.

Well I did consider paying someone to add a database based component management system to KiCAD. However I very much feared that the creators of KiCAD wouldn't even understand the benefits of having a database (or just ignore it) and I would be left with a fork of KiCAD which would get outdated quickly. Given the lack of understanding on why having a component database is the way forward shown in this thread it seems I made the right choice by bringing my money to Cadence. Also my experience trying to contribute to open source projects is that it is very hard to have your changes accepted by the creators. The only exception so far has been OpenOCD.

Have you approached the Kicad developers in any manner? Are you subscribed to the developers' email listserv? Do you know the state of Kicad development? Did you write a proposal for implementing this feature? Did you identify any developers with the skills necessary to implement them?

I think the answer to all of those questions is "No."

And since it's a lot easier to complain about something on an internet forum than it is to actually get involved, you took the typical way out.

[nctnico]

I have no doubt that the Kicad developers would not say no to anyone offering to spec out and implement a full-up library system underpinned by a proper relational database. Nobody is arguing that the database system is a bad idea. I agree that it's the way to go.

But ... has anyone stepped up to offer to do all of that work? No. The silence is deafening.

Well I did consider paying someone to add a database based component management system to KiCAD. However I very much feared that the creators of KiCAD wouldn't even understand the benefits of having a database (or just ignore it) and I would be left with a fork of KiCAD which would get outdated quickly. Given the lack of understanding on why having a component database is the way forward shown in this thread it seems I made the right choice by bringing my money to Cadence. Also my experience trying to contribute to open source projects is that it is very hard to have your changes accepted by the creators. The only exception so far has been OpenOCD.

Have you approached the Kicad developers in any manner? Are you subscribed to the developers' email listserv? Do you know the state of Kicad development? Did you write a proposal for implementing this feature? Did you identify any developers with the skills necessary to implement them?

I think the answer to all of those questions is "No."

I actually did some investigations in how feasible it would be including finding developers and chances on getting the work into the mainstream. This turned out negative. Remember I'd be paying this out of my own pocket with a limited budget so I had to minimise the risk to end up with a half baked solution. One risk for example: if none of the KiCAD developers would deem a component database a useful feature (or important enough) it could end up being abandoned. Especially with the changes to the schematics part of KiCAD in mind which have been announced a long time ago.

But no, I have not contacted any of the KiCAD developers. And that is on purpose; when you announce you want to contribute to an opensource project the developers usually say 'great, any help is welcome' but when push comes to shove you have to jump through all kinds of hoops to get changes committed / merged. It is (almost) never simple like writing an addition and supplying patches. I've based my decission on experience of other people trying to get changes made to KiCAD (see this forum message for example: https://www.eevblog.com/forum/kicad/one-annoyance-of-kicad-solved-f4-drag/msg933579/#msg933579 ). Some even ended up forking KiCAD which I really didn't want.

[free_electron]

that may be true for symbols. what about footprints. What if you restyle or fix a footprint that is in use on many compoentns.
let's say you are going to alter your 0603 footprtin from nominal to least. ...

that is whay the professional packages use a database.
footprints, symbols and parts. parts link to footprints and symbols and other data.
Cadence goes a step further. they have approved pad stacks. change a pad stack and all associated footprints update ! which causes all associated components to update and triggers board reviews on all affected projects ... ( something the altium vault can do as well.

proper design datamanagement is the most critical aspect of product design. screw up your library and even the best designed schematic and board will be useless if the parts don;t fit ...

Yes, everyone who does this for a living is well aware that a fucked-up library results in fucked-up boards and money wasted. Not only in the physical design with footprints and such, but also in what purchasing orders and what gets soldered to the boards.

But in your reply, you mentioned two expensive CAD packages, Altium and Cadence, both published by large companies with a lot of programmers on staff, and you're comparing their features to an open-source package with a dozen core developers.

I have no doubt that the Kicad developers would not say no to anyone offering to spec out and implement a full-up library system underpinned by a proper relational database. Nobody is arguing that the database system is a bad idea. I agree that it's the way to go.

But ... has anyone stepped up to offer to do all of that work? No. The silence is deafening.

It's the old saying, everyone talks about the weather but nobody does anything about it.

as is so common in the open source world : they are either
- all focussed on the eye candy. (example : 32 different desktop managers)
- stuck in their holy wars between VI and Emacs
- glued to their command line

as opposed to first understanding the problem.

The foundation of a cad system is data management. And that begins by a library system. Not by creating an glitzy user interface to draw lines form point a to point b.

I can spec out a system , but i am not a programmer. Don't look at me to write code. And the kicad programmers are clearly not PCB designers.
And THAT is the problem with all these open source things. The people who have to write the code have never gone through the pain of doing a 'real' pcb layout ( i'm not talking a double sided arduino style board where nothing matters ... but a real board taken fomr concept all the way to production hand-off including all documentation, life cycle , feedback management cradle to grave ...

The spec for a working system is not that hard
- have a way to edit and store symbols
- have a way to edit and store footprint
- have a component management system that allows you to tie one or more symbols to one or more footprints and add 'meta' data to create a 'component'. Meta data is textual data , files ( datasheets for example ). Meta data can also be links to external databases (through SQL , or through simple CSV files for example.) Meta data can contain anything you want and must allow for 'forking' of fields. for example the field 'Manufacturer' and Manufacturer part number' can have multiple entries
Same for field 'supplier'. I buy yageo form digikey and mouer and vishay from future electronics. so my meta data must allow for that. When bom production comes it must properly present that data.
Reason for multiple symbols per component : placement of pins. microcontrollers for example. have same part but with different pinplacements. Different footprints : least,nominal,largest)

- all of the above version controlled and journalled so i can roll/unroll and inspect revisions. the entire history needs to be maintained. Disk space is cheap. If a textual storage system (xml, json) is used it can be heavily compressed.
- all of it managed though a GUI, drag'n'drop copy paste.
- if you edit a symbol or footprint : notification of what is impacted where and push forward so parts update.
- symbols footprint and components have lifecycle levels : new, prototype, approved , eol , replaced. new parts can not be used in a design. they are being built. parts flagged as prototype cannot go in release builds. only approved parts can go to production. eol parts should no longer be placed. if a part has a replacement assigned : when trying to place such a part you get the replacement instead.

This master database should be stored in a concurrent multi-user format ( multiple people accessing the data pool at once without contention risk ) and using user permissions ( who can modify what) , Preferably without requiring a 'server' (a custom machine with dedicated application. should be able to run on a LAMP or WAMP setup) . It would be nice if all that stuff can be done through a SOAP connection and reside on a web server somewhere. Ideal from a collaboration perspective. you can have your own running on your own box, on a company box or on cloud space for the entire community

capability of connecting to more than one such source simultaneously and possibility to transfer data easily between systems so i can import or share between data pools easily.

The Cad editors pull from the component management system.
whenever a project is opened : rescan all instances of parts against the master database and approve/reject import changes. check for new,prototype, eol and replacements and provide detailed reports with approve/reject for each category.

That is your component front end. This can be completely split off from the cad system and even be a separate runtime, but there should be links. if i right click on a part in my schematic editor it should be able to show me all data , or open the correct editor directly

Next would be a similar setup but for projects. That would manage the files , output datasets etcetera. Same concept : manage revisions , manage output creation ( that really should be a one click thing... you set up a template once and forget about it. the tool should be smart enough to know what layers , and what all is needed )

[nctnico]

@free_electron: you forgot about BOM variants where one design can have multiple different BOMs... 

[SiliconWizard]

Indeed. There are actually many different use cases and workflows...

As the list grows, you'll quickly find out that what you really are after in the end is a full-fledged ERP. Anyone having worked on ERP systems knows how much work that means, and that beyond the base software, there is a lot of customization required for each customer's needs. Is your EDA software going to be customizable enough to fit everyone's needs? If so, you'll still be developing it in a few years.

My advice would be to keep it simple and make the database system interoperable with external systems, rather than trying to develop something that would end up being a monster.

One thing that comes to mind regarding the fact that each part of a design would be linked to a database entry. You absolutely need a revision system in your software if you want to do that. The fact that any modification of the database would impact ALL of your exisiting designs without any control would be very risky. It's a no-no when dealing with validated designs. It must be revision-controlled at the design AND database levels. Any modification of an exisiting design should be explicitely accepted, and not blindly automatic. So if any part modification in the database can impact any of your existing designs, the software should give you means of accepting or declining the changes for each specific design. If you decline, it would use the older revision of the part.

[nctnico]

One thing that comes to mind regarding the fact that each part of a design would be linked to a database entry. You absolutely need a revision system in your software if you want to do that. The fact that any modification of the database would impact ALL of your exisiting designs without any control would be very risky. It's a no-no when dealing with validated designs. It must be revision-controlled at the design AND database levels. Any modification of an exisiting design should be explicitely accepted, and not blindly automatic. So if any part modification in the database can impact any of your existing designs, the software should give you means of accepting or declining the changes for each specific design. If you decline, it would use the older revision of the part.

True. In Orcad Capture CIS you can choose to update the design with any changed components (in the database) and you can accept or deny a change for each part.

[free_electron]

@free_electron: you forgot about BOM variants where one design can have multiple different BOMs... 

That is in the output and project modules. the parts information database does not handle that.
I was mainly focused on the parts information database.

[donotdespisethesnake]

There are really two things here, which get conflated
(1) providing support in KiCad for an external database system
(2) creating and maintaining a database of parts which can be shared

I agree that proposing component database for KiCad (whatever that entails) is unlikely to get much traction with KiCad developers. Their focus is on a lot of other stuff, maybe in future they will come round to it. There is a related item on the KiCad "wishlist" https://bugs.launchpad.net/kicad/+bug/1750180. I don't recall seeing any roadmap items that relate to component databases.

In software, with a large chunk of legacy code, it's really hard changing the underlying data structures because your whole code relies on it. Changing to an "atomic" part system is a huge change for KiCad, so it's not going to happen quickly.

In the meantime, the only real options are
a) develop code outside of KiCad which provides similar features
b) fork KiCad
c) use something else, eg, Horizon

With a), you might be able to identify some minimal features that could be implemented in KiCad. For example, when searching for a part, there just needs to be an API that KiCad calls to get a list of parts matching a search query. KiCad does not need know anything about the database structure, or be bundled with a specific database engine.

I did actually try creating part databases by scraping DigiKey web site, but ran into scalability issues with KiCad. When there are a lot of parts in a library (e.g. 5,000) KiCad hangs then crashes.

So the way I see it working is that the main database holds as many parts as you like, and a subset of those is extracted to work with a specific KiCad project or set of projects. It's a two step process which is not ideal, but maybe better than nothing.

The real big issue with (2), and the reason I think it will never happen, is the sheer amount of labor required creating and maintaining a database. Vendors have people full time doing it, and their web sites are riddled with errors. Crowd sourced sites like kicadcloud.com (now defunct) had such poor data it was not worth going there.

I think any effort to create a component database system must be closely focused on the minimum change to interwork with kiCad, and to minimise the amount of labor required maintaining the data. While everyone talks about the need for component data, very few people are willing to pay for it. Those that do, probably go to one of the commercial suppliers.

[nick_d]

I have been planning to do something like this for a while, though it is not possible for me at the moment due to other commitments.

Currently I use KiCAD PCB to generate simple layouts with a textual annotation which I convert to G-Code using a custom Python script and then use to control a Mazak milling machine. And while KiCAD is up to the task it is definitely the weakest link, editing a layout takes ages and I am often dropping out to an editor to manually change things like coordinates, line thicknesses or radii. I tried using Inkscape but Inkscape is problematic for technical drawings for various reasons.

As an experienced developer who usually creates my own tools, my golden rule is this: create a framework that can someday accommodate all wanted features, BUT only implement the features that I immediately need now. That way I can get up and running fast. Many of my creations have been developed substantially from very simple and humble beginnings, the G-Code generator being a case in point.

In terms of the development tools, I have done fairly comprehensive experiments with wxWidgets and although I initially found it promising I have lost the enthusiasm slightly. C++ is also a very problematic language for a number of reaaons (I interviewed for a C++ job the other day and was horrified at the sample code given to me in a coding test).

The only way to use C++ well is to use it VERY sparingly, never use a template where a class will do, never use a class where a function or a variable will do, and so on. Use things like templates only for obvious cases such as std::vector. Luckily wxWidgets is written in a fairly economical style (it uses an early dialect of C++ before templates), but nevertheless. If you plan on a collaborative project it will quickly get gummed up by people who write C++ all day and never realize they are writing 100 lines of code in several files where 5  lines placed directly inline will do!

As a constraint programming researcher I must say I used C++ a lot, as the code was solving combinatorial problems so had to be fast to run through exponential solution spaces quickly. However, my research group produced truly horrible C++ code exemplifying everything you should not do and why you should not.

Instead I would do it in Python 3 with GTK 3, or perhaps I mean Gnome 3. Anyway, the latest GObject system is quite amazingly powerful for interfacing with high level languages, though it is far too clunky to try to code to the native C API as Gnome developers do. I agree that creating the pre packaged Windows and OSX installers is quite challenging by this route, however, it can be done.

cheers, Nick

[nick_d]

@freeelectron you have described a "dream CAD" system, which I suspect may be similar to a commercial system like Cadence, although I would not know as I have not used commercial PCB tools since MacCAD and Protel for DOS.

However, I must say that I would not need or want those advanced features. As a professional software developer who relies heavily on tools like Git, I can understand the importance of proper tooling, and intellectually I can kind of see why you would want all those advanced features if you are designing PCBs daily and maintaining the designs over a product lifecycle, but as a practical matter it seems to me that all those things would get in the way and create much added complexity and extra problems where there weren't problems before. I much prefer minimalism!

Having said that, everyone has their own way of working and feature set that is important to them. That is why I usually create my own tools and develop my own processes (complex sequences of scripts with small manual steps here and there), in an ad-hoc way. I feel this is more effective than the approach you describe which sounds sort of like the "Microsoft Word" of CAD packages where it attempts to anticipate the needs of some mythical advanced user and ends up featureful but clunky.

cheers, Nick

[Doctorandus_P]

Component databases in KiCad got mentioned on the KiCad forum recently:
https://forum.kicad.info/t/for-those-fans-of-atomic-part-libraries/15212
https://github.com/Projektanker/kicad-db-lib

I have no personal interest in the database thing for KiCad, but I find the list of 50+ external tools on
https://github.com/xesscorp/kicad-3rd-party-tools
very interesting.
It's a common side effect of open source projects. When they become popular side projects tend to sprout up, made by people with a personal interest in some extension, and sometimes they gain enough traction and a group forms around such an extention, or they just stay personal pet projects in various states of usefullness or decomposition.

[Bassman59]

I am a professional software engineer that has worked on projects at the embedded level and enterprise level. At my job, in the past couple of years, I have had more responsibility to be a systems integrator designing electronics. I have also worked extensively with embedded electronics through the years designing circuits and writing embedded firmware. Anyhow, I have done my fair share of research on the current state of EDA, and after the industry has lived for almost 40 years, I can say in 2019 I am appalled at the state of the tools.

Anyhow.. long story short, I think that a fresh start is needed for EDA with a focus directly on usability and the big problems with current EDA tools. Don't just run off and start a project without consulting the professional community, because 100 people will have 100 different ideas. If I tried to do that at my job, I would be destroyed, which is why I must consult first. I think what the OP has suggested to start with is good, and after using some of the tools and doing 'a lot' of research, I have some ideas of my own, which I would like to share. Additionally, I am contemplating starting development on this project (after approval of course, because if there is one thing you will find out about me, is I HATE doing useless work - there is no point to such a project if it is not used by others)

Payment scheme
Free - enough said. I am sick and tired of having to deal with a billion EDA tools that cost something for such a basic need. The tool would be open source as well.

Lastly, I would like to state that I found AutoTrax Dex recently, and after looking at the videos, it seemed like this tool would be a godsend for all the features I want to see in an EDA tool. However! I happened upon the long and exhaustive threads on this forum from the developer and.. yeah.. I won't use that software out of principle even though I think it is godly. I want to have a really great EDA tool like Dex that is free and without the risk of some jack charging for it.

So, lemme get this straight ...

You want the sun, the moon and the stars, but you are unwilling to pay for the creation of those celestial bodies.

Got it.

NB: I use Altium for the day job and Kicad for other work.

I will state that in order to produce a result in most of these tools, it is counter intuitive and users coming from a windows world especially expect easy point and click (including myself).

That right there is part of your problem: PCB design and layout for anything but the most trivial of designs is not 'easy point and click.'

You might wish to browse the Kicad developers' email archives. (I use Kicad as an example, because it's the most popular open-source PCB CAD tool out there and the developer discussions are easily accessed.) There is a lot of considered discussion about the user interface, hot keys, approaches to activating tools, all of that. It turns out that everyone has a different opinion and a different idea of how these things should work. What works well for me might drive you crazy. That's the nature of the beast, I suppose.

Finally:

The biggest issue with these tools is usability from the lay person's point of view.

There's the rub, eh? Seems to me that trying to design tools that work for both the casual lay person (a "maker" or hobbyist) and for the professional users is, uh, quite the challenge. The professional needs comprehensive design rules, net classes, hooks for proper BOMs, you name it, and those things might baffle the hobbyist.

-------------
I haven't even mentioned libraries ...

[pointhi]

I want to be able to select and highlight a part just with a single left click.. but instead I have to drag a selection rectangle around it, which is a huge hassle, especially if multiple things end up behind each other, and I want to delete one: this has happened to me at least 10 times since a few days ago because of the way KiCad movement semantics work. Again, maybe I have not found the right switches in KiCad to make it work how I expect.

When you try KiCad nightly you will see this behaviour is fixed with the new tool set .

You need to keep in mind there are technical restrictions which limit the possibility what type of UI developer can create in reasonable time. The Legacy Canvas + Tool system was such a restriction for a long time in KiCad. wxWidgets is another one. Their development speed is pretty slow, and many features planned for wx 3.2 are highly desired. Like proper high DPI support and dockable notebook pages.

In my point of view the best way to get a better ECAD is to improve it part by part.

[Bassman59]

I want to be able to select and highlight a part just with a single left click.. but instead I have to drag a selection rectangle around it, which is a huge hassle, especially if multiple things end up behind each other, and I want to delete one: this has happened to me at least 10 times since a few days ago because of the way KiCad movement semantics work. Again, maybe I have not found the right switches in KiCad to make it work how I expect.

When you try KiCad nightly you will see this behaviour is fixed with the new tool set .

You need to keep in mind there are technical restrictions which limit the possibility what type of UI developer can create in reasonable time. The Legacy Canvas + Tool system was such a restriction for a long time in KiCad. wxWidgets is another one. Their development speed is pretty slow, and many features planned for wx 3.2 are highly desired. Like proper high DPI support and dockable notebook pages.

In my point of view the best way to get a better ECAD is to improve it part by part.

I use the nightlies, and the advances made since even 5.0 was pushed out are great!

Improving part by part makes sure that a release doesn't break everything their users are doing.

Part of the reason for the "slow" development is that they try to ensure that things like the newest wxWidgets is available on all supported platforms. Python scripting is like that, too, with all of the various pieces all on different versions. It's crazy.

[Bassman59]

For all are interested in this thread: I had ditched this idea as I renewed my Altium subscription for another 2 years, and I learned KiCAD as a second plan.

There's a lot to be said for no longer tilting at windmills and instead just getting the tools you need to do you job.

[nctnico]

Bah.. I wanted an admin to change my username before I posted again, because.. I don't like my email as the username (made this account awhile ago and mistakenly used my email as the username). I sent an email to eevblog official to fix the issue.. but I don't think I will be taken care of in a timely fashion... so...... I digress

Multi components
Components with multiple sections - still thinking about the best way to do this

Look at how Orcad and Altium have implemented this. See below.

Integrated toolset
Schematic and layout are coupled in the same program. Same usability exists between schematic and layout (point and click features)

Not even remotely important in a professional tool. In Orcad I draw schematics in Windows and do PCB layout in Linux (this is a >$5k package without the very advanced options).

"Plotting"
PDFs are completely standard in my group.

PDFs are useless as output because you'll need at least Gerbers and ODB++.

One of the things you forget is that creating a board layout is all about logistics. Schematics and PCB layout are just two steps in the entire process. Managing parts and producing production data are even more important. The big error freeware and low cost CAD package often make is not thinking about the logistics first.

[free_electron]

Not even remotely important in a professional tool. In Orcad I draw schematics in Windows and do PCB layout in Linux (this is a >$5k package without the very advanced options).

You must not be doing a lot of crossprobing then .... ( click in pcb and have schematic zoom to part and vice versa ... )

It is VERY important to have the schematic and pcb talk to each other in realtime. only transferring netlists and pushing back eco's using simple files went away 20 years ago... There is not enough data in the file exchanges.
Protel 95 could already crossprobe.

I use that heavily.

[nctnico]

Look at how Orcad and Altium have implemented this. See below.

My first inclination is to just outright say to you: "NO". Because that statement reeks of: "They have been doing it this way for decades, so you do the same!"

Well, they have not been doing it this way forever. The part management in Orcad and Altium (and probably other high-end packages) has been born out of necessity (and went through many improvement cycles). Not because it is something they have thought about long and hard when they begun writing their software. My statement is all about not making the same mistake again.

In my experience people often want to create new software because they don't want to learn how an existing software package works and/or think they can come up with something better. But in reality those people often underestimate or don't understand the complexity. At one of my former employers the R&D manager wanted to have a project management tool. These are a dime an dozen but he insisted we should develop one ourselves because he didn't like (=understood) the existing packages. So we ended up with a very clever intern (no bad word about this person!) who wrote a web-application. When it was nearly finished it turned out all kind of features where missing so the intern got hired to finish it. In the end we got a project management tool which was just as complicated to use as an off-the-shelve solution but only paid more  Meanwhile we could have used the time the intern spend on the project  management on product development.

The moral: be really sure you can improve a 'perfect wheel' before creating something which already exists.

[nctnico]

Not even remotely important in a professional tool. In Orcad I draw schematics in Windows and do PCB layout in Linux (this is a >$5k package without the very advanced options).

You must not be doing a lot of crossprobing then .... ( click in pcb and have schematic zoom to part and vice versa ... )

It is VERY important to have the schematic and pcb talk to each other in realtime. only transferring netlists and pushing back eco's using simple files went away 20 years ago... There is not enough data in the file exchanges.
Protel 95 could already crossprobe.

I use that heavily.

I don't use cross probing at all. However when PCB editor and Capture both run on Windows Orcad can do it as well.