Using Altium with local libraries and github

[trevwhite]

I must admit that over the last few years I have become totally confused about how libraries are managed in Altium. It seems they want to merge them into the cloud and discourage local libraries. I am feeling that I want to use Altium without connecting to their server. I want to manage my own libraries locally and use Github to store them as well as all my projects. Is this practical with AD24? Any one got a decent tutorial on how to mange libraries locally and not use Altium with the cloud?

I am gearing up I feel to not renewing my subscription this year and want to spend the next few months seeing if that is going to be practical. I think it will also be a good excuse to properly organise my component libraries.


Thanks






 

[PlainName]

I'm on AD22 and unless they've massively changed things it's perfectly feasible to go your own way locally. But they are a bit like Microsoft and try to push you in their direction.

I set up a local DBLib library using sqlite (which needed the DevArt ODBC drivers to allow AD to see database  updates without having to restart). In Preferences | Data Management | File-based Libraries is where you add them and specify the local location.

If you don't want to touch the cloud at all, Preferences | System | Network Activity is where you can disable all that. I found disabling the Altium Cloud and Data Management Server options did it for me. The company's cloud name appears in the components list but there is nothing under it so easy to ignore.

[trevwhite]

What’s the benefit of a DBlib based library?

[PlainName]

Oh  boy

Good description here: https://www.eevblog.com/forum/altium/component-library-management/msg604147/#msg604147

It's not mentioned there, but if you have separate PCBLib and SCHLib for each component class then associating weirdo footprints with any schematic symbol is just a table cell entry. Creating a new component is just a matter of filling in a row of the table.

Although he talks about Excel there, a pukka database is better. AD knows of Access, but... well, it's Microsoft. Much better is sqlite via ODBC, and then the world is your (mostly free) oyster.

[trevwhite]

Thanks, this seems very powerful.

[dnparadice]

With up to AD22 I find that you can easily use github to manage library repos. You can even use Altium to do commits merges, pulls etc. The trick is to make an empty (except for the library) Altium project for each of your libraries and check the Altium project into the git repo. You can then interact with these libraries like they are local libraries because they are local libraries managed by git. IMO you don't need to mess with databases unless you need to manage multiple people working on design projects (or you have some spare time on your hands and like databases )

[trevwhite]

Thanks for this suggestion. It sounds the kind of thing I am looking for. I have a main PC with Altium on but on occasion I will use a laptop so I need a certain level of portability that I think the database system will not allow.


But if I was to have a project that just included a series of SCH and PCB library files then you are saying I can edit as required within that project and use Github to save/track changes.
 
To use these libraries in a new project do you link them in each time? You never install a library, you just add the SCH and PCB files to the new project each time?


Trev




Quote from: dnparadice on Yesterday at 07:10:31 am

With up to AD22 I find that you can easily use github to manage library repos. You can even use Altium to do commits merges, pulls etc. The trick is to make an empty (except for the library) Altium project for each of your libraries and check the Altium project into the git repo. You can then interact with these libraries like they are local libraries because they are local libraries managed by git. IMO you don't need to mess with databases unless you need to manage multiple people working on design projects (or you have some spare time on your hands and like databases )

[ajb]

Good description here: https://www.eevblog.com/forum/altium/component-library-management/msg604147/#msg604147

Hey that looks familiar    One specific revision to my workflow (or "our" workflow, since it's not just me anymore) since writing that: multiple tables for different part types was kind of a pain to manage, so everything is in one table now, and component types are a top-level category field.  Trying to transcribe a lot of key parameters into a database table is a waste of time IMO, so headline specs go into the part description, and for anything else there's a link to the datasheet (using the Altium db schema, you can add links to a right-click menu from the database, so this is super convenient). 

IMO you don't need to mess with databases unless you need to manage multiple people working on design projects (or you have some spare time on your hands and like databases )

Hard disagree, the main benefit of a database is in the ease of symbol and footprint reuse across multiple parts, and general ease of data management.  It's much easier to deal with design and manufacturing data across a whole library when all of that is in a single place, as opposed to spread across hundreds or thousands of individual library files. 


The downside of a dblib in the context of this thread is that it's not straightforward to manage changes to library files in concert with changes to the database.  We have all of our library files in a git repo (no need for an altium project, you can just have them all in a folder somewhere and use an external git tool to handle changes), but putting the backing database into the repo isn't workable.  It might work for a single user with a local Access or Excel file, but even then, Altium's ODBC connection probably plays havoc with updating the file via git, plus it's a binary file, which is less amenable to git anyway.  I'm not sure how to solve that.  I played with exporting the database to CSV, so that you could at least see changes to the database in the git history, but it adds a manual step to each change and it's a pain to get the formatting to be properly human-readable as a diff.  Some way of directly using a plaintext file as the backing database would be nice.

One of my goals for the new year is to move to a SQL database that directly synchronizes with our MRP, and at that point it may make sense to manage changes to the library files via that database rather than via git.  Haven't gotten very far with that idea yet, though.

Thanks for this suggestion. It sounds the kind of thing I am looking for. I have a main PC with Altium on but on occasion I will use a laptop so I need a certain level of portability that I think the database system will not allow.

You can make that work with a dblib.  I have for many years using an Access database.  As long as you have a way of keeping that file in sync between the two machines (offline-capable network drive, dropbox, etc) it's not a problem.  Doing that with multiple users is where it gets tricky.  At that point a VPN back to a network drive (or database server) at the office is probably the right move for remote work.  Although remember: you only need access to the database/library when adding or updating components to the design.  Once a component is in the design, its footprint and symbol data is cached in the project files, and you can do all of the design, layout, etc without touching the libraries at all.  I think the only other exception is if you're using database parameters in outjobs, you would need to be able to reach the database to generate those outputs -- but that's pretty niche, all of the component parameters (that are visible in the component properties) will be accessible to the outjob.

To use these libraries in a new project do you link them in each time? You never install a library, you just add the SCH and PCB files to the new project each time?

If the library files are in a library project (whether it's a dblib or intlib or whatever), you just need to have that library installed.  No need to add specific schlib or pcblib files to the project, the system handles all of that via the main library project/file.

[lex]

I played with exporting the database to CSV, so that you could at least see changes to the database in the git history, but it adds a manual step to each change and it's a pain to get the formatting to be properly human-readable as a diff.  Some way of directly using a plaintext file as the backing database would be nice.

I have been working for years following an opposite approach :-) My teammates manually maintain CSV files under git, and everyone locally makes his "database".

The details are at https://github.com/armusin/parts

[thekeystoneclimber]

I'm on AD22 and unless they've massively changed things it's perfectly feasible to go your own way locally. But they are a bit like Microsoft and try to push you in their direction.

I set up a local DBLib library using sqlite (which needed the DevArt ODBC drivers to allow AD to see database  updates without having to restart). In Preferences | Data Management | File-based Libraries is where you add them and specify the local location.

If you don't want to touch the cloud at all, Preferences | System | Network Activity is where you can disable all that. I found disabling the Altium Cloud and Data Management Server options did it for me. The company's cloud name appears in the components list but there is nothing under it so easy to ignore.

I did the same thing only using the ch-werner free open source SQLite ODBC driver.  Everything is working perfectly inside Altium with respect to being able to see changes to the database and placing components.  The problem I'm having a hard time solving is finding software that allows me to edit and write changes to the database over ODBC so it doesn't complain about the database being locked while Altium is open.

What do you use to edit your SQLite database?

[PlainName]

Generally I use DB Browser: https://sqlitebrowser.org/

Can't recall an issue with Altium holding the database open when trying to edit it, but Altium only ever views it and never makes changes. Perhaps the ODBC driver plays a big part.

[free_electron]

Here's how i do that :

Stuff you need and is free :
- github
- microsoft odbc driver (can be downloaded for free. you don't need an account or a office license )
- modern CSV (free version)

- Make a github project
- in the project folder make 3 subdirectories : schlibs , pcblibs , csv . throw all your schlibs in ./schlibs and pcblibs in ./pcblibs
- make a csv file with the columns you want. one column must be called ID and you need to have one column called "Library Ref". The rest can be whatever.
- it is an absolute must that anything that goes in the ID column has to be UNIQUE ! this is the library index.

- Create an odbc connection for "text mode" and point it to the CSV folder. Do not create this as a per-file connection but set it as "folder". That way you can have files like diodes.csv, transistors.csv,ic.csv and so on. Every file becomes a "table". save the connection and give it a name you can remember
- create a dblib and point it to the odbc connection

done.

Altium interacts with the odbc driver and doesn't know in reality this is a set of CSV files.
Adding parts is a simple as adding lines to the csv files. Simply hit refresh in Altium and the parts are immediately there.
You can use a program like "Modern CSV" to edit

I have a master Excel document with multiple sheets and a macro to automatically export CSV from the tables.
Adding parts is simply adding lines in excel.

Why not connect directly to excel ?
Excel files are a binary format that is unreadable. CSV is pure ascii text in an encoded format that is a standard. CSV plays nice with github and you can do diffs and versioning.

What's the advantage ?
- text format files that can be versioned in github
- human readable
- doesn't cost anything , even the odbc driver is free
- doesn't need any infrastructure like a database server
- doesn't need a closed cloud system
- versioning and distributed access is handled via github.
- installation on a new computer is simple : install altium, clone github repository , add dblib file to altium . Everything is contained in the github repo, including the dblib file, csv, schlibs, dblibs. push/pull . if mulitple people edited github can show and merge for you since they are simple text files.

There is one rule : one schlib = one symbol and one pcblib is one footprint. Those are binary files and github can't diff , but it can add files.
Use altiums library splitter of you have an existing library to make individual files.