EEVblog Electronics Community Forum

Electronics => Projects, Designs, and Technical Stuff => Topic started by: NivagSwerdna on January 25, 2019, 12:36:23 pm

Title: Keeping track of source code
Post by: NivagSwerdna on January 25, 2019, 12:36:23 pm
Just wondering how people control their source code... I have many small projects which end up getting distributed across many desktops/laptops over time...

For large projects I use a GIT repository but for the many small utilities or one-offs I have historically not created a repository for them (over the years this adds up to approx 100 projects in various places)...

Any hints and tips? 

Maybe a generic GIT Repo containing many small projects?

(And of course many of these projects pre-date GIT)


Title: Re: Keeping track of source code
Post by: tsman on January 25, 2019, 01:16:46 pm
I forced myself to always use a version control system for everything and now it is second nature for me. I migrated all the other version control systems (CVS, P4, Mercurial etc...) repos into git but I kept the old repos as sometimes operations don't have a git equivalent. Small projects just get shoved into a general repo and get moved out if they become important enough to warrant a separate repo.
Title: Re: Keeping track of source code
Post by: TomS_ on January 25, 2019, 01:53:16 pm
Quote from: NivagSwerdna on January 25, 2019, 12:36:23 pm
Maybe a generic GIT Repo containing many small projects?

This. Ive seen something similar done in one way or another through various companies I have worked at.

Get yourself a gitlab account, or perhaps github now that they offer free private repos, and store it all there so you can easily push/pull changes between different devices you work on. Saves the hassle of running your own git server.
Title: Re: Keeping track of source code
Post by: TiN on January 25, 2019, 02:28:38 pm
Mercurial repository for every project (running own server). Saves the trouble when i need on another machine "just that little 20 line code tool I wrote on sunday"...
Title: Re: Keeping track of source code
Post by: TomS_ on January 25, 2019, 08:22:29 pm
I have both a GitHub and GitLab account - originally GitHub but then realised they didnt do private repos for free, almost bought a subscription but then found GitLab.

I have all of my personal projects stored with GitLab. Removes the hassle of running your own server (unless that is your thing), and easy enough to access from anywhere you have an internet connection. If working from multiple devices, just commit and push before moving to the next where you can simply pull down and continue working.

Personally, I use SSH with a shared key to access my repos.

GitHub now offers free private repos, too.
Title: Re: Keeping track of source code
Post by: NorthGuy on January 25, 2019, 08:42:34 pm
I have local Git server.

I also have 100s of small projects which don't go to Git, although they are located in the same folder, and I have a hierarchy of them, so they look organized to me.

Title: Re: Keeping track of source code
Post by: Jeroen3 on January 25, 2019, 08:44:48 pm
Quote from: NivagSwerdna on January 25, 2019, 12:36:23 pm
... I have many small projects which end up getting distributed across many desktops/laptops over time...
(https://communityres.synology.com/images/picture/1280x1280/216/1538763132_k60AY.png)
Together with Cloud Station Drive keeps all my files in my private cloud.
It also runs git server.
Online backup to Synology C2, and your set.
Title: Re: Keeping track of source code
Post by: Doctorandus_P on January 26, 2019, 07:00:01 am
I'd go with Git and probably use a private server over SSH.
Any EUR 50 Linux capable board with an RJ-45 can be configured to do such a thing.
A SATA connector might be usefull, not all have them.
And think of some backup strategy, so there is never a single point of failure,

When Git is used properly, then any of the other clones can be used as a repositry, and you automatically already have a distributed backup system.

It seems to be advisory to use one GIT repository for each project, and the server seems to be the logical place to collect all respositories.
If the server fails it might be a lot of work to collect your 100+ projects again from all the distributed local copies, so backing up the server properly seems very wize. (Rotate a few USB sticks, or use rsync or git itself to backup to another server in another location via scripts.


Quote from: TomS_ on January 25, 2019, 08:22:29 pm
GitHub now offers free private repos, too.
I would not even consider using any M$ controlled service for "private stuff", but I may still use github to host some open source projects.

Quote from: NorthGuy
I also have 100s of small projects which don't go to Git, although they are located in the same folder, and I have a hierarchy of them, so they look organized to me.
With small projects an easy backup strategy is to simply compress them all together in a single file.
I have been using ISO8601 for years to add to filenames for backups. This way they are easy to sort by date.
https://en.wikipedia.org/wiki/ISO_8601 (https://en.wikipedia.org/wiki/ISO_8601)
ISO8601 is also the (only?) worlwide recognized date format without confusion in different locale's.

I still do not know whether to pre or post pend the ISO8601 date though. IE:
2018-01-26_project.zip
project_2018-01-26.zip




Title: Re: Keeping track of source code
Post by: bson on January 26, 2019, 07:26:38 am
I didn't know github has started allowing private repos with 3 or fewer collaborators for free accounts.  This saves me $5/mo now! :P

I use git submodules quite a bit for shared code.  Even if it doesn't fit your needs at the moment it's something that often comes in handy, and is good to be familiar with, so take a look at it!
Title: Re: Keeping track of source code
Post by: tsman on January 26, 2019, 01:34:57 pm
Quote from: bson on January 26, 2019, 07:26:38 am
I didn't know github has started allowing private repos with 3 or fewer collaborators for free accounts.  This saves me $5/mo now! :P
Yeah. They only announced the change a few weeks ago. (https://github.blog/2019-01-07-new-year-new-github/) Limited to 3 collaborators but that usually isn't a problem.
Title: Re: Keeping track of source code
Post by: jc101 on January 26, 2019, 04:12:19 pm
I've gone a bit hybrid, by using git with Dropbox.  I have a git folder at the root of my dropbox, which is on all machines I use.  For new projects I then create an empty repository in that folder from one of the machines...

Code: [Select]
git init --bare project-name.git

...then just push/pull etc. to the local dropbox files with dropbox keeping the folders in sync.  This is fine for me, as it's only me doing the development but using Macs and PCs interchangeably not necessarily with internet access if out and about.
Title: Re: Keeping track of source code
Post by: emece67 on January 26, 2019, 04:21:58 pm
.
Title: Re: Keeping track of source code
Post by: taydin on January 26, 2019, 04:23:35 pm
I would not recommend shoving everything into a single repo. You would want to tag some revisions of individual projects. if everything is in one monster repo, then all projects will receive all tags. Also, getting the log of the repo will give you the entire log. You would have to narrow down the log to that specific project.

The natural way is to use one repo for each project.

But if you have projects that are dependent on each other, or have common components, those can be put into the same repo so that when you tag revisions, those revision will be consistent with each other.
Title: Re: Keeping track of source code
Post by: ajb on January 28, 2019, 04:06:52 am
github for me, one project per repo.  I also use submodules a fair bit, which makes it fairly easy to control and reuse library code, and easily bounce updates to it between projects. 

Quote from: jc101 on January 26, 2019, 04:12:19 pm
I've gone a bit hybrid, by using git with Dropbox.  I have a git folder at the root of my dropbox, which is on all machines I use.  For new projects I then create an empty repository in that folder from one of the machines...

IME, git and dropbox/network drives don't get along very well, at least on Windows.  For a while I had my working directory in Dropbox so that I could seamlessly move between desktop and laptop as needed, but at various points git would have some sort of issue with the file synchronization and I'd have to delete and restore files to get it working again.  In the end I set up a separate working directory and just try to make sure I commit and push regularly--which is a good discipline to develop anyway. 

Title: Re: Keeping track of source code
Post by: james_s on January 28, 2019, 04:54:21 am
I've used a free program called Allway Sync to keep my project folder up to date between my desktop and laptop. I haven't tried adding more computers to the mix.
Title: Re: Keeping track of source code
Post by: NivagSwerdna on January 28, 2019, 01:58:27 pm
Thanks for the replies... it seems we are all in a similar boat.

I have a private git server (stash/bitbucket), directories sprinkled across by Synology NAS (mostly old projects) and some files that are shared across a few PCs using Synology CloudStation (this is now so confused after moving NAS devices that I have no idea where the latest copies are).  I also have a gitlab and github account where I collaborate or share publicly.

I think I will create a new GIT Uber project and use directories and start organising.  I don't think mutliple versions is much of a problem in GIT since branches are only pointers to commits.



Title: Re: Keeping track of source code
Post by: Bassman59 on January 28, 2019, 04:39:01 pm
Quote from: emece67 on January 26, 2019, 04:21:58 pm
SVN server running on a Synology NAS plus TortoiseSVN here.

Which Synology box are you using? I need to do some infrastructure upgrades at the house this year and a Synology NAS box seems like just the thing (especially since it can run applications like a Subversion server) but there are too many options!

FWIW I need to replace a Drobo which is connected to an Airport Extreme's USB port. It works but it's slow.
Title: Re: Keeping track of source code
Post by: Jeroen3 on January 28, 2019, 05:55:25 pm
I have a DS214 main server and DS410 cold storage.
Both are still supported!
Title: Re: Keeping track of source code
Post by: coppercone2 on January 28, 2019, 05:58:25 pm
Notepad.exe
Title: Re: Keeping track of source code
Post by: jc101 on January 28, 2019, 05:59:15 pm
Quote from: ajb on January 28, 2019, 04:06:52 am
github for me, one project per repo.  I also use submodules a fair bit, which makes it fairly easy to control and reuse library code, and easily bounce updates to it between projects. 

Quote from: jc101 on January 26, 2019, 04:12:19 pm
I've gone a bit hybrid, by using git with Dropbox.  I have a git folder at the root of my dropbox, which is on all machines I use.  For new projects I then create an empty repository in that folder from one of the machines...

IME, git and dropbox/network drives don't get along very well, at least on Windows.  For a while I had my working directory in Dropbox so that I could seamlessly move between desktop and laptop as needed, but at various points git would have some sort of issue with the file synchronization and I'd have to delete and restore files to get it working again.  In the end I set up a separate working directory and just try to make sure I commit and push regularly--which is a good discipline to develop anyway.

My working directories (the IDE environment etc.) isn't inside Dropbox, just the git repo I replicate to.  As it is very unlikely I will push an update, to the same repo, at the same time, from different machines it isn't a problem.  Actually running the programming environment in Dropbox does cause all sorts of interesting problems in itself, so I went with the hybrid solution.
Title: Re: Keeping track of source code
Post by: ajb on January 28, 2019, 06:38:16 pm
Quote from: jc101 on January 28, 2019, 05:59:15 pm
My working directories (the IDE environment etc.) isn't inside Dropbox, just the git repo I replicate to.  As it is very unlikely I will push an update, to the same repo, at the same time, from different machines it isn't a problem.  Actually running the programming environment in Dropbox does cause all sorts of interesting problems in itself, so I went with the hybrid solution.
The problem doesn't have anything to do with simultaneous pushes.  Just something about being in Dropbox or on a Windows shared drive has caused issues with git.  I don't know the exact mechanism, but just throwing it out there in case anyone else tries something similar.  When I looked into the problem it seemed to be a known thing.  It's possible that a repo you only ever push to may not manifest the same problems.  Since going back to a separate working directory git has been rock solid for me, as it should be.
Title: Re: Keeping track of source code
Post by: zitt on January 28, 2019, 10:38:45 pm
Quote from: emece67 on January 26, 2019, 04:21:58 pm
SVN server running on a Synology NAS plus TortoiseSVN here.

This is what I do.
I even store my Eagle files in a SVN repo.
I have the NAS backup the SVN repos to google drive on commit and then do a periodic sync to my backup service.

I'm forced to use GIT at work; just don't like it. Prefer SVN.
Title: Re: Keeping track of source code
Post by: bson on January 29, 2019, 04:35:12 am
I too have a Synology NAS (DS1515+, raid 5) and can't recommend them strongly enough.  It did have the boot issue due to some Intel Atom CE processor bug (I forget the details) where a boot config pin driver burns itself out over time - but Synology replaced it overnight.  They sent me a preconfigured replacement chassis with the same firmware version as mine, Fedex overnight, and all I had to do was move the drives and memory expansion over, and power it up (forget if I had to do anything else).  Then sent back the old unit in the same box, using the prepaid label they included.  If this one lasts 4-5 years like the previous one then I'll probably be ready for a replacement around that time anyway.

I back up all important data to pair of external USB drives; one drive is plugged in and the other kept with a friend who lives a few blocks away.  Every once in a while, like after I've added a bunch of photos and videos (of personal value) or something else important, I'll swap backup drives.  The builtin backup software handles it perfectly.  The bulk of stuff doesn't get backed up, especially things I can recreate as needed, like a significant assortment of toolchains and VM images for various uses.  Other than that I'm not in the habit of torrenting or downloading the Internet, so don't really have all that much.  I do keep installers and CD/DVD ISOs for more obscure or old versions of software.  (Like for example the software for my Imacon scanner, along with a Mac Mini G4 to run it on and a FW400-to-SCSI bridge, drivers, and a complete image of the MMG4's drive.)

I do sync a bunch of stuff to it - git repos, google drive, and so on, but I don't back this up.  I just keep it there to have a local copy, in case.  Whenever I order any parts I save the datasheets and 3D models to google drive (Mouser these days provides step models).  I really only back up what's impossible to redownload, recreate, reinstall, etc in worst case.  This data set is surprisingly small...
Title: Re: Keeping track of source code
Post by: Jeroen3 on January 29, 2019, 08:17:45 am
Quote from: bson on January 29, 2019, 04:35:12 am
I too have a Synology NAS (DS1515+, raid 5) and can't recommend them strongly enough.  It did have the boot issue due to some Intel Atom CE processor bug (I forget the details) where a boot config pin driver burns itself out over time - but Synology replaced it overnight.  They sent me a preconfigured replacement chassis with the same firmware version as mine, Fedex overnight, and all I had to do was move the drives and memory expansion over, and power it up (forget if I had to do anything else).  Then sent back the old unit in the same box, using the prepaid label they included.  If this one lasts 4-5 years like the previous one then I'll probably be ready for a replacement around that time anyway.
That is also the largest drawback of Synology. You can't really migrate the data to any other system. It must be synology. Yes, they do offer ubuntu image with recovery instructions, but you can't hot plug the volume to non-synology systems.
However, the time saved configuring and maintaining is worth it. Running your own server with some distribution with "hipsterfs"*, as often recommended on the forums, is only nice when it all works.

*zfs, btrfs
Title: Re: Keeping track of source code
Post by: emece67 on January 29, 2019, 09:05:06 am
.
Title: Re: Keeping track of source code
Post by: mrpackethead on January 29, 2019, 09:39:52 am
Git on AWS code commit. ( its free and private ).  every project has a repo.
Title: Re: Keeping track of source code
Post by: bd139 on January 29, 2019, 09:44:27 am
You don't need a remote server to run git or push to. Just keep one git repo per project in the project root. Even single file projects get a repo. It's so cheap and quick to do it. Makes experiments cheap to undo as well.
Title: Re: Keeping track of source code
Post by: mrpackethead on January 29, 2019, 09:53:42 am
remote server is handy when you are working across multiple locaitons with several machhiunes.
Title: Re: Keeping track of source code
Post by: bd139 on January 29, 2019, 10:00:53 am
I solve that by avoiding it.