EEVblog Electronics Community Forum

Electronics => PCB/EDA/CAD => KiCad => Topic started by: homebrew on May 12, 2017, 06:19:33 am

Title: Pulling all Github repositories at once
Post by: homebrew on May 12, 2017, 06:19:33 am

In case anybody would ever have the need to pull ALL KiCad repositories and be sure to catch all, here's how:

Code: [Select]

$ curl -s https://api.github.com/orgs/KiCad/repos?per_page=1000  | grep "svn_url" | grep ".pretty" | cut -d " " -f 6 | sed -e "s/\"\(.*\)\".*/\1/g" | sort | while read line; do git clone $line; done

This clones all available .pretty git repositories.

I prefer this approach over the premade lists that circulate around as this is always up to date ...

Modify: Corrected the script as it was mangled by the blog. Added support for direct cloning

Title: Re: Pulling all Github repositories at once
Post by: donotdespisethesnake on May 13, 2017, 10:13:36 am

You can ask for 1000, but github only returns max of 100 results, you have to get multiple pages.  As there are already over 100 repos, your one liner will miss some.

Another way is to use https://github.com/hairymnstr/kicad-getlibs, which also updates the library table for you.

Of course, if you just want a local copy of the footprints without a git clone, you can download them using KiCad.

Title: Re: Pulling all Github repositories at once
Post by: NivagSwerdna on May 13, 2017, 10:25:57 am

If you don't need history and only a single branch then do a shallow copy

git clone --depth 1 https://path/to/repo/foo.git -b bar

Title: Re: Pulling all Github repositories at once
Post by: homebrew on May 14, 2017, 12:47:00 pm

Quote from: donotdespisethesnake on May 13, 2017, 10:13:36 am

You can ask for 1000, but github only returns max of 100 results, you have to get multiple pages.  As there are already over 100 repos, your one liner will miss some.

Ah,  :palm: stupid me! Didn't check ...

So here's the updated one-liner:

Code: [Select]

$ lastpage=$(curl -s -I https://api.github.com/orgs/KiCad/repos  | grep "Link:" | sed -e "s/.*page=\(.*\)>; rel=\"last\".*/\1/g"); for ((page=1;page<=$lastpage;page++)); do curl -s https://api.github.com/orgs/KiCad/repos?page=$page  | grep "svn_url" | grep ".pretty" | cut -d " " -f 6 | sed -e "s/\"\(.*\)\".*/\1/g"; done | sort | while read line; do git clone --depth 1 $line; done

Or in a more readable form to put in a script file:

Code: [Select]

#!/bin/bash

lastpage=$(\
        curl -s -I https://api.github.com/orgs/KiCad/repos  \
        | grep "Link:" \
        | sed -e "s/.*page=\(.*\)>; rel=\"last\".*/\1/g"\
        )
for ((page=1;page<=$lastpage;page++));
do
        curl -s https://api.github.com/orgs/KiCad/repos?page=$page  \
                | grep "svn_url" \
                | grep ".pretty" \
                | cut -d " " -f 6 \
                | sed -e "s/\"\(.*\)\".*/\1/g";
done | sort | while read line;
do
        git clone --depth 1 $line;
done