Low Cost PCB's Low Cost Components

Author Topic: Pulling all Github repositories at once  (Read 1111 times)

0 Members and 1 Guest are viewing this topic.

Offline homebrew

  • Regular Contributor
  • *
  • Posts: 204
  • Country: ch
Pulling all Github repositories at once
« on: May 12, 2017, 04:19:33 PM »
In case anybody would ever have the need to pull ALL KiCad repositories and be sure to catch all, here's how:
Code: [Select]
$ curl -s https://api.github.com/orgs/KiCad/repos?per_page=1000  | grep "svn_url" | grep ".pretty" | cut -d " " -f 6 | sed -e "s/\"\(.*\)\".*/\1/g" | sort | while read line; do git clone $line; done

This clones all available .pretty git repositories.

I prefer this approach over the premade lists that circulate around as this is always up to date ...

Modify: Corrected the script as it was mangled by the blog. Added support for direct cloning
« Last Edit: May 13, 2017, 03:19:39 PM by homebrew »
 

Offline donotdespisethesnake

  • Frequent Contributor
  • **
  • Posts: 458
  • Country: gb
  • Embedded stuff
Re: Pulling all Github repositories at once
« Reply #1 on: May 13, 2017, 08:13:36 PM »
You can ask for 1000, but github only returns max of 100 results, you have to get multiple pages.  As there are already over 100 repos, your one liner will miss some.

Another way is to use https://github.com/hairymnstr/kicad-getlibs, which also updates the library table for you.

Of course, if you just want a local copy of the footprints without a git clone, you can download them using KiCad.
Bob
 
The following users thanked this post: homebrew

Offline NivagSwerdna

  • Frequent Contributor
  • **
  • Posts: 858
  • Country: gb
Re: Pulling all Github repositories at once
« Reply #2 on: May 13, 2017, 08:25:57 PM »
If you don't need history and only a single branch then do a shallow copy

git clone --depth 1 https://path/to/repo/foo.git -b bar



 
The following users thanked this post: homebrew

Offline homebrew

  • Regular Contributor
  • *
  • Posts: 204
  • Country: ch
Re: Pulling all Github repositories at once
« Reply #3 on: May 14, 2017, 10:47:00 PM »
You can ask for 1000, but github only returns max of 100 results, you have to get multiple pages.  As there are already over 100 repos, your one liner will miss some.

Ah,  :palm: stupid me! Didn't check ...

So here's the updated one-liner:
Code: [Select]
$ lastpage=$(curl -s -I https://api.github.com/orgs/KiCad/repos  | grep "Link:" | sed -e "s/.*page=\(.*\)>; rel=\"last\".*/\1/g"); for ((page=1;page<=$lastpage;page++)); do curl -s https://api.github.com/orgs/KiCad/repos?page=$page  | grep "svn_url" | grep ".pretty" | cut -d " " -f 6 | sed -e "s/\"\(.*\)\".*/\1/g"; done | sort | while read line; do git clone --depth 1 $line; done
Or in a more readable form to put in a script file:
Code: [Select]
#!/bin/bash

lastpage=$(\
        curl -s -I https://api.github.com/orgs/KiCad/repos  \
        | grep "Link:" \
        | sed -e "s/.*page=\(.*\)>; rel=\"last\".*/\1/g"\
        )
for ((page=1;page<=$lastpage;page++));
do
        curl -s https://api.github.com/orgs/KiCad/repos?page=$page  \
                | grep "svn_url" \
                | grep ".pretty" \
                | cut -d " " -f 6 \
                | sed -e "s/\"\(.*\)\".*/\1/g";
done | sort | while read line;
do
        git clone --depth 1 $line;
done
« Last Edit: May 15, 2017, 12:39:03 AM by homebrew »
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf