Git, weird issue

[DiTBho]

Code: [Select]

git clone https://github.com/...
Cloning
remote: Enumerating objects: 71526, done.
remote: Counting objects: 100% (3944/3944), done.
remote: Compressing objects: 100% (2518/2518), done.
error: RPC failed; result=18, HTTP code = 200iB | 1.01 MiB/s
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

Any idea? 

[DiTBho]

Found this, going to investigate and try it.

[golden_labels]

Seems like not the git client issue, but your connection to GitHub. At any stage: from GitHub itself misbehaving, through something inbetween interferring with the connection, ending with overzealous antivirus/firewall software.

I know this doesn’t solve your problem, but hopefully it will save your time by cutting down possibilities.

[magic]

Found this, going to investigate and try it.

It quite possibly has nothing to do with your issue.

When will people learn to always look at the first thing which went wrong?

edit
The connection is over HTTPS so I doubt it's antivirus or firewall.
The juxtaposition of "error" and "HTTP code 200" looks weird. I don't know the git protocol, but it probably means some error reported by the server, which may or may not be triggered by some avoidable behavior of the client.

[Nominal Animal]

One possibility is RST packet injection by an internet service provider in between.  It has been used variously in Europe, mostly to combat file sharing (even when completely legal, to reduce bandwidth use), and the git HTTPS connections might look similar enough to trigger that.

Both the libcurl error code (18, CURLE_PARTIAL_FILE), and the early EOF, indicate the connection was broken mid-transfer.  I have personally only seen that due to network hiccups (connection restored in seconds to minutes), actual connection loss (server or network failure), and RST packet injection.

TCP RST packet injection is a simple tool for the ISP to break existing connections they deem suspicious or has transferred too much data.  Because github is sometimes used to store large data dumps (consider Wikileaks), some ISP's consider it (and places like sourceforge mirrors) "suspicious", worthy of breaking large/long-duration transfers, by injecting TCP RST packets.  Of course, they do try hard to avoid doing that to approved internet use, like viewing Youtube, or downloading content from a whitelist of pre-approved large content delivery networks.

[SiliconWizard]

There may be some starting points there: https://stackoverflow.com/questions/26197171/git-clone-repository-error-rpc-failed-result-56-http-code-200

[DiTBho]

One possibility is RST packet injection by an internet service provider in between.  It has been used variously in Europe, mostly to combat file sharing (even when completely legal, to reduce bandwidth use), and the git HTTPS connections might look similar enough to trigger that.

damn, that's precicely what happened 
after trying several approaches, including this setup

Code: [Select]

# core.compression
# An integer -1..9, indicating a default compression level. -1 is the zlib default.
# 0 means no compression, and 1..9 are various speed/size tradeoffs, 9 being slowest.
# If set, this provides a default to other compression variables
# such as core.looseCompression and pack.compression.

[core]
        # compression = 0
        # compression = 9
        compression = -1
        packedGitLimit = 512m
        packedGitWindowSize = 512m
[pack]
        deltaCacheSize = 2047m
        packSizeLimit = 2047m
        windowMemory = 2047m

which worked for a while, then randomly failed, always at different % progress, I moved the mac-mini/G4 from home to work, so same hardware, same kernel (GNU/Linux v4), same user space (Gentoo/PPC32BE-2020, server-profile/glibc), I repeated the experiment 10 times, and it worked perfectly!

X____X

[Nominal Animal]

Bandwidth-limiting your traffic can help, and you can download the initial tree as a ZIP file (since that can be restarted/continued even if the connection is RST'd) but really, the only recourse is to either get your ISP to stop, or switch ISPs.

Submitting a helpdesk/maintenance/support ticket along the lines of

I am a software developer.  When working with large repositories at GitHub.com, your systems consistently interrupt my secure connections by injecting TCP RST packets, breaking my connection, and hindering my work flow.  Can you stop doing that, at least for connections to github.com, githubassets.com, gitlab.com, and kernel.org, or do I need to switch to a better internet service provider that does not interfere with perfectly legal and appropriate secure TCP connections?  Or should we just start a movement to ban this practice as illegal, EU-wide?

might help, but considering today's world, you're more likely to get an answer suggesting you switch to using Facebook or Reddit for your software development work instead, these already having been pre-whitelisted.

Using a VPN won't help, because all TCP traffic, encrypted or not, can trivially be disrupted via TCP RST by your ISP (as it can simply set the RST header bit in one of the TCP packets it is conveying, dropping the data payload).  Unencrypted connections to port 80 are often exempted from RST-killing, because too many customers would complain about "internet not working".

[magic]

Both the libcurl error code (18, CURLE_PARTIAL_FILE), and the early EOF, indicate the connection was broken mid-transfer.

One learns something new every day

I would never guess that this code came from curl. They surely made it sound like it's some remote error 18, successfully delivered to the client over HTTP.

[golden_labels]

The connection is over HTTPS so I doubt it's antivirus or firewall.

AV/FW software packages happen to MitM encrypted connections by installing their own certificates. Even without that TLS only provides a secure channel for data: it does not protect metadata.

I would never guess that this code came from curl. They surely made it sound like it's some remote error 18, successfully delivered to the client over HTTP.

The clue here is the EOF part. It means the connection has been closed and that unexpected termination itself is the problem.

[Nominal Animal]

Yup.  The wonky error message is generated in remote-curl.c:run_slot().

If one does a web search for the error message, almost all of them misunderstand what is going on, even though it is pretty common (compared to other errors when using git).  I wonder if sending a request for comments for a patch changing remote-curl.c:run_slot() into something like

Code: [Select]

static int run_slot(struct active_request_slot *slot,
                    struct slot_results *results)
{
        int err;
        struct slot_results results_buf;

        if (!results)
                results = &results_buf;

        err = run_one_slot(slot, results);

        if (err == HTTP_OK || err == HTTP_REAUTH)
                return err;

        /*
         * RPC failed.  Generate an understandable error message
         * for the case where the connection was unexpectedly closed,
         * for example due to TCP RST packet injection.
        */
        if (err == HTTP_ERROR && results->http_code == 200 &&
                results->curl_result == CURLE_PARTIAL_FILE) {
                error(_("Connection closed unexpectedly; TCP RST injection?"));
                return err;
        }

        /*
         * Default error reporting for all other errors.
        */
        struct strbuf msg = STRBUF_INIT;
        if (results->http_code && results->http_code != 200)
                strbuf_addf(&msg, "HTTP %ld", results->http_code);
        if (results->curl_result != CURLE_OK) {
                if (msg.len)
                        strbuf_addch(&msg, ' ');
                strbuf_addf(&msg, "curl %d", results->curl_result);
                if (curl_errorstr[0]) {
                        strbuf_addch(&msg, ' ');
                        strbuf_addstr(&msg, curl_errorstr);
                }
        }
        error(_("RPC failed; %s"), msg.buf);
        strbuf_release(&msg);

        return err;
}would be useful for users, possibly adding more dedicated cases for specific other curl errors, specifically CURLE_RECV_ERROR (indicating a problem in the local network stack), and explaining how no local or configuration changes can help with this particular error (contrary to many, many reports one can find via web searches), because it is either generated by the server, or by a router in between.  It cannot happen due to just a sudden loss of connection, temporary or permanent, because it can only be due to receiving an explicit end-of-connection packet, TCP RST, which intermediate devices are supposed to not generate, but many ISPs use in an effort to combat file sharing, legal or otherwise.  (Hardware connection loss at any point will show up as a timeout-related error instead, with the transfer "hanging", which is easy to detect by a human.)

Typical HTTP errors are already handled separately in http.c:handle_curl_result() called by http.c:run_one_slot() (causing the above remote-curl.c:run_slot() see err = HTTP_ERROR, and printing the additional error message).  Putting simplified error reports into remote-curl.c:run_slot() means only the simplified error message is shown, adding the likelihood that human users understand the cause of the error right, and don't get confused by the additional error information (which is useless in this particular error condition, due to the very limited set of error causes).

I'm personally not mentally prepared to deal with the social stuff in pushing changes to other peoples' code right now, especially because bringing up the habit of some ISPs using TCP RST injection to "combat file sharing" will cause a nasty chain at the git@vger.kernel.org mailing list, where one should be subscribed before submitting patches for comments or consideration.  As you can see from my suggested ISP "support ticket" above, I'd just pour fuel to the flames, because this annoys me so much (having gone through this myself a decade ago, downloading Linux distro DVD images).  Hell, I can't even do my favourite hobbies, like drive-by bug fix submissions to open source projects right now, for the same reason.

[magic]

IMO the first good step would be indicating that it's a curl failure and maybe translating the code into human readable form.

"RPC failed, result=18, HTTP code=200" very much sounds like an RPC has been called and sucessfully returned failure code 18 with HTTP status 200. Then the server closed connection.

I don't know what those guys were smoking

[Nominal Animal]

IMO the first good step would be indicating that it's a curl failure and maybe translating the code into human readable form.

Easy to do, just modify the above remote-curl.c:run_slot() function.  Here's the original; it's been like this for quite a long time:

Code: [Select]

static int run_slot(struct active_request_slot *slot,
    struct slot_results *results)
{
int err;
struct slot_results results_buf;

if (!results)
results = &results_buf;

err = run_one_slot(slot, results);

if (err != HTTP_OK && err != HTTP_REAUTH) {
struct strbuf msg = STRBUF_INIT;
if (results->http_code && results->http_code != 200)
strbuf_addf(&msg, "HTTP %ld", results->http_code);
if (results->curl_result != CURLE_OK) {
if (msg.len)
strbuf_addch(&msg, ' ');
strbuf_addf(&msg, "curl %d", results->curl_result);
if (curl_errorstr[0]) {
strbuf_addch(&msg, ' ');
strbuf_addstr(&msg, curl_errorstr);
}
}
error(_("RPC failed; %s"), msg.buf);
strbuf_release(&msg);
}

return err;
}It uses a struct strbuf to construct the message, before logging it.  Very straightforward code, especially considering most HTTP status errors have already been reported (and arrive here with err = HTTP_ERROR).

In Git 2.0 (circa 2014), it was even simpler:

Code: [Select]

static int run_slot(struct active_request_slot *slot,
    struct slot_results *results)
{
int err;
struct slot_results results_buf;

if (!results)
results = &results_buf;

err = run_one_slot(slot, results);

if (err != HTTP_OK && err != HTTP_REAUTH) {
error("RPC failed; result=%d, HTTP code = %ld",
      results->curl_result, results->http_code);
}

return err;
}but still the same fundamentally indecipherable message, unless you already know about it.  The newer version, as you can see, does at least hint that the '18' error code is from curl.

So, if DiTBho were to update their git client to somewhat more recent version, the error message would already have been clearer about the source being libcurl.

I don't know what those guys were smoking

I know the history, so I can say they never thought about it, because to them the error is unique (no other code generates an "RPC failed" error) and easily parsed.  It takes someone outside the developer circle to note how uninformative (and, stupid) the error message is.  (And more social skills than I can apply right now to describe the situation when submitting such a change to them.)

It becomes even funnier when you realize that the gettext message identifier is "RPC failed; %s", meaning the actual message part cannot be translated at all.  (My suggested version is, of course.  Later versions of git use the common _(message) gettext macro for i18n.)