As well a lot of ISP's do tend to bandwidth limit audio and video, so that you can buffer a few seconds to a minute at a time, and then slowly get the rest. Basically done to manage bursts of data in the aggregrate on the network, and often as well with larger sites like YT and other CDN's there are local cache servers to provide the actual data, instead of having a central server. thus the cache server gets the request, looks to see if it is in cache and then trickles it out, or goes upstream to get it, grabs a minute or so of it, then has the time to trickle it down to you, while getting the rest to cache.
Simple test is to have a encrypted and non encrypted file, and see that the non encrypted is going to be rate limited, while the encrypted file, as a big binary blob, is not going to be rate limited, but will come down at the maximum speed of the slowest point in the system, most likely the oversubscribed local link to the connection to you, closely followed by the minimum viable product ISP provided connection equipment if you are using it. Store both files as random name, and in the same directory on the same server.
Another thing is that most ISP's will deliberately prioritise data to the most common speed test sites, so that they will show an inflated number, as the traffic will be priority over all other users briefly, giving a false impression of fast speed. All others will be throttled, just how badly is a combination of rate limiting both locally and at peering points along the way where your traffic is passed from one network to another.