This whole thread is fundamentally about intellectual property (IP) rights: the inherent right of an intellectual property owner to control how their property is used, viewed, monetized or otherwise exploited.
Whether it’s private property, the property of the US government or a private foundation funded by the US government makes no difference: an IP owner is entitled to make any rules they like surrounding the use of its IP.
If the usage policies happen to be overly ham-handed because they were made prior to the rise of Youtubers and Vlogs, that’s likely because things move fairly glacially at the institutional level. And given the potential that the use of governmental IP might also be used to convey a certain legitimacy for off-the-wall or outright offensive use, perhaps they feel that there is no screaming rush to redress the semi-pro YouTube usage situation.
If such policies happen to offend those who think that government institutions should not have any such IP rights, or should waive them for whatever reason, then that’s a political issue, and good luck with that: Politicians have bigger fish to fry, and nobody’s chances of getting re-elected will ever be dependent on facilitating the making of entertaining videos for the internet.
Re: enforcement, there’s not even any need to get all lawyered up or debate the “who sues who” thing: if somebody complains and gets some bureaucrat exercised enough to actually do something, a simple takedown notice to YouTube on government letterhead would likely condemn the thing to oblivion for eternity.
As for trying to suppress an IP claim in court, either it’s “public domain” and not protectable as IP, or it’s not, and prior case law for the jurisdiction would prevail. And while I’m not a DC lawyer, I simply can’t believe that there isn’t already plenty of precedents that allow the US government (and the Smithsonian specifically) to claim full rights to their IP as they have done via their commercial usage policies.
My point here is to support the notion of IP rights for all legal entities, something Youtubers should be holding near and dear, and not to necessarily justify the specific IP control policies of the Smithsonian. Again, those are political and administrative in nature, and are made by people ultimately hired by those who are voted in by the electorate. And people always get the government they deserve.