This is my industry - I design and manufacture residential and commercial LED lighting. The stuff I make is on the high end (it's the only way to make it succeed building products in the USA). I also do a lot of one-off projects.
I completely agree that the existing solutions are complete band-aids. There aren't many applications less tailored to the nature of LED lighting than existing home lighting. What I mean is... you are starting with AC, LED's are DC. You usually have central fixtures, LED's produce less light so you need many of them, ideally spread around. The AC dimming solutions are sub-optimal (at best) for LED's. And the thermal problems are tricky - bulbs take heat fine, so one big light high up in a room is no problem. Not to mention the customer mindset is oriented towards bulbs... even when people are sold on the idea of LED's, they still think of light fixtures as being a single unit with a light source, a shade and a housing attached to the shade. LED's open up so many opportunities for lighting that it's hard to educate the customer on what is possible.
Having said that, I disagree that LED's could or would be something you install once and they could last for decades. From an electrical/performance standpoint, that's certainly possible. But LED technology is changing rapidly. CRI is getting better. Thermal performance is getting way better. Lumens/watt is getting way better. These advances open up new possibilities for the lighting designer, so things that were not possible 5 years ago can be possible now. And as someone mentioned above, aesthetics is at *least* 50% of the equation. If you look at a modern style home from the 70's, it looks so outdated today. An 80's modern home looks dated too - just in a different way. Same with a 90's modern home. Lighting is probably the single biggest contributor to style - because it has a style of design as well as a style of performance. In other words, you can really set the mood and style of a room depending on how you light it. But what the latest-and-greatest is in 1997 is radically different than what it is in 2017.
Someone above mentioned restrictiveness and rules... I personally feel it's highly restrictive, certainly compared to other markers I work in. For automotive, there aren't many rules (at least in the USA), and what rules there are operate on a self-certification process. So anyone can make anything for automotive use without onerous rules and testing requirements. But for lighting, your product *must* be UL listed. That costs a minimum of $10k for a single light fixture that uses off-the-shelf parts (off the shelf LED driver from Meanwell/GE/whoever) and an off-the-shelf light engine from Cree or similar. If you want to design your own driver, you're looking at about $20k additional costs for certification. And only that specific implementation is certified... if you take the same driver and light engine (which you already paid to be certified) and put them in a different light fixture (say, one made of bamboo and acrylic instead of steel and glass), you have to pay again to get that specific fixture certified. And these are the costs when you are using LED's and it falls under the low-voltage directive. It's more if you were making bulb lights! If you are not UL/CE/whatever certified, no shop will carry your products, no home inspector will sign off on an installation, no insurance company will cover you, no electrician will install them, etc. It's tantamount to a total blacklist from the industry if you aren't certified.
I type too fast for my own good - but the last thing I'd add is to answer the original question, the ideal solution would be if people were willing to pay more for lighting
. AC isn't actually too bad to work with, because it's easy to shift voltage with a transformer. If you started with DC, you'd want it at a high enough level to eliminate the need for boost converters... so would 48V even be sufficient? I am not sure it would be for someone who wanted to make a big light fixture with many LED's. But even starting at 48V, if you want to use a single high power LED, you're seriously limited in the number of buck converters that will accept 48V on the input. So cost is already a problem - may as well have just started with AC. On the control, DMX, 0-10v, DSI, DALI and custom solutions all have their limitations and none are really perfectly suited to home/commercial lighting. And now with WiFi and (more recently) BlueTooth and BLE, traditional dimming techniques are very much losing their monopoly on the market. Which is another reason I wouldn't want a fixed system that is intended to last for decades... because I'll wager than even in 10 years, residential LED lighting will be a lot different than it is today.