Yet I don't want my USB-charged device causing a fire and me getting sued for everything. Rock and a hard place.
==LEGAL==
Whether or not you get sued has zero bearing on your certifications. I can promise you there are plenty of ladder makers that have followed all the rules and they still get sued. That is what an LLC is for. Set up a company. It's quick and easy. And if someone sues the company, then you are insulated personally (as long as you are following certain rules--those are the things to get educated about).
==SAFETY==
The risk of fire from a 5V/1A wall wart that has UL cert is very small. Maybe you'd see something strange if you built 1M products. But if you build 100, the odds are on your side. If you want to convince yourself, get a 5V/1A wall wart and connect it to the 5 ohm 5W resistor. And then try to light tissue paper on fire with the resistor. You can't. And that's not even in a case! For the case, use metal case or flame retardant "drip proof" plastic. None of your electronic components (or PCB) will blink at being exposed to 250C. If you have a larger case, the problem is even easier (less watts per m2).
Do consider what might happen if the power supply fails. A PTC is cheap at the entry port of your product. It has some voltage drop, but if things get hot it will open and reset as they cool down. It's a great second line of defense.
== PCB ==
Layout out your PCB so that you have a solid ground plane on the bottom. Looks at standard shields from Digikey. If you are the paranoid type, include the ability to use a shield in your design if needed. if you have a solid ground plane on the bottom and the parts with clocks are under a shield, you will sail through EMC without any issues.
==EMISSIONS==
Buy yourself a spectrum analyzer. There are kits that let you "pre-cert" so that you know if you if you have problems before you get to the test lab. These include antennas for a variety of frequencies. But more important, the makers explain the steps you need to take. Even though you don't have the chamber, they help you understand how to work around that (the best answer is head out to the sticks where signals are down in the noise). If you can pre-cert yourself, you go into the compliance lab and sail through without any surprises. The Signal Hound folks have a lot of info here, and their USB Spectrum Analyzers (both low end and high end) are far superior to desktop units.
==SUSCEPTIBILITY==
Things like ESD and proper operation in the presence of interferers are important in Europe, but the US doesn't worry too much about it (FCC does for cell phones, but not for wifi). Use standard parts to ensure ESD is clamped at your input ports. For your own warranty concerns, ensure your product can take a zap on any port with an ESD gun.
== Reliability==
Get yourself a temperature chamber. Set it to 80 degrees (assuming you are making a product for consumers). Run your product continuously at that temperature. If you've missed a voltage spec on something, this will find it. Silicon (and passives) that are overstressed will fail very quickly (hours) at elevated temperatures. It depends on your product, but roughly for a product designed for room temp operation, running it at 80C for 3 days will simulate 6 months of continuous operation (depends on your precise product though). You need to study this in more detail, but the search terms are HTOL and arrenius.
== Mechanical==
Take 10 of your prototypes. Drop them over and over from a height of 1m in different orientations onto concrete. You will see some common failures. Be on the lookout for cracked parts on PCB (too much board flex, not enough support/screws/etc), solder joints that fail, etc. Chances are if the case is off the shelf and plastic, it is thick enough you won't have any problems. But do the drops UNTIL you see a failure. Your aim here is to find failures. If on 2-3 units you see the same failure after 2-3 drops, be very concerned. If you get to 20 drops on 10 units (200 drops total) from 1m without any failures, you should feel pretty good.
==US==
US will be fine just ensuring the FCC is happy.
==EU==
If you want a distributor in EU, you'll need EU compliance and as others have mentioned, there are a myriad of specs. Many of these you can simply attest and self-cert. For emissions, you'll need a test house to sign off (so request EU and US emissions at the same time). Find a product that is similar to your from a big company. They will specify what IEC specs they have passed, and what sections were relevant for that product. Do the same. Usually, this paperwork is called "declaration of conformity". If it's not in the owner's manual, it's often on their website.
If you just want to sell to the EU from the US, just put them up for sales and ship them when the order comes. Even without CE you'll see 99% of them make it through to the customer.
The above is just friendly advice. Verify it all yourself.