Have you tried resetting the fuse settings to default and running at 5V? Then try 5V with modified fuse settings, then 3.3V. Basically, start with the only difference between your setup and the UNO being the chip is off the board. Then change one thing at a time and see what specifically causes the problem.
I think the 3.3V and/or noise may be an issue.
First,
the 3.3V - I am using 5V and I have a couple of LCD 20x4. I found that if I loaded the board down and pull the power below about 4.8V, my LCD and other devices go nuts frequently. I am wondering if you have devices not being happy at 3.3V.
Second, noise. I did some extensive tests back around Christmas time after making a DS1307 clock. I was trying to minimize noise impact on my devices. I have 3 I2C devices (not including the MCU) and found the I2C rather works reliable when noise is low.
My test was to write than read back the 56 bytes of nvram on the DS1307. The noise I was trying to reduce was when the DS1307 has squarewave turned on.
When noise was very low (no square wave), I was able to do 11 million read-after-write with 3 errors (total) - I encounter my error after 10million, so 11 million became the bar for the second run. With the two tests that took hours and hours to finish, the average was 3 errors in 11 million.
When noise is high (32MHz square wave on, square wave line running close to I2C lines), I would hit error error within a minute or two. With 8MHz wave on, I would hit errors in minutes.
I swapped devices (I have a "real" Arduino, and a home made), swapped I2C clock, and swapped I2C LCD. When noise is low (and power is good), things run well.
I was trying to get some idea about how to reduce noise, so I posted it back then and had a discussion with fellow forum members. That post may give you some ideas.
https://www.eevblog.com/forum/beginners/sloppy-hardware-work-or-to-be-expected/