So I finally opened up my 5017 to test some caps and do a visual inspection. I ended up changing all the electrolytic on the analog board (tested bad) and some in the power supply.
I didn't really check the dimension of the replacement capacitors I ordered, so I had to install 2 of them on the side.
Did another comparison with my 34401a and the 10V LT1021 source. Integration time was 200ms (averaging disabled) on the 5017 and NPLC 10 on the 34401a.
Standard deviation
5017: 0.0000447757571
34401a: .0000172379238
Look like changing the bad caps helped a lot. I need to check the other measurement mode though. I think there still a problem with 4Wires resistance mode.