Floating inputs - you are likely picking up mains hum. Which is roughly sinusoidal, so it looks like a circle on the screen. Two perfect sinewaves would create a circle in XY mode. However, since there is noise and distortion on the mains hum, you get that distorted shape.
Basically you have stuck two antennas into the inputs of your scope.
If you want to build this curve tracer, you need to build it into a shielded enclosure and ground it, this cannot be built "free-form" and with no grounding at all. The three ground connections connect together, to the shielded (metal) enclosure - and the ground of your scope through the outer conductor of the coaxes you use to connect this to the scope inputs.