Good news about the sound quality improving drastically.
As you can hear there is a lot of good audio information stored in the groove of the record. If you happen to have 78 RPM records the 78 side is useful but most records that are still lying around are more likely to be LPs ("Long Playing" 33 1/3 RPMs). As you noticed the tip of the cantilever (the stylus) and the cantilever itself change position relative to the record when you switch from one side of the needle to the other. Records were (are) made with the anticipation of a particular stylus and cantilever geometry. In addition to getting the right cartridge loading on the input, getting the geometry right is a key step in accurately retrieving the information in the grooves. The geometry generally includes getting the tonearm parallel (or nearly parallel) to the record and at the right height above the record so that the cartridge when mounted in the headshell puts the cantilever and stylus at the proper angles relative to the record. Additionally more "modern"/advanced tonearms will include a setting for the tracking force (in grams high enough to keep the stylus properly engaged in the record groove so as to properly retrieve the information in the grooves but not so high as to wear out the record, the cantilever, or the stylus). Further, tonearms often have an anti-skating mechanism that counter-acts the tendency of the record groove (one long spiral) from pulling the record too much to the inside of the groove; the proper anti-skate setting will keep the stylus evenly balanced between the inside and outside of the groove. These are not 100% precise settings but they can have an audible impact on the sound (as well as the long term wear). The reason it's important to get this geometry set correctly is that in stereo records each side of the groove represents one channel (left or right) of the original recording - which probably had two microphones placed to the right and left of the center of the stage and musicians being recorded.
The reason for noting this is that you can continue to make some improvements with your cabling and input and also the gain you get (perhaps by changing the potentiometer) but if you want to continue further down the path of "drastically improving the sound quality" a next step might include getting two speakers. With just the one speaker built into the record player a listener hears mono rather than stereo. With a stereo system (including two speakers on a plane apart from one another) you can better recreate the sense of left to right sound imaging that was captured when a stereo recording was made. Whether this is important to a listener is a personal decision but in general the purpose of "hifi" was to reproduce sound in a manner that had "high fidelity" to the original sound being recorded. With two speakers you will hear more of what the audience would have heard had they been present for a symphony or orchestra, or rock and roll band, or jazz club performance, etc.
The sound reproduction process can go further as once you get the mechanical energy accurately captured in the record groove and convert it to a magnetic and then electrical signal and then amplify it enough to drive the speakers, the speakers then convert the electrical energy back to mechanical energy as the speakers generate sound waves. What you will find next is that the system extends beyond the speakers to the enclosure (ie, the room) in which the speakers reside. As the sound waves travel out from the speakers they will begin to bounce off the floor, the ceiling, the sidewalls, the wall behind the listener, the wall behind the speakers, and anything else in the room. Depending on the room and it's furnishings some of the frequencies will be overly emphasized (reflected) and some will be deemphasized (absorbed). Altogether his will effectively act as a physical tone control that impacts the sound you hear well beyond the electronic tone control(s) of the system. By moving the speakers relative to the walls you can change the sound to some extent. Not only will moving the speakers relative to the room impact the frequencies emphasized or de-emphasized but you will find that in some rooms if you place the speakers at a distance apart from one another that is comparable to distance from the speaker plane to where you sit facing the speakers the sound will begin to "image" with more focus, and not only left to right but also front to back. In this manner you can begin to recreate not only left to right spatial imaging but also front to back depth imaging of the audio signal. With some recordings you will be able to hear the relative placement of the instruments and/or vocalists on the stage both side to side and front to back. This doesn't necessarily require expensive stereo equipment but it would ideally include a tonearm and cartridge setup with good geometry and two speakers for stereo. (You will find some recordings were much better made than others.)
Net, net: you might try a new potentiometer to smooth out the gain control but after that the next step might be to consider two speakers for stereo. Enjoy the journey!
PS, somebody here will say skip the records and go for CDs but my recommendation would be to stick with the turntable and records until you get the stereo speakers and the room placement dialed-in and then simply add a CD player which can be connected to an aux input. Then you can find a copy of the same music recorded in a record format and a CD format and determine if you hear any differences and if you have any preferences for one or the other.
PSS, somebody else will say skip the entire system including the CD and just listen to MP3's or AAC's with earbuds on an i-something. YMMV
EF