You can try to find a cheap smartphone that's supported by LineageOS, but the developers don't even seem to bother with the super cheap ones nowadays - a $50 phone takes about the same time to hack as a $10 phone but ends up with much better performance. I'm not aware of any cheaper way to get a cellular radio that supports modern networks - 2G is basically dead and 3G is being phased out.
If you don't need cellular, Raspberry Pi Zero is hard to beat for the low price.