I've wondered if the IPA was dissolving the toothbrush bristles. I had an eBay PCB brush and that left a residue. I know some people use brushes with natural bristles but I'd prefer not to.
Mostly the residue is from dirty bristles when using purely IPA with toohbrush from my experiences, clean it thoroughly or just use new one. IPA wont dissolve nylon brush at all.
Try a drop of IPA on a clean glass surface, rub it with your brush, then blow it dry with you breath, observe the surface, if there is a haze or thin film formed when the IPA evaporated, then the brush is dirty. Assuming the IPA is not contaminated of course.
Also certain natural bristles like from animal hair, will leave residue when in contact with IPA, probably some chemical left in it when the manufacturer processed the hair.