As tooki suggested, by far teh easiest and cheapest way to go aobut this - if you really want to make one yourself and not buy one for $10 - is to buy a "module" from amazon or ebay.
These often come on breakout boards making prototyping easier, and just require a microphone, speaker, and power supply, perhaps a couple of buttons. I assume your "USB adapter" is the CSR-SPI type made with a FT232R - that can indeed be used with csr8635 modules to change the configuration (which is stored on the module in EEPROM), the chip itself isn't programmable, they are generally ROM's.