I've made few pseudo-oscilloscopes based on STM32 (e.g.
http://tomeko.net/miniscope_v2f/ based on STM32F042 - device in TSSOP20 package) so I may have some answers and opinions.
I implemented USB on MCU as CDC device. From what I understand it is suitable for that purpose. The question is - since CDC device is really a virtual COM port - does it have Baud rate? I read somewhere that if it is USB CDC than Baud rate doesn't matter - it will be 12 Mbits/s for USB FS and numbers for baud rate will simply be ignored. Obviously just serial speed is not enough for oscilloscope, e.g. I have 7.2 MSPS which makes DSO bandwidth around 1 MHz, for real-time display I can't get away with UART speeds(though read more in question 4).
Baudrate does not matter. With original CDC example (USB-UART converter) it may matter as it may control physical UART baudrate (perhaps using separate endpoint) but there is no point in using this functionality.
When it comes to speed I prefer limiting it to what interface offers. ~900kB/s is not fast, but if data is transmitted in real time it can serve as signal recorder using PC HDD - function usually not available with bench oscilloscopes. It allows implementing trigger on PC side or just not worrying with trigger, recording all data and then going through it. I think this is kind of sweet spot - it can be very cheap (simple analog part, no big requirements for PCB) but offers something not available with bench scope. With high sampling speed everything gets complicated and off-the-shelf bench scope would be better choice.
This is so far the biggest question: Now I'm trying to fiddle around with PySerial library for interaction with usb, it works fine on low data rates but absolutely crashes for anything big(even when I send 10k sample points in one second). I just found out about PyUSB, but it is more complicated and I am not sure if it will work with CDC(COM Port), though I will try it soon. But the question is - what is the good way for interacting with USB device on 12 Mbit/s speed so it doesn't crashes or lags? Preferably Python, but I will consider any good option.
Personally I'm using libusb and it seems to be what PyUSB is using too. I don't know how python interface looks like but it shouldn't be complicated. One feature worth using to get close to 12Mbit/s is asynchronous transfer (usb_bulk_setup_async(), usb_submit_async(), etc.).
How to use Bulk or Isochronous transmissions? STM32 USB Library has just one Transmit function(e.g. CDC_Transmit_HS for CDC or USBD_LL_Transmit for HID) which doesn't state anywhere if it is Bulk or Isochronous, nor does it provide a way to choose. I read all the source code comments, docs and forums, haven't found anything at all.
USB CDC class device is with very few exceptions (*) using bulk transfer - this is standard. For isochronous you would probably have to start with sound card example but I would stay with bulk.
(*) AVR-based USB-UART converter
What is the best way to send waveform data to PC? I.e. I can send one sample at a time or wait for lets say 300 samples and send them. Which way is more appropriate? I suggest to send in in chunks of N samples, but then again - how many samples to collect? Obviously within RAM limits, but I suspect too many would not be good either.
For FS bulk transfer 19 full-size packets (transactions) / frame is the theoretical max. This means 19 x 64B each 1ms. It may be lower depending on device and firmware, but in general don't try sending other amount that N x 64B. In my case interrupt handler waits for SOF (Start Of Frame) and initiates transmission only if half of buffer is full (2kB or more). It also allows distinguish channels without any extra synchronization bytes (if anything is lost - this would be multiple of 64B and channel A would still be first in each 64B packet). No extra synchronization means that data can be also sent as binary (no bandwidth wasted for text and processing).
Is there way to test on PC if I receive data in Isochronous or Bulk mode?
It would not depend on PC but rather device (and appropriate driver).
There are some USB monitors, although from my observations they may have some stability issues with bigger traffic.