Which version of RaspBMC are you using? I have Rpi(s) at my student home, and one at parent's house.
Student home has NFS server for all content, on average takes 2 seconds to open Movies page with ~200 movies and 3-4 seconds to start playing content. You can turn down fanart and iconart resolution to speed it up a bit more, that is with 720p fanart (1080p GUI) and 512p image art.
At parent's house it's a little slower because the USB driver is a bit more aggressive in putting the drive to sleep. So if you haven't used it for a while, there's about 5 seconds extra delay waiting for drive to spin up.
I'd say overall the GUI is usable with minor lag, given the cost of hardware it's pretty good. I'm awaiting XBMC v13, which is a lot faster, this video is XBMC v13 with Amber skin:
Built in CEC hardware is great, tested and works fine on Panasonic, Sharp and Samsung TVs out of the box. Android/iOS support is neat, obviously not unique to Rpi but somewhat overcomes parts of the GUI which lag in places.
A key element of a complete Raspbmc install IMO is a fast, 8GB minimum SD card. I've been using 30MB/s cards from SanDisk. The SD card interface maxes out around 24MB/s on this card.
Complaints: minor lag now and then, fastfoward/rewind is laggy on streams with lots of network data, max 45Mbit/s network (would prefer dedicated USB bus for ethernet chipset.)
Also, if you want to get content over the network, avoid wifi where possible. It's just too laggy except for the lowest bitrate content, and there will be more contention with other hosts. Every room in the student house is networked directly to with 100Mbit/s minimum, so this works fine. I can play 40Mbit/s bluray rip of Life of Pi, Avatar, Iron Man 3, etc. with absolutely no lag or buffering.
For it's price though, I'm hardly complaining. It's an amazing bit of kit, given I can basically throw RaspBMC on to an SD card, stick it into a Pi and tell it where to find the content, and it's done. Minor tweaks can be done for better performance later.