You just have to be very careful and target your assessments depending on where people are from and what industry they work in (i.e.: Collect as much information about them before you meet). For example, I work in an environment full of Government employees, most with Police or Military backgrounds and hold Masters degrees (some have a Doctorate). Without knowing those details, it would be nearly impossible to assess their intelligence or knowledge level because most of the time, they only reveal the bare minimum and a lot of things are kept secret. It would be really easy to make a false assessment and ultimately make yourself look foolish. I think if you met them in the street and didn't know them, you would just think they are some kind of middle-grade IT professional.
There are some rather sad statistics about the correlation between interview performance and job performance. Current studies make it look so bad that some think job interviews might disappear, perhaps being legally banned as useful for nothing more than allowing the interviewer to apply their prejudices. I find this very odd. In real life a lot of people can be quickly seen as far better or far worse than you might have expected from anything you could see before their interview. Most people who come across well enough in front of competent interviewers to be hired seem to perform somewhat close to expectations. Competent is the key word, here. Some people seem to consistently hire the worst of the candidates, but surely at least some well run studies should have tried to separate "do interviews work?" from "are there good and bad interviews/interviewers"?
As with everything, I don't think there is a "one size fits all" approach. In my experience on interview panels (both as an interviewer and as a candidate), the biggest issue is with those on the panel itself. It's common to find people who don't know how to conduct an interview, don't understand the position and the various requirements or ask completely irrelevant questions which don't have "right or wrong" responses.
If you're interviewing for a technical role, ask about the candidates prior experience, qualification and
specific technical questions. An interviewer shouldn't be asking meaningless questions like "If you were an animal, what animal would you be and why?", they not only make the interviewer appear inexperienced, but the question (and any response) is completely irrelevant, unless you're interviewing for an acting role dressed up in an animal costume.
The last interview I sat in years ago was great. The questions were mostly all technical before the discussion moved onto my personnel record. That was it. Straight to the point and relevant.