I wonder would someone program a CPU that only allows a limited number of nested function call?
Depends on the application, and more importantly, how it handles the "overflow". Obviously you'd need to be pretty careful never to reach the limit. But if said CPU would (for instance) generate an exception in case of "return stack overflow", then why not. Now if there was no exception system and no means of knowing it happened, then it's a lot more questionable. In any case, for any moderately complex software, you need to ensure from static analysis that the max call depth never reaches the maximum.
Are there examples of real CPUs with this "feature" (let's call it "feature") implemented in hardware?
Yes of course, as I mentioned in another thread, some PIC MCUs, for instance, do have a hardware return stack. The smallest being 8-entry deep AFAIR. And they didn't have any means of knowing in case of overflow, although I may have forgotten. I have programmed such MCUs and they are usable, but you need to be very careful. Of course software for those MCUs tends to be relatively simple, and written either in assembly or in C, but with proper care.
If you think about it, call depth is ALWAYS limited, even with a conventional stack system, given that the stack itself in this case is itself limited. Of course, unless you use a lot of local data, even with relatively small stacks, the max call depth is usually a lot more than 8, or even 32.
Anyway, that's linked to the talk about dedicated return stacks, and I find the idea interesting (as already said), but I certainly would much prefer return stacks to be configurable rather than fixed.
Although such limits can be annoying, they can also be useful to track issues with your code, especially if the actual call depth is higher than what you expected. But as I said, I personally think that would be more useful yet if that can be configured depending on the application.
To get an idea of the max call depth in your programs, you can instrument the code to figure it out. That can be an interesting experience.