It irritates me that he has my prototype and can take it to another firm who can understand exactly what I have done.
Wait... he hasn't paid you
at all, and there is no
written contract? Everything was verbal, and he's relying on that to not pay you?
He's stolen your prototype, and you should treat it as theft unless and until he pays for it. It is
yours and there is no way he can prove otherwise. Tomorrow, you must write to him (don't phone or talk, you want this on record) and ask for the prototype back. Mention that it if is not returned intact by the end of the week you will treat it as being sold to him for £x. That is
not agreeing to his PO, it is selling a single item to him. Remind him that the IP is entirely yours and he should not divulge it to any third party on penalty of legal action being taken against him for recovery of any loss caused by same.
Then next week (because he will ignore this or call your bluff), send him a letter before action for the £x you threatened to charge him, and state that if he doesn't cough within 2 weeks you will take action against him. Then, because he will ignore that, you toddle off to the Money Claim Online webby and sue him (this assumes the amount is within the small claims range).
Don't get into a conversation or argument with him about anything - if he wants to settle in some way they consider his proposal, but don't get into explaining or arguing or anything like that because he will walk all over you. Arms length, strictly only as needed is what you want here.
Importantly, don't waver once you have started down this path, because he will call your bluff. You are in the right unless there is paperwork to say differently, and the worst case is you lose £40 should your claim fail. It
has to be worth that just to put the shits up him
Edit: Goes without saying, other than what Lewis did above, but: I am not a lawyer, and you shouldn't take advice from an internet forum without checking it out first.