Let's say you want to optimize a manufacturing process to minimize defect, by doing an experiment with different settings of a certain variable. Suppose for each setting, you product 100 parts, and make note of the defect rate. Let's say your experiment is composed of 10 different settings, and after all it's done, you make use of the setting that gives the least defect. But is there a rigorous statistical test to truly tell whether the outcome didn't happen due to chance?
And here's a related scenario. Suppose you want to optimize the current process. Based on the history of the previous 10000 parts made, you know the defect rate is x%. You use a new process, but due to expense, you only make an experimental run with 500 parts, and you get a defect rate of y%. What statistical test can you use to determine if the new process truly had an effect?
Thanks