Controversies
Despite the fact that Fisher's test gives exact p-values, some authors have argued that it is conservative, i.e. that its actual rejection rate is below the nominal significance level. The apparent contradiction stems from the combination of a discrete statistic with fixed significance levels. To be more precise, consider the following proposal for a significance test at the 5%-level: reject the null hypothesis for each table to which Fisher's test assigns a p-value equal to or smaller than 5%. Because the set of all tables is discrete, there may not be a table for which equality is achieved. If is the largest p-value smaller than 5% which can actually occur for some table, then the proposed test effectively tests at the -level. For small sample sizes, might be significantly lower than 5%. While this effect occurs for any discrete statistic (not just in contingency tables, or for Fisher's test), it has been argued that the problem is compounded by the fact that Fisher's test conditions on the marginals. To avoid the problem, many authors discourage the use of fixed significance levels when dealing with discrete problems.
Another early discussion revolved around the necessity to condition on the marginals. Fisher's test gives exact p-values both for fixed and for random marginals. Other tests, most prominently Barnard's, require random marginals. Some authors (including, later, Barnard himself) have criticized Barnard's test based on this property. They argue that the marginal totals are an (almost) ancillary statistic, containing (almost) no information about the tested property.
Read more about this topic: Fisher's Exact Test