The reason for writing a text to explain the concept of statistical significance in clinical studies by a doctor is the finding at medical conferences, in scientific discussions, but also in statements by scientists who are members of evaluation committees and regulatory authorities that this concept does not is fully understood, as is the way in which it turns out that a drug is effective e.g. in the prevention of migraine. So one can read the following as a text by a doctor for doctors or for people trying to understand how clinical research and especially studies work in medicine today.

First of all, let's emphasize that probabilities and statistics are not easily accepted concepts by our brain. Things that seem self-evident or unimportant or important to us, the mathematics of probability and statistics PROVE to be just the opposite. An easy example: if we are asked in a family with four children what is the most likely combination of boys and girls (given that each child has 50% for one sex) most of us will answer: 2 boys and 2 girls. Error! The most likely combination is 3 children of one gender and 1 of the other (easily with pencil and paper you will understand the... fallacy).

On the other hand, a doctor today, of any specialty, cannot understand the results of a study she will read or that will be presented at a conference if she does not know the basic principles of probability mathematics and medical statistics. He cannot do any study, any PhD
thesis or publication without the basic knowledge of statistical analysis. Soon the train of evidence based medicine will depart and leave him alone to watch. As for the concept of probability, already in the 19th century the great doctor William Osler (1849-1919) had understood its importance and had said a lot
wisely that medicine is "the art of probability and the science of uncertainty."

Today the probability is perceived as a relative frequency which is the dominant interpretation in Medical clinical research. Probability is defined as the long-term marginal frequency of similar events in a set of events. So if we have a coin and we do flips we want to see in the long run in the number of flips how many times "crown" will appear and how many "letter". And we say long term because if
if we throw the coin 3 times we can have "Crown" all three times, while if we throw it 1000 times, it will be about 500 times "crown" and 500 "letter".

Before referring to statistical significance, a few words should be said about the studies with which a drug is approved, how they are structured so that researchers can then use the results to determine if there is statistical significance, that is, if a drug affects a disease. These studies are double-blind randomized studies. In double-blind and randomized studies, participants in a study are divided into two groups at random with a design so that the individual differences of the participants will be equally represented in both groups. This means "randomization". E.g. in these two groups the average age of the participants should be approximately the same and the migraine days per month as well. Randomization is now done by computer programs and no researcher is involved. This process finally allows for statistical analysis of the results and is also quite safe
generalization of these results to the general population if the number of participants is large (typically > 500 in each group). One group will receive the study drug and the other a placebo. No one, neither the participants nor the researchers, will know who is receiving medication and who is receiving a placebo.

This means "blind". After a period of time e.g. After 12 weeks and recording migraine attacks in an electronic diary, it is determined whether the group that received the drug had a greater reduction than the group that received a placebo. If this reduction was statistically significant, it means that the drug works and is not a random phenomenon. Studies of this type are a method that is accepted to ensure causal claims and hypotheses regarding a drug's ability to produce clinical effects.

This statistical significance is denoted mathematically by the so-called p-value, which if it is less than 0.05 means that something is statistically significant (with 95% certainty). Some, even doctors, looking at the results of the studies, comment that the effect of a new drug may be statistically
important, but that e.g. only differed by 2 migraine days per month compared to placebo. That is, the placebo group may have reduced their migraine days from 9 in the month before taking the placebo to 7 days after, while the drug group went from 9 days to 5 days. They claim it "only" decreased by 2 days compared to Placebo, so it's "slightly" statistically significant.

The big mistake here is that it has not been understood that a particular study is so designed (and overseen by the regulatory authorities) to answer us in relation to some goals that it sets, if it meets them in the certain period of time. Ie whether the drug you are researching works or not. Only that. If its effect is statistically significant compared to the placebo, the drug acts on the disease being studied. This is the only information. To put it in the words of a well-known statistician:
“…there is no difference between a p-value of 0.049 and a p-value of 0.00001. Both provide the same degree of evidence against the null hypothesis. It makes no sense to say that a result is statistically "very" or "very" significant. A result is either statistically significant or it is not.”
Salsburg David (1993) “The use of the statistical methods in the analysis of clinical studies. Journal of clinical epidemiology

And finally, something that in a 12-week study has a p-value of 0.001 may, if we extend the study to 52 weeks, have a p-value of 0.049 and so on. For this very reason, long-term studies are carried out.
In conclusion, the correct reading of the results of a study is not an easy task because it requires the knowledge for correct interpretation. And this knowledge is provided by medical statistics which is as necessary in modern medicine as the hammer of the neurologist.