"What's Your Success Rate?": Understanding IVF Pregnancy Statistics: Part I
Joseph D. Schulman, MD
"They considered that an individual is a population of one million divided by one million."
"We know how to speak many falsehoods which resemble real things, but we know, when we will, how to speak true things."
"Failing the possibility of measuring that which you desire, the lust for measurement may result in measuring something else - and perhaps forgetting the difference - or in ignoring some things because they cannot be measured."
G. Udney Yule, as quoted by Nobel laureate Friedrich Hayek
"Everything should be made as simple as possible, but not simpler."
Click here to read the new supplement: "What's Your Success Rate?": Part II
Probably no topic in the complex field of infertility is as confusing and misleading as IVF "success rate" statistics. This is well recognized by many experts in the field and by independent statisticians. The following discussion is a substantially modified version of an article on this subject in a major medical journal (Human Reproduction), and has resulted from many years of experience in the IVF field (since 1973), numerous discussions with professional colleagues, and input from some knowledgeable the Genetics & IVF Institute patients, including professors of statistics. The principles developed in this discussion also apply to other ART (assisted reproductive technology) methods, such as GIFT, and to comparisons between different ART methods as well as to comparisons between ART programs. IVF is by far the most important ART method, and for purposes of simplicity will be used as the model in the following discussion.
The central thesis, which staff of GIVF have lectured on intermittently since 1984, is that success rates in an IVF program can be made to be almost any number that is desired for public image, without accomplishing anything of value for individual patients. While this is well known and privately acknowledged by many of the leaders in the IVF field, the public does not generally understand this. Hence the most commonly asked question of patients contacting IVF programs, a least in the United States, is still "What's your success rate?" The honest answer, of course, is "Less than 1% to about 50%, once we understand what type, duration, and severity of infertility problems you have", but this is not the answer necessarily provided.
The famous magician, Harry Houdini, became well known not only for his own magic tricks and escape artistry, but also for exposing the clever methods used to create illusions by individuals who claimed to have supernatural powers. In brief, Houdini said, there is always another explanation that is quite simple and perfectly natural, once you understand it. The same is true for IVF success rates. While there are talented individuals in the IVF field, and the technology was created by at least one individual, Professor R.G. Edwards, who is a legitimate genius, there are no individuals in this medical discipline with supernatural powers, neither physicians nor embryologists. Yet, from time to time, even major newspapers will run articles about some apparent Uri Geller who can do the impossible in the IVF field, such as achieving pregnancy rates twice as high as all other IVF centers in their region. Some reporters, and many patients, fail to ask "How is this really done?" and instead choose to believe in miracle workers. In one recent example in a prominent newspaper, an IVF center with already exceptional "statistics" even claimed that when a large part of their embryology team departed their results further improved! And in many American cities, it is a frequent occurrence to see "success rate" advertisements aimed at patients, or for the offices of obstetricians and gynecologists to receive advertisements (called "informational brochures") from some local IVF program, often of small size and otherwise undistinguished, showing that they have the "best success rates" in the area, and even claiming to support this assertion with comparative data from publicly available data sources.
So, what's the trick? What is really going on here? To understand this, one essential principle of statistics needs to be kept in mind. All statistical methods, including medical statistics, always assess data from populations. These populations are sampled, averaged, etc. But a patient is not a population. Each patient has a unique set of circumstances that defines their own specific likelihood of achieving a pregnancy if they receive medical care at a highly experienced IVF facility. THE KEY TO MAKING SUCCESS RATE STATISTICS LOOK GOOD IS TO CONTROL THE POPULATION DATA. This can be and is done systematically at some IVF centers. Populations cannot be equalized between different IVF centers, and this manipulation (known technically as "selection") is invisible in published or advertised statistics. So many except the most scientifically sophisticated patients may be inclined to draw the conclusion that the center with the best statistics provides the best health care.
To show how this works, let's consider some of the systematic ways in which success rate statistics can be manipulated. The cumulative impact of these biases and manipulations on the final statistics is enormous, as will be shown. A major theme in these strategies is selection, but it is not the only one. We assume in this discussion that the goal is to maximize, for the purposes of public perception, the pregnancy rate following fresh embryo transfer in an IVF cycle.
1. Enrich the patient population with the best patients. In the words of a famous European infertility expert, "The patients who are easiest to get pregnant get pregnant most easily". Since individual IVF couples have pregnancy likelihoods varying enormously, a key is to enrich the statistically reported population with the better patients, and to discourage, reject, or reclassify the worse. This is accomplished by a combination of positive and negative selection.
Positive selection criteria include but are not limited to: young females, previous normal uterine or ectopic pregnancies, regular menstrual cycles, limited or no prior treatment, short duration of infertility, no or few prior IVF or gonadotropin (Metrodin, Humegon, or Pergonal) IUI failures, low FSH values, normal clomiphene challenge tests, known "high responders", completely normal semen quality, and absence of prior pregnancy losses. These are just a few of the factors utilized. Negative selection involves avoiding or reducing the cycle frequency of the opposite types of patients, such as failures in other programs, low responders, borderline FSH values, prolonged unexplained infertility, known or suspected immune/implantation problems, "low-quality" embryos, encouraging all except the best patients especially those over 40 to move to donor egg therapy, and so forth. To further positive selection, some centers use IVF or GIFT as an initial or very early therapy instead of other feasible alternatives. Some IVF centers even offer multicycle pricing or cheaper cycle prices to "qualified" (ideal) patients, another form of positive selection. Other centers classify more difficult patients as "research patients", and omit the "research" results from their reported program statistics.
2. Use a "waiting list" and program downtimes to facilitate the above types of selection. "Good patients" go to the head of the waiting list; "bad patients" are rejected, must wait to recycle, are told to do donor egg, etc. In some cases, particularly "bad patients" are courteously referred to a local competing IVF center. Well known to some IVF professionals is the Dr. Anonymous who, called in to fix a languishing IVF program, carefully studied over 100 medical records on a waiting list, selected the 20 best patients, initiated 8 pregnancies, and then advertised a 40% pregnancy rate.
3. Reclassify cycles. Poor cycles are converted to insemination cycles, or canceled. Patients having particularly good gonadotropin-stimulated IUI cycles are converted to IVF. A method used to force cycle cancellations in less favorable patients is to limit gonadotropin dosage, or to actually decrease, rather than increase, gonadotropin dosage in a cycle where initial response is poor, thus virtually guaranteeing that the cycle will be stopped.
4. Encourage or require cancellation when follicle response is limited, so that less favorable patients will opt to cancel early.
5. Encourage difficult patients to exit the program. There are many direct or subtle ways to achieve this, and there will always be centers like GIVF who will be willing to take over their care.
Once embryos are formed in non-canceled cycles, there are a number of other strategies to improve program statistics, of which two only are described immediately below. Many others can be utilized, but a detailed discussion would require a lengthy preliminary dissertation on human embryology.
6. Transfer larger numbers of embryos to the uterus. This is one of the simplest ways to increase pregnancy rates, and is widely practiced. In records we have reviewed, pseudonymous Dr. Triplet advises in writing to his patients that the usual number of embryos to be transferred is 5, in women in their mid-thirties it is 6, and at age 40 and above it is 7. Other centers simply don't put it in writing. 5 or more carefully selected embryos are frequently replaced in the uterus in some IVF programs; obviously, pregnancy rates will be higher than when a more conservative embryo policy is advised, and the difficulties associated with high multiple gestations such as prematurity and selective terminations do not appear in the IVF statistics.
7. Discourage cryopreservation. By highlighting that some embryos are lost through freezing and thawing, or by stating that the embryos "don't look good enough" to freeze, higher numbers of embryos can be "justifiably" transferred. (In addition, the fresh transfer or actual discarding of embryos with lower scores will enhance pregnancy rates in subsequent frozen embryos transfer cycles.)
Consider the following table, where the percentages shown in parentheses reference the current number divided by the number above it:
The "pregnancy rate" statistics look like this:
Note that in the foregoing example, Dr. Lookgood looks about twice as good. Yet the differences in pregnancy rates are achieved assuming identical skill in the performing of IVF by both programs. Dr. Helpall treats patients according to a philosophy typical of medicine in most other fields: take patients in the order they come to you, help those who have a chance of being helped, do your best for each individual patient, and let the population statistics fall where they may; Dr. Lookgood has a different vision.
The above examples illustrate how easy it is to create the impression of IVF program superiority with statistics which are "honest", even leaving aside other issues not reflected in the table such as cycle reclassification.
The above discussion has several implications which will be apparent to the reader who has followed the above analysis:
1. It is easy to "game" the system to maximize IVF statistics. The cumulative effects of a systematic strategy to do so are large. The misguided focus on "success rates" has created strong incentives, economic and otherwise, for IVF programs to act this way. It takes very strong confidence and discipline for an IVF team to place individual patients first, however difficult their problems, and to ignore their impact on the team's reported "success rates". We are very proud that physicians in so many IVF centers, including GIVF, have continued to place health care before statistics despite powerful temptations to do otherwise.
2. IVF statistics as currently gathered cannot, even in theory, prevent gaming. While exaggeration, lying, and cheating may be detectable in some cases (via external auditing, legal action, etc.) population statistics will still be grossly incorrect measures of IVF program quality.
3. Attempts to feed the appetite of patients, politicians, and insurance companies for information on IVF programs by the providing of statistical results summaries under the auspices of well-intentioned professional societies or potential licensing or regulatory authorities actually make the problem worse. This is because despite all best intentions the data receive an imprimatur of authenticity, objectivity, and fairness that they, even theoretically, cannot have. The limitations of scientifically unsound, statistical comparisons have been widely recognized in other fields of medicine, and there are many in the community of infertility experts who also acknowledge that this applies to infertility treatment as well. At Genetics & IVF Institute, we believe that this is a message which should be highlighted to all patients.
How, then, can couples with complex infertility problems be best advised to seek professional care? The answer is not simple, and cannot be reduced to a statistic, but it is really the same advice which thoughtful individuals usually follow in seeking professional services for any important purpose.
1. Look for long and extensive experience. Centers where thousands of IVF cycles have been performed and many hundreds or thousands of IVF pregnancies have been established almost surely have mastered IVF. If you may need ICSI, donor egg, reproductive immunology, or other special services, look for in-depth staff and experience with these.
2. Ask multiple physicians where they would go if they had a difficult infertility problem. Include in your questioning physicians outside of the infertility field.
3. Do not limit your consideration to local IVF programs only. The Mayo Clinic, the Massachusetts General Hospital, and other leading hospitals attracts patients with complex health care needs from throughout the United States. Many patients travel to find the best infertility care at leading national centers.
4. Take account of a record of innovation, since being at the cutting edge of medical advances almost always implies a high degree of skill and knowledge.
5. Talk to medical and other staff at potential IVF centers and attempt to develop your own sense of their integrity, intelligence, responsiveness, and compassion.
6. Learn as much as you can. Review the general reputation of the organization and its professionals locally and nationally.
7. Try to contact former patients. Talk to your friends with infertility problems.
8. Refuse to be directed anywhere by a health care plan. Fight for your right to choose. Insist upon alternatives. And, if necessary, be willing to spend your own money wisely to get the best health care.
9. Distrust waiting lists. Avoid apparent economic bargains. Ignore gimmicks.
10. Lastly, think hard and trust your own judgment. Your health care is very important, and the final decisions are yours!
Click here to continue to "What's Your Success Rate?": Part II