User login
As defined by the American Academy of Orthopaedic Surgeons (AAOS) in 1996, conflict of interest (COI) is the “circumstance that exists when, because of personal financial gain, an individual has the potential to be less than objective when called on to reach a judgment or interpret a result.”1 In medical research, COIs often occur in relationships between physician-researchers and pharmaceutical, medical device, and biotechnology companies. These relationships usually take the form of research grants but can also arise when the researcher has a financial interest in the product being tested or in the company that manufactures the product.
Although constructive collaboration between academic medicine and industry has worked to improve health care and ultimately benefit patients, potential drawbacks of such relationships include sequestration and suppression of results that may be disadvantageous to the industry sponsor,2 increased likelihood of reporting positive results (pro-industry),3-7 and biased study designs.8 The nature of such relationships may threaten the integrity of scientific studies, the objectivity of medical education, the quality of patient care, and the public’s trust in medicine.9
Financial relationships and affiliations are increasing as we seek to answer a growing number of clinical questions—with funding often being a limiting factor. At national scientific meetings, the number of presentations reporting COIs reflects this trend. Paper and poster presentations accepted for annual meetings of the Orthopaedic Trauma Association (OTA) and reporting a COI increased from 7.6% in 1993 to 12.6% in 2002 (P = .0129).2
Medical subspecialties outside of orthopedics are experiencing similar trends. Most notable is the American Psychiatric Association (APA). After the APA published a mandatory financial COI disclosure policy in the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5), the percentage of task force members reporting industry relationships increased by 12%.10 Analysis of the DSM-5 panels demonstrated that the panels with the largest percentage of reported COIs are those for which pharmacological treatment is the first-line intervention, including the panels for mood disorders (67%), psychotic disorders (83%) and sleep/wake disorders (100%).10 Moreover, the industry ties reported are to the pharmaceutical companies that manufacture the medications used to treat these disorders or to companies that service the pharmaceutical industry.10
The degree to which financial COIs affect the interpretation of the orthopedic literature has never been quantified. Although it is clear that COIs can confound the results and reporting of data, how the medical community uses disclosures when interpreting the literature and when formulating opinions that may or may not affect their practice patterns is largely unknown.
We conducted a study to evaluate how a hypothetical financial COI disclosure would influence the interpretation of data by orthopedic clinicians. We also wanted to determine the reliability of the data as perceived in association with different study designs, levels of evidence, research institutional settings, and reporting of positive or negative results.
Methods
We asked members of the Arthroscopy Association of North America (AANA) and the American Orthopaedic Society for Sports Medicine (AOSSM) to complete a multiple-choice situational questionnaire (Table). The questionnaire assesses the degree to which respondents use COI disclosures when interpreting the literature. It further explores the perceived clinical value of a study with a given reported COI, assuming variations in study design, research institutional setting, and significance of results. The fictional research team disclosed the project was funded by a pharmaceutical company and all team members received consulting compensation. The survey and study were reviewed and approved by our institutional review board. The survey consisted of 14 multiple-choice questions that allowed for only 1 answer selection per person and allowed survey takers to skip questions they did not wish to answer. The survey questions and associated response options appear in edited form in the Table. A link to the questionnaire (https://www.surveymonkey.com/s/MPCCLCX) was sent with a message explaining the study. The responses to the questionnaire constituted the data.
Results
We sent a request to participate in the survey to 750 physicians and received 522 responses (overall response rate, 70%). The response rate for each question equaled or exceeded 98%.
The majority of respondents (95.6%) were male. Ninety-nine percent of respondents were orthopedic surgeons. The Northeast (US) was the most common geographical practice location of respondents (32%), followed by the Midwest (19.1%) and the Southeast (16.6%). Most respondents (40%) had been in practice for more than 20 years; 67% had been in practice a minimum of 10 years. The majority (68.8%) were employed by private practice groups, either single specialty (57.8%) or multispecialty (11%).
Eighty percent of respondents strongly agreed that COI disclosure is important when interpreting study results, 62% reported always reading the disclosure slide during academy or other meeting presentations, and 41% reported always using this information when deciding how to interpret scientific data.
Seventy-five percent of respondents thought the study—an academic-center case series with significant results in favor of the pharmaceutical company funding the study—was biased (42% indicated biased with merit, 33% biased without merit). Twenty-three percent thought the study was possibly biased, but likely trustworthy given the academic institutional affiliation. When the study setting was changed to community hospital, 95% thought the study was biased (51% biased with merit, 44% biased without merit). With the same study performed at an academic center, and no statistically significant results (not in favor of the pharmaceutical company funding the study), 88% thought the study had merit (46% biased with merit, 42% unbiased with merit).
When the study design was changed to a randomized controlled trial (level I evidence) conducted at an academic center with negative results, an overwhelming 95% of respondents thought the study had merit (33% biased with merit, 62% unbiased with merit). Given the same study design at an academic center, with positive results, 78% still thought the study had merit (39% biased with merit, 39% unbiased with merit). An additional 18% thought the study was biased, but still likely trustworthy given the academic institutional affiliation. Finally, given a randomized controlled trial and positive results, but with the research setting a small community practice, 90% thought the study had merit (51% biased with merit, 39% unbiased with merit). The percentage of respondents who found the study biased and likely without merit increased from 3.7% to 9.5% when the institutional affiliation changed from academic to community.
Discussion
As governmental funding sources become increasingly limited, the role of industry sponsorship of orthopedic research has grown. Potential drawbacks and biases of such research support have been well described—most notably, increased positive result reporting, suppression of results that may be disadvantageous to the industry sponsor, and biased study designs.2-8 However, the extent to which financial COIs affect the orthopedic medical community’s interpretation of the literature has never been quantified. To our knowledge, the present study is the first to quantify the impact of reported COI on study interpretation.
Our goal was to examine how reported financial COIs influence the interpretation of the literature by the orthopedic medical community. Moreover, we wanted to determine the perceived reliability of the data when variables (study design, institutional affiliation, positive vs negative results) were changed. The results of our survey indicate that, when a financial COI is reported, study reliability is perceived as highest when negative results were found.
Our survey noted a discrepancy between the documented importance of the hypothetical research team’s COI disclosure and the use of such disclosures when interpreting study results. Eighty percent of respondents agreed that COI disclosure is important when interpreting study results, but only 62% reported always reading disclosures, and even fewer (41%) reported always using the information when interpreting results. It is unclear exactly why this trend exists, as one would expect the percentages to be more similar. These particular survey questions were formed around using COI disclosures when interpreting study results during academic presentations at national meetings and not during the review of published literature. It is possible that positioning the COI disclosure at the beginning of a presentation has an effect, but only 3.7% of respondents indicated they seldom remembered the disclosure by the end of the presentation. The results of our survey may have varied if the questions had targeted reading and interpreting the literature.
Interestingly, the results of these survey questions tended to be more consistent with rates of reported financial COI by presenters at national orthopedic meetings. A study published in the New England Journal of Medicine found that less than 80% of orthopedic surgeons reported their disclosures at a large annual meeting (AAOS), even when the disclosure involved payments pertinent to the research they were presenting.5 When the payments were indirectly related to the research, the percentage of surgeons reporting disclosures was 50%, almost the same as the disclosure rate for unrelated payments.5
When the study was changed to a level I randomized controlled trial, more survey respondents found it to be less biased and have more merit. Although it would seem intuitive for a study with a higher level of evidence to carry more clinical value during interpretation, this may not hold true in the setting of industry-sponsored clinical trials. Several studies have documented a significant association between the reporting of positive results and industry-sponsored randomized clinical trials. In 2008, Khan and colleagues3 examined 100 orthopedic randomized clinical trials reported in 5 major orthopedic subspecialty journals over a 2-year period. The association between industry funding and favorable outcome in all original randomized clinical trials was strong and significant (P < .001). This is not surprising, given the amount of time and money required for a well-designed clinical study. Commercial products with preclinical promise are pushed to testing in a clinical trial, whereas resources would not be wasted on products lacking preclinical merit.
The most important variable affecting interpretation of study merit by survey respondents was the reporting of negative results. As more researchers are developing COIs, many studies are discovering a relationship between COIs and outcomes of research studies. Reviewing the adult total joint literature, Ezzet8 found an industry funding rate of 50%. Positive results were reported in 93% of cases in commercially funded studies versus 37% of cases in independently funded studies. Furthermore, no negative results were reported by investigators who were receiving royalties from the respective companies.
Studies across the medical literature have also found this association between industry sponsorship and reporting of positive results. One such study, reported by Valachis and colleagues7 in the Journal of Clinical Oncology, examined more than 80 economic analyses of targeted oncologic therapies and found the studies funded by pharmaceutical companies were more likely to report favorable qualitative cost estimates. In addition, when studies with a COI disclosure were examined, those reporting any financial relationship with a manufacturer (eg, author affiliation, funding) were more likely than those without such a relationship to report favorable results.
Our study had several limitations. First, as most of the survey respondents were orthopedic surgeons, extrapolating their data to the medical community at large may not be appropriate, as each specialty may view industry affiliations differently. In addition, respondents were asked to base their interpretations of a study on conclusions we predetermined—no direct visualization of the data set or statistical testing methods. It is possible that these responses may have been different had the respondents had the opportunity to further evaluate the study in question. In a recent study, Altwairgi and colleagues11 found that 10% of randomized clinical trials involving lung cancer treatment were reported with different conclusions in their full manuscripts relative to their abstracts. We think our survey design perhaps best mimics an annual meeting environment in which participants have very limited ability to interpret studies and may rely more heavily on the factors we investigated—study design, significance of findings, and setting, all similar to information presented in an abstract—when making informed decisions. Although our response rate was only 70%, this is comparable to or better than the rates in similar survey studies that used email-based questionnaires.12,13
Another limitation was that our survey may have forced respondents into answers they did not entirely agree with, given the limited options of the multiple-choice response format and the specific wording of the questions. Our conclusions may have been more dramatic when we were evaluating whether the study was deemed meritorious or not. However, there is no adopted standard for evaluating the extent of bias perceived by a clinician. We thought it was important to include answer options indicating a study had merit despite obvious bias in design and execution. That a study had merit can mean different things. It may change clinical practice, may require further study and reproducibility, or may not be significant enough to matter. Asking follow-up questions to evaluate this perception among the respondents could have provided validity to the term merit. Further studies in this field are needed to determine how studies are interpreted and translated into clinical practice by various clinicians.
Conclusion
Although the present study is not a quantitative analysis of the determination of bias in the orthopedic community, it is the first to evaluate orthopedic surgeons’ perceptions on the basis of key fundamentals of orthopedic research relative to COI. It is clear from our study results that introducing levels of evidence to the orthopedic milieu has had a significant impact both on the quality of research and on the foundational use of deductive reasoning when interpreting the literature. Reporting negative outcomes is perhaps the most important factor in eliminating the perception of bias among orthopedic surgeons. To what extent a perceived COI plays into medical decision-making and the ultimate treatment of patients is still relatively unknown.
1. Lubahn JD, Mankin CJ, Mankin HJ, Kuhn PJ. Orthopaedics, ethics, and industry. Appropriateness of gifts, grants, and awards. Clin Orthop Relat Res. 2000;(371):256-263.
2. Kubiak EN, Park SS, Egol K, Zuckerman JD, Koval KJ. Increasingly conflicted: an analysis of conflicts of interest reported at the annual meetings of the Orthopaedic Trauma Association. Bull Hosp Jt Dis. 2006;63(3-4):83-87.
3. Khan SN, Mermer MJ, Myers E, Sandhu HS. The roles of funding source, clinical trial outcome, and quality of reporting in orthopedic surgery literature. Am J Orthop. 2008;37(12):E205-E212.
4. Okike K, Kocher MS, Mehlman CT, Bhandari M. Conflict of interest in orthopaedic research. An association between findings and funding in scientific presentations. J Bone Joint Surg Am. 2007;89(3):608-613.
5. Okike K, Kocher MS, Wei EX, Mehlman CT, Bhandari M. Accuracy of conflict-of-interest disclosures reported by physicians. N Engl J Med. 2009;361(15):1466-1474.
6. Shah RV, Albert TJ, Bruegel-Sanchez V, Vaccaro AR, Hilibrand AS, Grauer JN. Industry support and correlation to study outcome for papers published in Spine. Spine. 2005;30(9):1099-1104.
7. Valachis A, Polyzos NP, Nearchou A, Lind P, Mauri D. Financial relationships in economic analyses of targeted therapies in oncology. J Clin Oncol. 2012;30(12):1316-1320.
8. Ezzet KA. The prevalence of corporate funding in adult lower extremity research and its correlation with reported results. J Arthroplasty. 2003;18(7 suppl 1):138-145.
9. Lo B, Field MJ, eds; Institute of Medicine, Committee on Conflict of Interest in Medical Research, Education, and Practice, Board on Health Sciences Policy. Conflict of Interest in Medical Research, Education, and Practice. Washington, DC: National Academies Press; 2009. http://www.ncbi.nlm.nih.gov/books/NBK22942. Accessed September 29, 2015.
10. Cosgrove L, Krimsky S. A comparison of DSM-IV and DSM-5 panel members’ financial associations with industry: a pernicious problem persists. PLoS Med. 2012;9(3):e1001190.
11. Altwairgi AK, Booth CM, Hopman WM, Baetz TD. Discordance between conclusions stated in the abstract and conclusions in the article: analysis of published randomized controlled trials of systemic therapy in lung cancer. J Clin Oncol. 2012;30(28):3552-3557.
12. Decoster LC, Vailas JC, Swartz WG. Functional ACL bracing. A survey of current opinion and practice. Am J Orthop. 1995;24(11):838-843.
13. Mann BJ, Grana WA, Indelicato PA, O’Neill DF, George SZ. A survey of sports medicine physicians regarding psychological issues in patient-athletes. Am J Sports Med. 2007;35(12):2140-2147.
As defined by the American Academy of Orthopaedic Surgeons (AAOS) in 1996, conflict of interest (COI) is the “circumstance that exists when, because of personal financial gain, an individual has the potential to be less than objective when called on to reach a judgment or interpret a result.”1 In medical research, COIs often occur in relationships between physician-researchers and pharmaceutical, medical device, and biotechnology companies. These relationships usually take the form of research grants but can also arise when the researcher has a financial interest in the product being tested or in the company that manufactures the product.
Although constructive collaboration between academic medicine and industry has worked to improve health care and ultimately benefit patients, potential drawbacks of such relationships include sequestration and suppression of results that may be disadvantageous to the industry sponsor,2 increased likelihood of reporting positive results (pro-industry),3-7 and biased study designs.8 The nature of such relationships may threaten the integrity of scientific studies, the objectivity of medical education, the quality of patient care, and the public’s trust in medicine.9
Financial relationships and affiliations are increasing as we seek to answer a growing number of clinical questions—with funding often being a limiting factor. At national scientific meetings, the number of presentations reporting COIs reflects this trend. Paper and poster presentations accepted for annual meetings of the Orthopaedic Trauma Association (OTA) and reporting a COI increased from 7.6% in 1993 to 12.6% in 2002 (P = .0129).2
Medical subspecialties outside of orthopedics are experiencing similar trends. Most notable is the American Psychiatric Association (APA). After the APA published a mandatory financial COI disclosure policy in the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5), the percentage of task force members reporting industry relationships increased by 12%.10 Analysis of the DSM-5 panels demonstrated that the panels with the largest percentage of reported COIs are those for which pharmacological treatment is the first-line intervention, including the panels for mood disorders (67%), psychotic disorders (83%) and sleep/wake disorders (100%).10 Moreover, the industry ties reported are to the pharmaceutical companies that manufacture the medications used to treat these disorders or to companies that service the pharmaceutical industry.10
The degree to which financial COIs affect the interpretation of the orthopedic literature has never been quantified. Although it is clear that COIs can confound the results and reporting of data, how the medical community uses disclosures when interpreting the literature and when formulating opinions that may or may not affect their practice patterns is largely unknown.
We conducted a study to evaluate how a hypothetical financial COI disclosure would influence the interpretation of data by orthopedic clinicians. We also wanted to determine the reliability of the data as perceived in association with different study designs, levels of evidence, research institutional settings, and reporting of positive or negative results.
Methods
We asked members of the Arthroscopy Association of North America (AANA) and the American Orthopaedic Society for Sports Medicine (AOSSM) to complete a multiple-choice situational questionnaire (Table). The questionnaire assesses the degree to which respondents use COI disclosures when interpreting the literature. It further explores the perceived clinical value of a study with a given reported COI, assuming variations in study design, research institutional setting, and significance of results. The fictional research team disclosed the project was funded by a pharmaceutical company and all team members received consulting compensation. The survey and study were reviewed and approved by our institutional review board. The survey consisted of 14 multiple-choice questions that allowed for only 1 answer selection per person and allowed survey takers to skip questions they did not wish to answer. The survey questions and associated response options appear in edited form in the Table. A link to the questionnaire (https://www.surveymonkey.com/s/MPCCLCX) was sent with a message explaining the study. The responses to the questionnaire constituted the data.
Results
We sent a request to participate in the survey to 750 physicians and received 522 responses (overall response rate, 70%). The response rate for each question equaled or exceeded 98%.
The majority of respondents (95.6%) were male. Ninety-nine percent of respondents were orthopedic surgeons. The Northeast (US) was the most common geographical practice location of respondents (32%), followed by the Midwest (19.1%) and the Southeast (16.6%). Most respondents (40%) had been in practice for more than 20 years; 67% had been in practice a minimum of 10 years. The majority (68.8%) were employed by private practice groups, either single specialty (57.8%) or multispecialty (11%).
Eighty percent of respondents strongly agreed that COI disclosure is important when interpreting study results, 62% reported always reading the disclosure slide during academy or other meeting presentations, and 41% reported always using this information when deciding how to interpret scientific data.
Seventy-five percent of respondents thought the study—an academic-center case series with significant results in favor of the pharmaceutical company funding the study—was biased (42% indicated biased with merit, 33% biased without merit). Twenty-three percent thought the study was possibly biased, but likely trustworthy given the academic institutional affiliation. When the study setting was changed to community hospital, 95% thought the study was biased (51% biased with merit, 44% biased without merit). With the same study performed at an academic center, and no statistically significant results (not in favor of the pharmaceutical company funding the study), 88% thought the study had merit (46% biased with merit, 42% unbiased with merit).
When the study design was changed to a randomized controlled trial (level I evidence) conducted at an academic center with negative results, an overwhelming 95% of respondents thought the study had merit (33% biased with merit, 62% unbiased with merit). Given the same study design at an academic center, with positive results, 78% still thought the study had merit (39% biased with merit, 39% unbiased with merit). An additional 18% thought the study was biased, but still likely trustworthy given the academic institutional affiliation. Finally, given a randomized controlled trial and positive results, but with the research setting a small community practice, 90% thought the study had merit (51% biased with merit, 39% unbiased with merit). The percentage of respondents who found the study biased and likely without merit increased from 3.7% to 9.5% when the institutional affiliation changed from academic to community.
Discussion
As governmental funding sources become increasingly limited, the role of industry sponsorship of orthopedic research has grown. Potential drawbacks and biases of such research support have been well described—most notably, increased positive result reporting, suppression of results that may be disadvantageous to the industry sponsor, and biased study designs.2-8 However, the extent to which financial COIs affect the orthopedic medical community’s interpretation of the literature has never been quantified. To our knowledge, the present study is the first to quantify the impact of reported COI on study interpretation.
Our goal was to examine how reported financial COIs influence the interpretation of the literature by the orthopedic medical community. Moreover, we wanted to determine the perceived reliability of the data when variables (study design, institutional affiliation, positive vs negative results) were changed. The results of our survey indicate that, when a financial COI is reported, study reliability is perceived as highest when negative results were found.
Our survey noted a discrepancy between the documented importance of the hypothetical research team’s COI disclosure and the use of such disclosures when interpreting study results. Eighty percent of respondents agreed that COI disclosure is important when interpreting study results, but only 62% reported always reading disclosures, and even fewer (41%) reported always using the information when interpreting results. It is unclear exactly why this trend exists, as one would expect the percentages to be more similar. These particular survey questions were formed around using COI disclosures when interpreting study results during academic presentations at national meetings and not during the review of published literature. It is possible that positioning the COI disclosure at the beginning of a presentation has an effect, but only 3.7% of respondents indicated they seldom remembered the disclosure by the end of the presentation. The results of our survey may have varied if the questions had targeted reading and interpreting the literature.
Interestingly, the results of these survey questions tended to be more consistent with rates of reported financial COI by presenters at national orthopedic meetings. A study published in the New England Journal of Medicine found that less than 80% of orthopedic surgeons reported their disclosures at a large annual meeting (AAOS), even when the disclosure involved payments pertinent to the research they were presenting.5 When the payments were indirectly related to the research, the percentage of surgeons reporting disclosures was 50%, almost the same as the disclosure rate for unrelated payments.5
When the study was changed to a level I randomized controlled trial, more survey respondents found it to be less biased and have more merit. Although it would seem intuitive for a study with a higher level of evidence to carry more clinical value during interpretation, this may not hold true in the setting of industry-sponsored clinical trials. Several studies have documented a significant association between the reporting of positive results and industry-sponsored randomized clinical trials. In 2008, Khan and colleagues3 examined 100 orthopedic randomized clinical trials reported in 5 major orthopedic subspecialty journals over a 2-year period. The association between industry funding and favorable outcome in all original randomized clinical trials was strong and significant (P < .001). This is not surprising, given the amount of time and money required for a well-designed clinical study. Commercial products with preclinical promise are pushed to testing in a clinical trial, whereas resources would not be wasted on products lacking preclinical merit.
The most important variable affecting interpretation of study merit by survey respondents was the reporting of negative results. As more researchers are developing COIs, many studies are discovering a relationship between COIs and outcomes of research studies. Reviewing the adult total joint literature, Ezzet8 found an industry funding rate of 50%. Positive results were reported in 93% of cases in commercially funded studies versus 37% of cases in independently funded studies. Furthermore, no negative results were reported by investigators who were receiving royalties from the respective companies.
Studies across the medical literature have also found this association between industry sponsorship and reporting of positive results. One such study, reported by Valachis and colleagues7 in the Journal of Clinical Oncology, examined more than 80 economic analyses of targeted oncologic therapies and found the studies funded by pharmaceutical companies were more likely to report favorable qualitative cost estimates. In addition, when studies with a COI disclosure were examined, those reporting any financial relationship with a manufacturer (eg, author affiliation, funding) were more likely than those without such a relationship to report favorable results.
Our study had several limitations. First, as most of the survey respondents were orthopedic surgeons, extrapolating their data to the medical community at large may not be appropriate, as each specialty may view industry affiliations differently. In addition, respondents were asked to base their interpretations of a study on conclusions we predetermined—no direct visualization of the data set or statistical testing methods. It is possible that these responses may have been different had the respondents had the opportunity to further evaluate the study in question. In a recent study, Altwairgi and colleagues11 found that 10% of randomized clinical trials involving lung cancer treatment were reported with different conclusions in their full manuscripts relative to their abstracts. We think our survey design perhaps best mimics an annual meeting environment in which participants have very limited ability to interpret studies and may rely more heavily on the factors we investigated—study design, significance of findings, and setting, all similar to information presented in an abstract—when making informed decisions. Although our response rate was only 70%, this is comparable to or better than the rates in similar survey studies that used email-based questionnaires.12,13
Another limitation was that our survey may have forced respondents into answers they did not entirely agree with, given the limited options of the multiple-choice response format and the specific wording of the questions. Our conclusions may have been more dramatic when we were evaluating whether the study was deemed meritorious or not. However, there is no adopted standard for evaluating the extent of bias perceived by a clinician. We thought it was important to include answer options indicating a study had merit despite obvious bias in design and execution. That a study had merit can mean different things. It may change clinical practice, may require further study and reproducibility, or may not be significant enough to matter. Asking follow-up questions to evaluate this perception among the respondents could have provided validity to the term merit. Further studies in this field are needed to determine how studies are interpreted and translated into clinical practice by various clinicians.
Conclusion
Although the present study is not a quantitative analysis of the determination of bias in the orthopedic community, it is the first to evaluate orthopedic surgeons’ perceptions on the basis of key fundamentals of orthopedic research relative to COI. It is clear from our study results that introducing levels of evidence to the orthopedic milieu has had a significant impact both on the quality of research and on the foundational use of deductive reasoning when interpreting the literature. Reporting negative outcomes is perhaps the most important factor in eliminating the perception of bias among orthopedic surgeons. To what extent a perceived COI plays into medical decision-making and the ultimate treatment of patients is still relatively unknown.
As defined by the American Academy of Orthopaedic Surgeons (AAOS) in 1996, conflict of interest (COI) is the “circumstance that exists when, because of personal financial gain, an individual has the potential to be less than objective when called on to reach a judgment or interpret a result.”1 In medical research, COIs often occur in relationships between physician-researchers and pharmaceutical, medical device, and biotechnology companies. These relationships usually take the form of research grants but can also arise when the researcher has a financial interest in the product being tested or in the company that manufactures the product.
Although constructive collaboration between academic medicine and industry has worked to improve health care and ultimately benefit patients, potential drawbacks of such relationships include sequestration and suppression of results that may be disadvantageous to the industry sponsor,2 increased likelihood of reporting positive results (pro-industry),3-7 and biased study designs.8 The nature of such relationships may threaten the integrity of scientific studies, the objectivity of medical education, the quality of patient care, and the public’s trust in medicine.9
Financial relationships and affiliations are increasing as we seek to answer a growing number of clinical questions—with funding often being a limiting factor. At national scientific meetings, the number of presentations reporting COIs reflects this trend. Paper and poster presentations accepted for annual meetings of the Orthopaedic Trauma Association (OTA) and reporting a COI increased from 7.6% in 1993 to 12.6% in 2002 (P = .0129).2
Medical subspecialties outside of orthopedics are experiencing similar trends. Most notable is the American Psychiatric Association (APA). After the APA published a mandatory financial COI disclosure policy in the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5), the percentage of task force members reporting industry relationships increased by 12%.10 Analysis of the DSM-5 panels demonstrated that the panels with the largest percentage of reported COIs are those for which pharmacological treatment is the first-line intervention, including the panels for mood disorders (67%), psychotic disorders (83%) and sleep/wake disorders (100%).10 Moreover, the industry ties reported are to the pharmaceutical companies that manufacture the medications used to treat these disorders or to companies that service the pharmaceutical industry.10
The degree to which financial COIs affect the interpretation of the orthopedic literature has never been quantified. Although it is clear that COIs can confound the results and reporting of data, how the medical community uses disclosures when interpreting the literature and when formulating opinions that may or may not affect their practice patterns is largely unknown.
We conducted a study to evaluate how a hypothetical financial COI disclosure would influence the interpretation of data by orthopedic clinicians. We also wanted to determine the reliability of the data as perceived in association with different study designs, levels of evidence, research institutional settings, and reporting of positive or negative results.
Methods
We asked members of the Arthroscopy Association of North America (AANA) and the American Orthopaedic Society for Sports Medicine (AOSSM) to complete a multiple-choice situational questionnaire (Table). The questionnaire assesses the degree to which respondents use COI disclosures when interpreting the literature. It further explores the perceived clinical value of a study with a given reported COI, assuming variations in study design, research institutional setting, and significance of results. The fictional research team disclosed the project was funded by a pharmaceutical company and all team members received consulting compensation. The survey and study were reviewed and approved by our institutional review board. The survey consisted of 14 multiple-choice questions that allowed for only 1 answer selection per person and allowed survey takers to skip questions they did not wish to answer. The survey questions and associated response options appear in edited form in the Table. A link to the questionnaire (https://www.surveymonkey.com/s/MPCCLCX) was sent with a message explaining the study. The responses to the questionnaire constituted the data.
Results
We sent a request to participate in the survey to 750 physicians and received 522 responses (overall response rate, 70%). The response rate for each question equaled or exceeded 98%.
The majority of respondents (95.6%) were male. Ninety-nine percent of respondents were orthopedic surgeons. The Northeast (US) was the most common geographical practice location of respondents (32%), followed by the Midwest (19.1%) and the Southeast (16.6%). Most respondents (40%) had been in practice for more than 20 years; 67% had been in practice a minimum of 10 years. The majority (68.8%) were employed by private practice groups, either single specialty (57.8%) or multispecialty (11%).
Eighty percent of respondents strongly agreed that COI disclosure is important when interpreting study results, 62% reported always reading the disclosure slide during academy or other meeting presentations, and 41% reported always using this information when deciding how to interpret scientific data.
Seventy-five percent of respondents thought the study—an academic-center case series with significant results in favor of the pharmaceutical company funding the study—was biased (42% indicated biased with merit, 33% biased without merit). Twenty-three percent thought the study was possibly biased, but likely trustworthy given the academic institutional affiliation. When the study setting was changed to community hospital, 95% thought the study was biased (51% biased with merit, 44% biased without merit). With the same study performed at an academic center, and no statistically significant results (not in favor of the pharmaceutical company funding the study), 88% thought the study had merit (46% biased with merit, 42% unbiased with merit).
When the study design was changed to a randomized controlled trial (level I evidence) conducted at an academic center with negative results, an overwhelming 95% of respondents thought the study had merit (33% biased with merit, 62% unbiased with merit). Given the same study design at an academic center, with positive results, 78% still thought the study had merit (39% biased with merit, 39% unbiased with merit). An additional 18% thought the study was biased, but still likely trustworthy given the academic institutional affiliation. Finally, given a randomized controlled trial and positive results, but with the research setting a small community practice, 90% thought the study had merit (51% biased with merit, 39% unbiased with merit). The percentage of respondents who found the study biased and likely without merit increased from 3.7% to 9.5% when the institutional affiliation changed from academic to community.
Discussion
As governmental funding sources become increasingly limited, the role of industry sponsorship of orthopedic research has grown. Potential drawbacks and biases of such research support have been well described—most notably, increased positive result reporting, suppression of results that may be disadvantageous to the industry sponsor, and biased study designs.2-8 However, the extent to which financial COIs affect the orthopedic medical community’s interpretation of the literature has never been quantified. To our knowledge, the present study is the first to quantify the impact of reported COI on study interpretation.
Our goal was to examine how reported financial COIs influence the interpretation of the literature by the orthopedic medical community. Moreover, we wanted to determine the perceived reliability of the data when variables (study design, institutional affiliation, positive vs negative results) were changed. The results of our survey indicate that, when a financial COI is reported, study reliability is perceived as highest when negative results were found.
Our survey noted a discrepancy between the documented importance of the hypothetical research team’s COI disclosure and the use of such disclosures when interpreting study results. Eighty percent of respondents agreed that COI disclosure is important when interpreting study results, but only 62% reported always reading disclosures, and even fewer (41%) reported always using the information when interpreting results. It is unclear exactly why this trend exists, as one would expect the percentages to be more similar. These particular survey questions were formed around using COI disclosures when interpreting study results during academic presentations at national meetings and not during the review of published literature. It is possible that positioning the COI disclosure at the beginning of a presentation has an effect, but only 3.7% of respondents indicated they seldom remembered the disclosure by the end of the presentation. The results of our survey may have varied if the questions had targeted reading and interpreting the literature.
Interestingly, the results of these survey questions tended to be more consistent with rates of reported financial COI by presenters at national orthopedic meetings. A study published in the New England Journal of Medicine found that less than 80% of orthopedic surgeons reported their disclosures at a large annual meeting (AAOS), even when the disclosure involved payments pertinent to the research they were presenting.5 When the payments were indirectly related to the research, the percentage of surgeons reporting disclosures was 50%, almost the same as the disclosure rate for unrelated payments.5
When the study was changed to a level I randomized controlled trial, more survey respondents found it to be less biased and have more merit. Although it would seem intuitive for a study with a higher level of evidence to carry more clinical value during interpretation, this may not hold true in the setting of industry-sponsored clinical trials. Several studies have documented a significant association between the reporting of positive results and industry-sponsored randomized clinical trials. In 2008, Khan and colleagues3 examined 100 orthopedic randomized clinical trials reported in 5 major orthopedic subspecialty journals over a 2-year period. The association between industry funding and favorable outcome in all original randomized clinical trials was strong and significant (P < .001). This is not surprising, given the amount of time and money required for a well-designed clinical study. Commercial products with preclinical promise are pushed to testing in a clinical trial, whereas resources would not be wasted on products lacking preclinical merit.
The most important variable affecting interpretation of study merit by survey respondents was the reporting of negative results. As more researchers are developing COIs, many studies are discovering a relationship between COIs and outcomes of research studies. Reviewing the adult total joint literature, Ezzet8 found an industry funding rate of 50%. Positive results were reported in 93% of cases in commercially funded studies versus 37% of cases in independently funded studies. Furthermore, no negative results were reported by investigators who were receiving royalties from the respective companies.
Studies across the medical literature have also found this association between industry sponsorship and reporting of positive results. One such study, reported by Valachis and colleagues7 in the Journal of Clinical Oncology, examined more than 80 economic analyses of targeted oncologic therapies and found the studies funded by pharmaceutical companies were more likely to report favorable qualitative cost estimates. In addition, when studies with a COI disclosure were examined, those reporting any financial relationship with a manufacturer (eg, author affiliation, funding) were more likely than those without such a relationship to report favorable results.
Our study had several limitations. First, as most of the survey respondents were orthopedic surgeons, extrapolating their data to the medical community at large may not be appropriate, as each specialty may view industry affiliations differently. In addition, respondents were asked to base their interpretations of a study on conclusions we predetermined—no direct visualization of the data set or statistical testing methods. It is possible that these responses may have been different had the respondents had the opportunity to further evaluate the study in question. In a recent study, Altwairgi and colleagues11 found that 10% of randomized clinical trials involving lung cancer treatment were reported with different conclusions in their full manuscripts relative to their abstracts. We think our survey design perhaps best mimics an annual meeting environment in which participants have very limited ability to interpret studies and may rely more heavily on the factors we investigated—study design, significance of findings, and setting, all similar to information presented in an abstract—when making informed decisions. Although our response rate was only 70%, this is comparable to or better than the rates in similar survey studies that used email-based questionnaires.12,13
Another limitation was that our survey may have forced respondents into answers they did not entirely agree with, given the limited options of the multiple-choice response format and the specific wording of the questions. Our conclusions may have been more dramatic when we were evaluating whether the study was deemed meritorious or not. However, there is no adopted standard for evaluating the extent of bias perceived by a clinician. We thought it was important to include answer options indicating a study had merit despite obvious bias in design and execution. That a study had merit can mean different things. It may change clinical practice, may require further study and reproducibility, or may not be significant enough to matter. Asking follow-up questions to evaluate this perception among the respondents could have provided validity to the term merit. Further studies in this field are needed to determine how studies are interpreted and translated into clinical practice by various clinicians.
Conclusion
Although the present study is not a quantitative analysis of the determination of bias in the orthopedic community, it is the first to evaluate orthopedic surgeons’ perceptions on the basis of key fundamentals of orthopedic research relative to COI. It is clear from our study results that introducing levels of evidence to the orthopedic milieu has had a significant impact both on the quality of research and on the foundational use of deductive reasoning when interpreting the literature. Reporting negative outcomes is perhaps the most important factor in eliminating the perception of bias among orthopedic surgeons. To what extent a perceived COI plays into medical decision-making and the ultimate treatment of patients is still relatively unknown.
1. Lubahn JD, Mankin CJ, Mankin HJ, Kuhn PJ. Orthopaedics, ethics, and industry. Appropriateness of gifts, grants, and awards. Clin Orthop Relat Res. 2000;(371):256-263.
2. Kubiak EN, Park SS, Egol K, Zuckerman JD, Koval KJ. Increasingly conflicted: an analysis of conflicts of interest reported at the annual meetings of the Orthopaedic Trauma Association. Bull Hosp Jt Dis. 2006;63(3-4):83-87.
3. Khan SN, Mermer MJ, Myers E, Sandhu HS. The roles of funding source, clinical trial outcome, and quality of reporting in orthopedic surgery literature. Am J Orthop. 2008;37(12):E205-E212.
4. Okike K, Kocher MS, Mehlman CT, Bhandari M. Conflict of interest in orthopaedic research. An association between findings and funding in scientific presentations. J Bone Joint Surg Am. 2007;89(3):608-613.
5. Okike K, Kocher MS, Wei EX, Mehlman CT, Bhandari M. Accuracy of conflict-of-interest disclosures reported by physicians. N Engl J Med. 2009;361(15):1466-1474.
6. Shah RV, Albert TJ, Bruegel-Sanchez V, Vaccaro AR, Hilibrand AS, Grauer JN. Industry support and correlation to study outcome for papers published in Spine. Spine. 2005;30(9):1099-1104.
7. Valachis A, Polyzos NP, Nearchou A, Lind P, Mauri D. Financial relationships in economic analyses of targeted therapies in oncology. J Clin Oncol. 2012;30(12):1316-1320.
8. Ezzet KA. The prevalence of corporate funding in adult lower extremity research and its correlation with reported results. J Arthroplasty. 2003;18(7 suppl 1):138-145.
9. Lo B, Field MJ, eds; Institute of Medicine, Committee on Conflict of Interest in Medical Research, Education, and Practice, Board on Health Sciences Policy. Conflict of Interest in Medical Research, Education, and Practice. Washington, DC: National Academies Press; 2009. http://www.ncbi.nlm.nih.gov/books/NBK22942. Accessed September 29, 2015.
10. Cosgrove L, Krimsky S. A comparison of DSM-IV and DSM-5 panel members’ financial associations with industry: a pernicious problem persists. PLoS Med. 2012;9(3):e1001190.
11. Altwairgi AK, Booth CM, Hopman WM, Baetz TD. Discordance between conclusions stated in the abstract and conclusions in the article: analysis of published randomized controlled trials of systemic therapy in lung cancer. J Clin Oncol. 2012;30(28):3552-3557.
12. Decoster LC, Vailas JC, Swartz WG. Functional ACL bracing. A survey of current opinion and practice. Am J Orthop. 1995;24(11):838-843.
13. Mann BJ, Grana WA, Indelicato PA, O’Neill DF, George SZ. A survey of sports medicine physicians regarding psychological issues in patient-athletes. Am J Sports Med. 2007;35(12):2140-2147.
1. Lubahn JD, Mankin CJ, Mankin HJ, Kuhn PJ. Orthopaedics, ethics, and industry. Appropriateness of gifts, grants, and awards. Clin Orthop Relat Res. 2000;(371):256-263.
2. Kubiak EN, Park SS, Egol K, Zuckerman JD, Koval KJ. Increasingly conflicted: an analysis of conflicts of interest reported at the annual meetings of the Orthopaedic Trauma Association. Bull Hosp Jt Dis. 2006;63(3-4):83-87.
3. Khan SN, Mermer MJ, Myers E, Sandhu HS. The roles of funding source, clinical trial outcome, and quality of reporting in orthopedic surgery literature. Am J Orthop. 2008;37(12):E205-E212.
4. Okike K, Kocher MS, Mehlman CT, Bhandari M. Conflict of interest in orthopaedic research. An association between findings and funding in scientific presentations. J Bone Joint Surg Am. 2007;89(3):608-613.
5. Okike K, Kocher MS, Wei EX, Mehlman CT, Bhandari M. Accuracy of conflict-of-interest disclosures reported by physicians. N Engl J Med. 2009;361(15):1466-1474.
6. Shah RV, Albert TJ, Bruegel-Sanchez V, Vaccaro AR, Hilibrand AS, Grauer JN. Industry support and correlation to study outcome for papers published in Spine. Spine. 2005;30(9):1099-1104.
7. Valachis A, Polyzos NP, Nearchou A, Lind P, Mauri D. Financial relationships in economic analyses of targeted therapies in oncology. J Clin Oncol. 2012;30(12):1316-1320.
8. Ezzet KA. The prevalence of corporate funding in adult lower extremity research and its correlation with reported results. J Arthroplasty. 2003;18(7 suppl 1):138-145.
9. Lo B, Field MJ, eds; Institute of Medicine, Committee on Conflict of Interest in Medical Research, Education, and Practice, Board on Health Sciences Policy. Conflict of Interest in Medical Research, Education, and Practice. Washington, DC: National Academies Press; 2009. http://www.ncbi.nlm.nih.gov/books/NBK22942. Accessed September 29, 2015.
10. Cosgrove L, Krimsky S. A comparison of DSM-IV and DSM-5 panel members’ financial associations with industry: a pernicious problem persists. PLoS Med. 2012;9(3):e1001190.
11. Altwairgi AK, Booth CM, Hopman WM, Baetz TD. Discordance between conclusions stated in the abstract and conclusions in the article: analysis of published randomized controlled trials of systemic therapy in lung cancer. J Clin Oncol. 2012;30(28):3552-3557.
12. Decoster LC, Vailas JC, Swartz WG. Functional ACL bracing. A survey of current opinion and practice. Am J Orthop. 1995;24(11):838-843.
13. Mann BJ, Grana WA, Indelicato PA, O’Neill DF, George SZ. A survey of sports medicine physicians regarding psychological issues in patient-athletes. Am J Sports Med. 2007;35(12):2140-2147.