正文 · AI 翻译
Joint evaluation by 250 leading doctors from China's top-tier hospitals
By Luyu, reporting from Aofei Temple
QbitAI | WeChat public account QbitAI
By Luyu, reporting from Aofei Temple
QbitAI | WeChat public account QbitAI
Remarkable! A Chinese AI has completed its first large-scale empirical lung cancer study, achieving a major breakthrough in the challenge against the “number one cancer killer”:
Across 40 complex lung cancer cases, under a fully standardized evaluation, AI versus doctors: 88 to 12.
In clinical communication, that is almost a commanding lead.
The 250 doctors who participated in the experiment came from top-tier hospitals across China.
Most were chief or associate chief physicians, the kind whose appointments are usually impossible to get... Their credentials speak for themselves.
The study was jointly conducted by the Chinese medical AI company “Future Doctor” and a thoracic surgery team from a leading top-tier hospital in China. It placed medical AI directly in one of the clinical settings that most tests comprehensive judgment:
Complex tumors, treatment options that constrain one another, and no single correct answer.
The system behind this was Future Doctor's medical large language model MedGPT (hereafter, Future Doctor MedGPT).
It showed clear advantages over leading general-purpose international models such as GPT-5.4, Claude Opus 4.7 and Gemini 3.1 Pro; the changes in human-AI collaboration were equally concrete.
With assistance from Future Doctor MedGPT, doctors' overall scores rose from 53.40 to 66.27, a relative increase of 24.1%.
Its performance also drew praise from nearly 10 leading frontline clinicians, including experts who chair committees of the Chinese Medical Association and leading American oncology experts:
In China, several experts who chair committees of the Chinese Medical Association, including Li Yin, director of the esophageal disease ward in thoracic surgery at the Cancer Hospital of the Chinese Academy of Medical Sciences; Wang Yongsheng, director of the Breast Disease Center at Shandong Cancer Hospital; and Chen Longqi, head of thoracic surgery at West China Hospital of Sichuan University, recognized its practical clinical value. They said this is the medical AI frontline doctors urgently need.
Leading experts including Robert Warren, director of surgical oncology at UCSF Health; Hope S. Rugo, director of breast medical oncology at the US City of Hope National Medical Center; Mark Pomper, described as a leading prostate cancer radiotherapy expert in the Miami area; and Marc Shuman, emeritus director of hematologic oncology at the University of California, San Francisco, also spoke out, affirming the study's value to the field and its potential for practical use.
For the industry, this lung cancer experiment offers empirical evidence from China that can inform discussion of medical AI's ability to participate in complex diagnosis and treatment.
From theory to practice, AI is opening up new possibilities for cancer treatment for more people.
250 doctors from top-tier hospitals take on the gray areas of lung cancer care
Frankly, “conquering cancer” has long been one of the most closely watched directions for AI in medicine.
Over the past few years, AI breakthroughs in related fields have extended into protein structure prediction, genetic research and drug development.
But before it can reach patients, a more specific clinical question remains:
How should they be treated?
That is especially true for major diseases such as lung cancer. According to the World Health Organization, lung cancer is the leading cause of cancer death worldwide and caused approximately 1.9 million deaths in 2024.
Nearly 1 in every 5 cancer deaths worldwide is due to lung cancer.
In China, the burden is even more concentrated: lung cancer has ranked first among malignant tumors in both incidence and mortality for many years running.
In actual care, patients' circumstances vary widely. The suitability of the same treatment changes with a patient's organ function, coexisting conditions and prior treatments, placing great demands on doctors' and AI's ability to adapt.
Professor Liang, a noted thoracic surgery expert who led this experiment, has often faced such challenges in his long clinical career.
In his view, treating lung cancer in gray areas requires applying evidence-based medicine and cutting-edge research to each patient's individual circumstances to find a reasonable treatment plan. His team's collaboration with Future Doctor was intended to organize these clinical judgments into a decision process in which AI can participate and doctors can review the results.
The study selected 40 previously unpublished cases from actual clinical practice. All had gray-area characteristics, such as unclear guideline recommendations or conflicting constraints from coexisting conditions, so the specific treatment path required weighing multiple factors together.
The comparison group was also carefully designed.
The 250 participating doctors came from 98 standardized centers across 27 provincial-level regions of China and worked in thoracic surgery, respiratory medicine or oncology. Of them, 159, or 63.6%, were chief or associate chief physicians.
This allowed clinical experience and treatment perspectives from different regions to enter the evaluation, giving the experiment a fuller representation of China's diverse clinical practices.
Each doctor then completed 3 different cases, respectively without assistance, with search references and with support from Future Doctor MedGPT, yielding 2250 written strategies in total. The study used these to observe differences in how doctors handled similar complex lung cancer problems.
First, the human-versus-AI comparison.
Compared with 750 treatment plans submitted by doctors working without assistance, Future Doctor MedGPT received a higher score 660 times, beating the 250 doctors from top-tier hospitals by 88 to 12.
Its standardized responses were also, on average, more than 40% shorter than the doctors' responses, conveying more complete clinical information in fewer words.
In the human-AI collaboration comparison, doctors' overall scores also improved significantly.
With assistance from Future Doctor MedGPT, doctors considered candidate paths more comprehensively and focused more closely on key judgments and risk assessments when selecting a path.
In a comparison with general-purpose large language models under the same conditions, Future Doctor MedGPT scored 96.22 overall, far surpassing GPT-5.4 (71.38), Claude Opus 4.7 (61.81) and Gemini 3.1 Pro (54.62).
Its advantage lies in generating candidate treatment paths. General-purpose models typically provide a single treatment plan, while Future Doctor MedGPT is more inclined to propose several reasonable alternatives at once, along with the conditions for entering or exiting each path.
This is the conditional, multiple-path strategy at the core of Future Doctor MedGPT's architecture, discussed in more detail below.
To ensure the experimental results were credible, the research team also established scoring criteria in advance.
Through a 3-round process, 20 multidisciplinary experts developed 240 case evaluation rules, specifying for each case the clinical issues to address, the information needed before treatment and the available paths. Doctors' follow-up reviews were also included in the evaluation system.
Of these, 40 rules defined safety boundaries for treatment eligibility and intensity; breaching one meant an automatic loss. Professor Liang described the reasoning this way:
Because in medical practice, safety often takes priority over efficacy.
Because in medical practice, safety often takes priority over efficacy.
After the rules were fixed, 3 AI judges—GPT-5.5, Claude Opus 4.7 and Gemini 3.1 Pro—scored the answers independently, with their sources concealed throughout the review.
To be useful for subsequent clinical discussion, a final strategy with a high score had to clearly explain both the basis for its treatment choices and the conditions for carrying them out.
Consult first, then simulate: A look inside Future Doctor MedGPT's technology
The clinical AI agent system used in this study was Future Doctor MedGPT.
Future Doctor MedGPT is a medical large language model designed from the outset for clinical use. It is also China's first approved generative AI model algorithm for disease diagnosis and treatment recommendations.
The 1.0 version mainly handled general medical consultations, controlling the safety and effectiveness of model outputs; the newly upgraded 2.0 version tackles complex treatment decisions more thoroughly.
Compared with general-purpose models and medical search tools, Future Doctor MedGPT can integrate cutting-edge medical evidence, organizing candidate paths, eligibility conditions and responses to risks into a more complete treatment strategy.
The system's design starts with a multidisciplinary consultation.
Take this 68-year-old lung cancer patient, for example. The case is complex: a recurrence after surgery is suspected, and the patient also has interstitial lung disease. The doctor's dilemma is whether to treat the local lesion while fearing that radiotherapy could severely damage lungs that are already fragile.
After the patient's medical record enters Future Doctor MedGPT, agents representing multiple specialties first analyze the patient's problems separately. They then compile points of disagreement that need further assessment for later simulation and verification.
The system then compiles the case, arranging the medical history and test results chronologically, clarifying the patient's current condition and setting constraints according to the treatment goals.
Assessing the diagnosis before proceeding with treatment also follows real clinical practice.
On that basis, the system generates multiple sets of candidate paths in sequence:
Standard path E-Path: Generated from guidelines, consensus statements and drug labels, this option follows conventional standards;
Frontier path F-Path: Combining the latest research and conference findings, it generates diverse evidence-supported options for the patient.
This step depends heavily on the granularity of the evidence. Future Doctor MedGPT's lung cancer evidence database currently contains approximately 13,000 original documents, yielding 201,000 clinical evidence units (CEUs).
Each CEU retains information such as the applicable patient population and treatment circumstances. The system matches these against the patient's background item by item, and doctors can use a CEU at any time to retrieve the original research and check the scope of its conclusions.
After forming E-Path and F-Path, the system also retrieves and checks supplementary evidence. For example, when choosing an F-Path, it must identify what evidence is missing and what additional information is needed to determine whether that path is feasible.
Future Doctor MedGPT searches databases for points of dispute and gaps in evidence, then uses world-model simulation to examine each candidate path in light of the patient's circumstances.
Starting from the patient's current condition, it predicts the treatment process ahead:
If it encounters a risk that can be addressed with supporting evidence, it adjusts the path after gathering that evidence. If a risk cannot be resolved, it lowers the corresponding path's priority or removes it, then checks the remaining options.
Through repeated simulations, the system gradually narrows the options to a strategy that can be reviewed and creates a V-Path verification path specifying the tests and judgments needed next.
The entire process repeats until the existing evidence yields no new actionable path.
What the doctor ultimately receives is a report that can be checked, including the main current treatment paths, alternatives and implementation arrangements.
Each path also includes an analysis of patient benefit, drawing on study data such as overall survival (OS) and progression-free survival (PFS) to help doctors compare the options comprehensively.
AI mainly handles multidisciplinary analysis, evidence matching and path verification. Once doctors receive the material, they can review the key conditions affecting the choice, then make an individualized judgment based on drug availability, the patient's wishes and family circumstances.
This approach also builds on the team's earlier research.
The CSEDB clinical safety and efficacy dual-track benchmark, published in the Nature portfolio journal npj Digital Medicine, had already incorporated medication safety, recognition of critical illness and optimization of treatment paths into its evaluation framework.
This study goes further into the gray areas of lung cancer care, seeking to ensure that the new generation of medical AI gives equal weight to safety and effectiveness when helping develop complex strategies.
Medical AI expands the scope of doctors' capabilities
From this perspective, it is easier to understand why experts in China and the US have collectively focused on this experiment:
Medical AI is at a pivotal point in its shift from simple medical questions and answers to complex clinical decision-making. By focusing on the gray areas of lung cancer care, Future Doctor MedGPT offers a concrete case supporting that progress.
It demonstrates an advance in technical capability and addresses medical AI's most urgent clinical need today.
Can AI be used? To what extent? How reliable is it?
The results are clear: medical AI represented by Future Doctor MedGPT has begun to show potential for use in clinical diagnosis and treatment.
For doctors, this means more complete information to guide decisions. AI assists first, doctors make their judgments afterward, and together they select and carry out the plan.
Beyond established paths, such a system may also be more capable of exploring entirely new approaches to complex diseases such as cancer.
As these ideas are gradually validated, humanity may gain the chance to prevail in its prolonged struggle against disease.
In the words of Professor Wang Yongsheng, director of the Breast Disease Center at Shandong Cancer Hospital:
Future Doctor MedGPT is a medical AI that can be of great help to clinical oncologists.
Future Doctor MedGPT is a medical AI that can be of great help to clinical oncologists.
Beyond its practical value to clinicians, Dr. Robert Warren, director of surgical oncology at the University of California, San Francisco (UCSF) Medical Center, also sees the model's potential beneath the surface:
This is an important study! AI's role in medical diagnosis and treatment decisions is evolving and has the potential to provide important informational support to patients and caregivers.
This is an important study! AI's role in medical diagnosis and treatment decisions is evolving and has the potential to provide important informational support to patients and caregivers.
In response to the specific experimental results, medical experts from China and the United States expressed their support for AI's involvement in diagnostic and treatment decisions, as well as interest in exploring potential collaboration.
Future Doctor has thus become a closely watched Chinese representative in this round of clinical exploration of medical AI.
In fact, enhancing doctors' capabilities has always been Future Doctor's starting point for investing in AI.
Founder and CEO Wang Shirui has long used the phrase “carry on lost learning and bring about peace” to explain the company's approach to medical AGI.
Drawing on the ideas in Zhang Zai's Four Sentences of Hengqu, he hopes to replicate the medical capabilities accumulated by top doctors on a broad scale, bring high-quality resources to more patients, and involve AI in the study of difficult diseases to drive the continued advancement of medical knowledge.
Both of these goals found concrete expression in this joint experiment.
The study organized past expert experience into evaluation criteria, giving AI a reasoned basis to follow. In other words, this is also a way to disseminate high-quality medical expertise.
Seeking candidate approaches in the gray areas of complex lung cancer cases and testing AI's ability to support difficult decisions is precisely an empirical exploration by Future Doctor guided by the idea of “bringing about peace.”
Guided by this idea, subsequent research and development will remain closely tied to clinical work. As medical records are updated, the system needs to keep up with changes in patients' conditions; questions doctors raise as they review and use it can, in turn, become the starting point for the next round of research.
The wheels may not be turning fast enough yet, but at least, as of today, they have been built.
The 88:12 score shows progress, while the 24.1% increase in scores provides a more concrete measure of the value of human-AI collaboration.
Future Doctor's next step is to have this approach widely validated across more clinical tasks.