If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Address for reprints: Amir H. Sadeghi, MD, MSc, Department of Cardiothoracic Surgery, Erasmus University Medical Center, Room Rg-635, PO Box 2040, 3015 GD Rotterdam, The Netherlands.
There has been an increasing trend toward pulmonary segmentectomies to treat early-stage lung cancer, small intrapulmonary metastases, and localized benign pathology. A complete preoperative understanding of pulmonary anatomy is essential for accurate surgical planning and case selection. Identifying intersegmental divisions is extremely difficult when performed on computed tomography. For the preoperative planning of segmentectomies, virtual reality (VR) and artificial intelligence could allow 3-dimensional visualization of the complex anatomy of pulmonary segmental divisions, vascular arborization, and bronchial anatomy. This technology can be applied by surgeons preoperatively to gain better insight into a patient's anatomy for planning segmentectomy.
Methods
In this prospective observational pilot study, we aim to assess and demonstrate the technical feasibility and clinical applicability of the first dedicated artificial intelligence-based and immersive 3-dimensional-VR platform (PulmoVR; jointly developed and manufactured by Department of Cardiothoracic Surgery [Erasmus Medical Center, Rotterdam, The Netherlands], MedicalVR [Amsterdam, The Netherlands], EVOCS Medical Image Communication [Fysicon BV, Oss, The Netherlands], and Thirona [Nijmegen, The Netherlands]) for preoperative planning of video-assisted thoracoscopic segmentectomies.
Results
A total of 10 eligible patients for segmentectomy were included in this study after referral through the institutional thoracic oncology multidisciplinary team. PulmoVR was successfully applied as a supplementary imaging tool to perform video-assisted thoracoscopic segmentectomies. In 40% of the cases, the surgical strategy was adjusted due to the 3-dimensional-VR–based evaluation of anatomy. This underlines the potential benefit of additional VR-guided planning of segmentectomy for both surgeon and patient.
Conclusions
Our study demonstrates the successful development and clinical application of the first dedicated artificial intelligence and VR platform for the planning of pulmonary segmentectomy. This is the first study that shows an immersive virtual reality-based application for preoperative planning of segmentectomy to the best of our knowledge.
PulmoVR, a novel and immersive virtual reality-based application, is a feasible and clinically applicable method for accurate and patient-tailored surgical planning of pulmonary segmentectomy.
We present the development and application of an immersive virtual reality-based platform (PulmoVR) that enables personalized surgical planning for pulmonary segmentectomies. PulmoVR is the first dedicated virtual reality segmentectomy planning tool that provides accurate surgical planning for segmentectomy. Future clinical studies should demonstrate clinical benefits of such technology.
See Commentary on page 322.
Due to an increase in the frequency of small ground-glass opacity and incidental small intrapulmonary metastases detection, there has been an increasing trend toward lung-sparing resection techniques such as segmentectomy.
Video-assisted thoracoscopic surgery is a safe and effective alternative to thoracotomy for anatomical segmentectomy in patients with clinical stage I non–small cell lung cancer.
In our hospital, we have adopted segmentectomy for specific surgical procedures, including resections of small (≤2 cm) stage I non–small cell lung cancer (NSCLC), benign intrapulmonary disease limited to segments (eg, bronchiectasis), and deeply located or nonpalpable metastases of other primary malignancies.
Thoracoscopic lung segmentectomies are generally more complex than thoracoscopic lobectomies.
Identifying intraparenchymal planes and ligation of target branches of pulmonary arteries, veins, and bronchi can be more challenging, partly due to the existence of many anatomical variations and abnormalities in pulmonary vascular anatomy.
Conventional computed tomography (CT) imaging is the gold standard for preoperative planning of anatomical (sub)lobar resections. Unfortunately, the identification of segmental borders and segmental branches of arteries, veins, and bronchi is challenging when conventional 2-dimensional (2D) CT is used.
The success of pulmonary segmentectomies largely depends on the surgeon's preoperative understanding of the anatomy. Previous studies have already demonstrated the feasibility and added value of 3-dimensional (3D) reconstructions of pulmonary anatomy for preoperative and intraoperative planning of pulmonary segment resections.
Interactive 3D reconstruction of pulmonary anatomy for preoperative planning, virtual simulation, and intraoperative guiding in video-assisted thoracoscopic lung surgery.
Over the past few years, virtual reality (VR) devices and software are gaining popularity and have been shown to improve the understanding of patient anatomy by surgeons.
Effect of 3-dimensional virtual reality models for surgical planning of robotic-assisted partial nephrectomy on surgical outcomes: a randomized clinical trial.
A VR platform has more features and functions than existing 2D or 3D planning software, including immersive and interactive manipulation, realistic in-depth perception, and visualization of complex relationships of anatomic structures that are ready at hand and can be applied by surgeons to gain a more realistic insight into a patient's anatomy.
Effect of 3-dimensional virtual reality models for surgical planning of robotic-assisted partial nephrectomy on surgical outcomes: a randomized clinical trial.
Moreover, automated imaging algorithms could potentially create a more efficient planning by enabling automatic visualization of anatomic structures of interest.
Artificial intelligence-based fully automated per lobe segmentation and emphysema-quantification based on chest computed tomography compared with global initiative for chronic obstructive lung disease severity of smokers.
By developing an artificial intelligence (AI)-based and immersive 3D-VR platform as a supplementary preoperative planning tool to traditional CT imaging, a novelty can be added to the thoracic surgeon's armamentarium.
To overcome most of the shortcomings of the currently available imaging modalities and to create an easy-to-use and dedicated platform for CT imaging review, we have developed a 3D VR-based digital imaging and communication in medicine (DICOM) viewer. In addition, we developed PulmoVR (jointly developed and manufactured by Department of Cardiothoracic Surgery [Erasmus Medical Center], MedicalVR, EVOCS Medical Image Communication [Fysicon BV], and Thirona), a dedicated immersive 3D-VR and AI-based segmentectomy surgical planning tool. We hypothesized that such a platform could provide a more accurate preoperative understanding of the patient-specific segmental and intersegmental anatomy.
The present prospective single-center pilot study shows our early results on the clinical application of our innovative immersive AI and VR-based segmentectomy planning tool (PulmoVR) in 10 consecutive patients. The main objectives are to assess the technical feasibility and clinical applicability of PulmoVR in performing 3D VR-guided video-assisted thoracoscopic surgery (VATS) segmentectomies.
Methods
Patient Selection
The medical ethical committee of the Erasmus Medical Center approved this study (MEC-2020-0702). A total number of 10 consecutive patients who were eligible and accepted for pulmonary segmentectomy at the Erasmus Medical Center, Rotterdam, The Netherlands, were enrolled in this prospective study. All patients were consecutively included after referral and confirmation of eligibility for segmentectomy by the institutional local thoracic oncology multidisciplinary team meeting (Figure 1, A). After approval of a patient's eligibility for surgery, patients were planned for VATS segmentectomy. Subsequently, written informed consent was obtained from all patients and patients were included prospectively for VR-guided planning of segmentectomy as a supplement to conventional CT-guided planning. The inclusion criteria were as follows: age >18 years and pulmonary pathology suitable for pulmonary segmentectomy (ie, confirmed/clinical suspicion of stage I NSCLC ≤2 cm, limited benign lesion, and intrapulmonary metastases of other primary malignancies).
Figure 1Schematic representation of virtual reality-based preoperative and intraoperative clinical workflow. A and B, After referral through our local thoracic oncology multidisciplinary team meeting a patient's anatomy is reviewed (by a surgeon) and the target segment is selected on conventional 2-dimensional-computed tomography (2D-CT) scan images. C and D, Digital imaging and communication in medicine (DICOM) files of CT scans are extracted from the local patient archiving and communication system (PACS) and reviewed in a 3-dimensional (3D) virtual reality (VR)-based DICOM viewer. E and F, Within this virtual environment, the surgeon can review grayscale patient-specific lung anatomy in 3D within a virtual reality digital room. After that, anonymized DICOM files are transferred for segmentation by an artificial intelligence (AI)-based algorithm. This algorithm automatically segments (ie, highlights) anatomic structures including arteries, veins, bronchi, and segments within a CT scan. G, Subsequently, DICOM files together with the segmentation files can be loaded into PulmoVR software (jointly developed and manufactured by Department of Cardiothoracic Surgery [Erasmus Medical Center], MedicalVR, EVOCS Medical Image Communication [Fysicon BV], and Thirona) on a VR workstation and 3D in-color VR-based images can be evaluated by the surgeon for preoperative planning of segmentectomy. H, During 3D video-assisted thoracoscopic surgery (VATS), static 3D reconstructions can be presented on monitors in the operating room to provide intraoperative guidance. MDT, Multidisciplinary team meeting.
Two of our dedicated lung surgeons and a surgery resident physician participated in the study. All participants received a brief (∼15 minutes) audiovisual tutorial on how to use the software and hardware before the start of the study.
VR- and AI-based Surgical Planning Modalities
An immersive VR-based CT (grayscale) DICOM viewer was developed together with MedicalVR.
In addition, a dedicated 3D-VR and AI-based pulmonary segmentectomy planning tool called PulmoVR was successfully developed together with MedicalVR, EVOCS Medical Image Communication (Fysicon BV), and Thirona. Segmental anatomy was determined automatically using LungQ software (Thirona). The AI-based delineation of pulmonary segments was based on determination of the bronchial and arterial intersegmental planes. Based on this delineation, the intersegmental veins were located within the border of segments. The anatomical 3D segmentations were visually checked by trained analysts for accuracy and corrected, as needed, within Thirona's ISO 13485-certified image analysis service. There were no specific technical requirements for 3D immersive evaluation of CT scans in VR besides a maximum of 1000 CT images and at least an adequate image quality for 2D evaluation. The AI analysis was possible on both contrast and noncontrast-enhanced inspiratory thoracic CT scans with a maximum slice thickness of 1.5 mm.
Preoperative Planning Workflow
A schematic overview of the preoperative planning workflow is outlined in Figure 1 and Figure E1. After referral by our local multidisciplinary team (Figure 1, A), a surgeon independently identified and documented the target segment(s) on preoperative diagnostic CT scans. (Figure 1, B). Subsequently, the DICOM files were extracted from the patient archiving and communication system (Figure 1, C) and VR-based evaluation was carried out by a cardiothoracic surgery resident physician who was blinded to the study to independently evaluate the anatomy in our 3D-VR DICOM viewer (Figure 1, D). In this phase, the VR-based images were only a grayscale, 3D volume rendering of the CT scan without postprocessing edits (Figure 1, D, and Figure 2, A). The resident physician identified the lesion first, after which he identified the target segment based on anatomical landmarks (eg, intersegmental vein). Next, a surgeon also reviewed the anatomy in 3D-VR (Figure E1). The DICOM files were then transferred to Thirona and the pulmonary arteries, veins, segments, and the bronchi were segmented using LungQ (Thirona) AI-based lung quantification software (Figure 2, F). These analyses were transferred to our hospital within 72 hours. Then, PulmoVR enabled 3D visualization of both CT images and AI-based segmentation (segmentation means digital labeling of anatomic structures) files in immersive VR (Figure 2, G). Reconstructions were created for all patients undergoing segmentectomy and the target segment and patient-specific anatomy was evaluated in PulmoVR for the final time by the surgeon (Figure 2, G).
Figure 2Evaluation of 3-dimensional (3D) reconstructions of pulmonary segmental anatomy in 3D virtual reality (VR) and PulmoVR (jointly developed and manufactured by Department of Cardiothoracic Surgery [Erasmus Medical Center], MedicalVR, EVOCS Medical Image Communication [Fysicon BV], and Thirona) for preoperative planning. A, A presentation of grayscale 3D-VR images from three different angles and cut planes. These images represent screenshots of the 3D models in virtual reality. B, PulmoVR views of a patient's segmental anatomy in which the tumor (yellow), bronchi (green), and segmental arteries (blue)/veins (red) of the right lower lobe are highlighted in different colors. C, PulmoVR-views of the same patient with segmentation of all right lower lobe anatomical segments (S); S6 (orange), S7 (red), S8 (white), S9 (purple), S10 (pink), bronchi (green), and tumor (yellow). Middle panel, Lateral view demonstrates the location of the tumor at the intersection of S6 (orange), S9 (purple), and S10 (pink). Note: Quality and in-depth perception are significantly higher when viewed in VR.
During surgery, a VR workstation was located in the operating theatre and provided an additional review of the anatomy when necessary. Moreover, preoperatively acquired 3D reconstructions of anatomy were displayed on operating theatre monitors intraoperatively. (Figure 2, H).
Surgical Technique
Two dedicated cardiothoracic surgeons performed the 3D VATS segmentectomies independently (6 and 4 each). Surgical assistance was provided by a resident physician. All patients underwent 3D 3-port VATS under general anesthesia and double-lumen intubation and single lung ventilation were applied. Intraoperative intersegmental border demarcation was achieved using intravenous indocyanine green as described previously by Iizuka and colleagues.
Transection of parenchyma was performed with endostaplers.
Clinical End Point and Qualitative Assessment
All data were analyzed using Microsoft Office Excel 2015 (Microsoft, Redmond, Wash). This study aimed to study technical feasibility on the development of an AI- and VR-based tool for preoperative planning and performance of segmentectomy. Technical feasibility is defined as the ability to create an AI-based segmentation of patient-specific CT scans and to subsequently review pulmonary arteries, veins, bronchi, and segmental planes in an immersive VR application. The following clinical results were selected to serve as a clinical end point to demonstrate potential clinical safety: success of segmentectomy, resection margin (radical R0 resection), and intraoperative blood loss.
The participating cardiothoracic surgeons were asked to fill out a questionnaire (Appendix E1) after each procedure to assess intraoperative anatomy correspondence with their VR-based planning images. Additional questionnaires were filled out by the surgeons to evaluate the ease of use, satisfaction, and attitude toward future use of VR-guided surgical planning with PulmoVR (Appendix E2). These qualitative results served only as a subjective measurement to illustrate the potential usability of VR-based surgical planning.
Results
Patient Characteristics
A total of 10 consecutive patients (5 men and 5 women; mean age, 51 ± 13.4 years) were referred for elective segmentectomy and were enrolled in this study (Table 1). All baseline patient characteristics, pathology results, and target segments are outlined in Table 1.
Table 1Clinical patient characteristics, surgical strategy, and perioperative outcome
Patient
Sex
Age (y)
Indication for segmentectomy
Target segment and surgical plan (after 2D-CT evaluation)
Adjusted target segment and surgical plan after 3D evaluation
Confirmation of target segment and surgical plan after review in PulmoVR
PulmoVR was jointly developed and manufactured by Department of Cardiothoracic Surgery [Erasmus Medical Center], MedicalVR, EVOCS Medical Image Communication (Fysicon BV), and Thirona.
Final pathology
Resection margin (cm)
Tumor size (cm)
Intraoperative blood loss (mL)
1
Male
71
Metastasis (colorectal carcinoma)
RLL S9 (lateral basal segment)
RLL S9 + S10 + en-bloc wedge resection S6 (lateral, posterior and superior segment)
✓
Colorectal carcinoma metastasis
1.5 (R0)
5
100
2
Female
65
Primary NSCLC (adenocarcinoma)
RLL S6 (superior segment)
N/A
✓
Adenocarcinoma
1 (R0)
1.2
75
3
Female
64
Primary NSCLC (clinical suspicion)
LUL S1 (apical segment)
N/A
✓
Adenocarcinoma
1.7 (R0)
1
600
4
Male
56
Suspicious for metastasis (hypopharyngeal carcinoma)
Suspicious for metastasis/loco-regional recurrence of stage IV NSCLC
LUL S4+5 (superior and inferior lingular segment)
LUL S5 (inferior lingular segment)
✓
Necrotizing granulomatous inflammation
N/A
N/A
0
10
Male
40
Suspicious for metastasis/loco-regional recurrence of stage IV NSCLC
LUL S1-3 (apical, posterior, and anterior segment)
N/A
✓
Adenocarcinoma
2.5 (R0)
2.8
200
2D-CT, 2-Dimensional computed tomography; 3D, 3-dimensional; RLL, right lower lobe; NSCLS, non–small cell lung cancer; N/A, not available; LUL, left upper lobe; RUL, right upper lobe; LLL, left lower lobe.
∗ PulmoVR was jointly developed and manufactured by Department of Cardiothoracic Surgery [Erasmus Medical Center], MedicalVR, EVOCS Medical Image Communication (Fysicon BV), and Thirona.
Technical Feasibility: 3D-VR and PulmoVR Reconstructions
In all patients, VR reconstructions were made successfully. As demonstrated in Figures 2 and 3, for all patients, the segmental arteries, segmental veins, bronchi, and segmental borders were visualized in PulmoVR. Segmentations of vasculature and segmental borders were made only in the lobe of interest (Figures 2 and 3). All other thoracic structures were automatically visualized in grayscale as well. Based on the questionnaire that was filled out by the surgeons postoperatively after each case (Appendix E1), in 100% of the cases, intraoperative anatomy corresponded entirely with preoperative VR images. Videos 1 and 2 demonstrate the VR-guided evaluation of the intrapulmonary segmental anatomy from various perspectives in 2 different cases.
Figure 3Selective segmentation and visualization of patient-specific anatomy in PulmoVR (jointly developed and manufactured by Department of Cardiothoracic Surgery [Erasmus Medical Center], MedicalVR, EVOCS Medical Image Communication [Fysicon BV], and Thirona). A, All segmental bronchi and the tumor (pink) are highlighted in different colors (top and bottom line) and shown from 3 different angles (front, lateral, and posterior). B, The pulmonary segments are translucent and the tumor (yellow in middle and pink in top and bottom line) extension through segmental borders can be seen when viewed from different angles. C, Pulmonary artery segmental branches are highlighted in blue to evaluate the relation of segmental arteries with the tumor (pink). Note: Quality and in-depth perception are much higher when viewed in virtual reality.
Clinical Applicability: Perioperative Outcomes and Surgical Adjustments
All segmentectomy procedures were performed successfully through 3D VATS, assisted by preoperative VR planning. A complete resection (ie, R0) with adequate surgical margins was obtained in all cases (Table 1).
In 4 out of 10 cases, the VR-based evaluation resulted in a change in the selection of the target segment than the initially selected target segment by 2D-CT. We will briefly highlight these 4 cases. For all operations, initial surgical strategies and changes after review in VR are outlined in Table 1.
Patient 1
A patient with colorectal metastasis in the right lower lobe was referred for segmentectomy of segment (S) 9. In this patient VR evaluation showed tumor expansion through the segmental borders of S6, S9, and S10. Consequently, surgical strategy after evaluation in VR was adjusted accordingly and en-bloc segmentectomy of S9 and S10 and a subsegmentectomy of S6 was performed (see Figures 2 and 3 for VR images of this patient).
Patient 4
A patient with a tumor in the apico-dorsal part of S1 of the right upper lobe was referred for segmentectomy of S1. VR evaluation showed the relatively small tumor margin in relation with the S1/S2 border (Figure 4, A and B), which was not visible on preoperative CT (Figure 4, D). Consequently, the patient underwent an en-bloc resection of S1 and a subsegment of S2 to maintain safe surgical margins. Intraoperatively obtained indocyanine green-guided intersegmental border visualization corresponded with the preoperatively acquired VR reconstructions (Figure 4, C).
Figure 4Two representative cases of virtual reality (VR)-adjusted segmentectomy planning. A and B, Two snapshots of the VR-rendered images in a patient with a tumor in segment (S) 1. The tumor (yellow arrow) and its small margins to S2 are visualized (red dotted line). C, Intraoperative screenshot of video-assisted thoracoscopic surgery (VATS) during indocyanine green (ICG)-based demarcation of the segmental border (red dotted line) between S1 and S2, showing correspondence with the VR images in A and B. D, Representative snapshots of transversal and sagittal computed tomography scan of patient 4. E and F, Two snapshots of patient 7 with a tumor (green arrow) in S8. The close relation of the tumor with the lower part of S6 can be seen in panel F. G, Intraoperative screenshot of VATS after ligation of segmental artery (A) 8. H, Representative snapshots of transversal and sagittal computed tomography scan of patient 7. B8, Segmental bronchus 8.
Patient 7 had a history of pulmonary metastasized mixed germ cell tumor for which he underwent wedge resection of the right lower lobe during September 2019. During follow-up, there was a new pulmonary lesion (suspicious recurrent metastasis) for which he received chemotherapy. Chemotherapy did not fully resolve the lesion and the patient was accepted for redo VATS segmentectomy for the cystic lesion. After review in CT, the lesion was believed to be located in S6 (Figure 4, H). However, after evaluation in VR, the tumor seemed to be located clearly in S8 (Figure 4, E and F). A successful segmentectomy of S8 was performed (Figure 4, G).
Patient 9
A patient with a tumor in the lingula (suspicious for metastasis/loco-regional recurrence of stage IV NSCLC) based on CT images was referred for lingulectomy (S4 and S5). However, VR evaluation showed that the lesion was located entirely within S5 and consequently resulted in a successful segmentectomy of S5.
Surgeons’ Experiences
Table 2 depicts the experience of the surgeons. Overall, PulmoVR was rated to be easy to use, useful, and efficient for preoperative planning of segmentectomy.
Table 2Questionnaire results on the virtual reality experience of the participating surgeons
PulmoVR was jointly developed and manufactured by Department of Cardiothoracic Surgery, MedicalVR, EVOCS Medical Image Communication (Fysicon BV), and Thirona.
was easy
5
4
2. Rotating the CT scan in PulmoVR was easy
5
4
3. Cutting (slicing in 3D) through the CT scan was easy
5
4
4. Reviewing the CT scan from 360° was easy
5
4
5. Selectively reviewing the segmental anatomy (eg, arteries, veins, bronchi, and segmental borders) was easy
4
4
Usefulness and efficiency of PulmoVR
1. PulmoVR helps me to review segmental anatomy more accurately compared to reviewing CT scans in 2D
5
5
2. The assessment pulmonary segmental anatomy in PulmoVR takes less time than conventional CT scan review
4
5
3. Segmentation (highlighting arteries, veins, and bronchi in color) of the CT scans in PulmoVR is of added value for CT assessment
5
5
4. PulmoVR helps me to prepare a surgical strategy more efficiently compared with conventional methods (eg, CT scans).
5
5
5. PulmoVR review of CT scans creates more awareness of spatial orientation of anatomical structures and therefore contributes to the safer performance of segmentectomies
5
5
Attitude toward future use of PulmoVR in preoperative planning of segmentectomy
1. I would like to work with PulmoVR in the future
5
5
2. The future, I prefer using PulmoVR for preoperative planning as a supplement to conventional methods
5
5
3. In the future, I prefer using PulmoVR for preoperative planning instead of conventional methods
2
1
4. I believe, in the future, PulmoVR-like technologies will become a standard of care imaging modality for preoperative planning of segmentectomies
5
5
5. I would recommend PulmoVR as a preoperative planning tool to my colleagues
∗ Scale was 1 = strongly disagree to 5 = strongly agree.
† PulmoVR was jointly developed and manufactured by Department of Cardiothoracic Surgery, MedicalVR, EVOCS Medical Image Communication (Fysicon BV), and Thirona.
In general, pulmonary segmentectomies are considered technically challenging and difficult procedures, especially when compared with lobectomy or other sublobar resections, such as wedge resection. Several factors have been identified that contribute to the technical difficulties of segmentectomy, including understanding of the individual pulmonary anatomy and its relation to the tumor localization. Accurate evaluation of segmental pulmonary anatomy remains a challenge when assessed on 2D-CT images. For this reason, we codeveloped an innovative AI and immersive VR-based segmentectomy planning tool (the PulmoVR). In this study we report our early results on the technical feasibility and clinical applicability of our PulmoVR application in patients undergoing VATS segmentectomy.
Recent studies have explored and highlighted the role of 3D-CT angiography and manual 3D virtual modeling of CT images to examine patient-specific (segmental) anatomy preoperatively.
Moreover, reports from other studies that have used custom-made 3D software to simulate a segmentectomy procedure have been published and underline the benefits that 3D modeling offers in the context of segmentectomy planning.
An important disadvantage of 3D modeling is the loss of data because most often these applications focus on the visualization of specifically selected and requested anatomic structures such as pulmonary vessels, tumor, bronchi, and parenchyma.
As a result, information on intrapulmonary lymph nodes, thoracic cage anatomy (eg, intercostal spaces and sternum), the heart, diaphragm, and other intrathoracic vessels is lost or needs to be selected and manually segmented separately. Our 3D-VR DICOM viewer, as described in the current study, enables direct VR visualization by virtually stacking the CT images and allowing evaluation in VR. This provides a fast method (<1 minute) to evaluate patient-specific CT scans in 3D and with reality-like in-depth perception in 360° without the need for postprocessing techniques and without loss of imaging data.
Artificial intelligence-based fully automated per lobe segmentation and emphysema-quantification based on chest computed tomography compared with global initiative for chronic obstructive lung disease severity of smokers.
Furthermore, in currently available techniques 3D segmentation (which is the identification and selective coloring of an anatomic structure) is often performed manually (by technicians), which increases the risk for error, requires manual labor, is time-consuming, and for this reason is more expensive.
Another shortcoming of the available 3D modeling software is that it commonly requires contrast-enhanced CT scanning for accurate segmentation, which results in radiation and iodine-contrast agent exposure and consequently in additional health risks for patients.
In this study, we have demonstrated that our PulmoVR application provides a unique environment to review patient-specific segmental anatomy by facilitating selective AI-based automatic labeling, visualization, and assessment of pulmonary structures, without the need for contrast-enhanced CT imaging. Moreover, automated imaging algorithms could potentially create more efficient planning.
In addition, 3D imaging review on a computer screen remains a 3D visualization on a 2D monitor and requires extensive (and separate) review of pulmonary structures from different angles to understand the spatial relations and in-depth anatomy of structures. Finally, obtaining accurate 3D images in this case also requires additional investment in hardware (3D monitors, high-performance computers, and glasses). In accordance to the reported data from the literature on VR applications, in this study we have demonstrated that our PulmoVR tool enables immersive interactive image manipulation, realistic in-depth perception, and immersive visualization of complex relationships of anatomic structures (eg, vascular, bronchial, tumor margins, and segmental borders) that can be applied by surgeons preoperatively (without the need to consult radiologists or imaging technicians) to gain a better and more realistic insight into a patient's specific anatomy.
Effect of 3-dimensional virtual reality models for surgical planning of robotic-assisted partial nephrectomy on surgical outcomes: a randomized clinical trial.
Additionally, we have showed that PulmoVR tool is easy and ready to use after only a single demonstration and tutorial of 15 minutes as was concluded from the questioners filled out by the participating surgeons. The software can be installed on every computer or laptop with sufficient graphical and processing power and the off-the-shelf VR head mounted displays are widely available. There is no need for additional 3D screens, hardware, or glasses.
Future Outlook
In more comparative research we aim to study the application of PulmoVR in larger patient populations. In addition, our goal is to design a multiuser platform in which a patient's anatomy can be evaluated by more users at the same time. Hopefully, this would also pave the way for the development of surgical simulations for education and training purposes.
Other exciting and futuristic VR applications in surgical planning might be the application of VR or augmented reality to enhance intraoperative views in robotic procedures, such as robotic segmentectomies. In terms of preoperative planning, VR can assess the ideal thoracoscopic port placement locations. Also, the intraoperative 3D view could be merged with preoperatively acquired reconstructions. This could potentially enhance intraoperative navigation and also the identification of very small intrapulmonary lesions, lymph nodes, and small vascular branches. Other interesting examples of VR-based guidance and planning are navigational (ie, VR guided) bronchoscopy or augmented reality guided detection of small and deeply located lesions.
Further development and use of these novel technologies could potentially result in better preoperative diagnostics.
Limitations
We acknowledge that the current pilot study has some limitations in terms of technology and study setup. First, only a small number of patients were included, and only 2 cardiothoracic surgeons have used and evaluated the platform. To draw definite conclusions to structurally implement such technology, more prospective, extensive, and comparative analyses are required. Also, cost-effectiveness studies are important to evaluate implementation in the clinic. In addition, in the current situation, only segmental branches of blood vessels are visualized in PulmoVR. To further improve on the software, the entire pulmonary artery should ideally be segmented to ease evaluation in PulmoVR. Ideally, also (semi)automatic segmentation of intrapulmonary and mediastinal lymph nodes could ease surgical planning and performance of lymph node dissection.
Another important limitation of PulmoVR is the lack of the possibility to simulate lung deflation, manipulation, and virtual resection/transection. As a result, PulmoVR cannot yet be used to create augmented virtual projections of anatomy within the physical surgical field. Future development of augmented reality and digital navigation systems should hopefully overcome this challenge within the next decade.
Conclusions
This study presents the first artificial intelligence and immersive VR-based visualization method for segmentectomy planning (Figure 5). We have demonstrated technical feasibility and clinical applicability in 10 consecutive patients undergoing thoracoscopic segmentectomy. Notably, the application of PulmoVR resulted in essential changes in surgical strategy in 40% of the cases. All segmentectomy procedures were performed successfully, and all oncologic resections adequately. More extensive clinical trials are needed to study safety and clinical benefits and to provide clinical evidence for the application of virtual reality in segmentectomy planning.
Figure 5The identification of segmental anatomy for surgical planning of lung segmentectomy is quite challenging with conventional imaging. In the current study, a novel method is presented to create artificial intelligence based virtual reality reconstructions to prepare for thoracoscopic segmentectomies in 10 consecutive patients. CT, Computed tomography; VR, virtual reality.
Drs Sadeghi and Mahtab are coinventors of the virtual reality-based technology presented in this article. All other authors reported no conflicts of interest. This work was supported by Koers23, Erasmus Medical Center, The Netherlands.
The Journal policy requires editors and reviewers to disclose conflicts of interest and to decline handling or reviewing manuscripts for which they may have a conflict of interest. The editors and reviewers of this article have no conflicts of interest.
Appendix E1
Questionnaire presented to surgeons after each procedure to assess intraoperative anatomy correspondence with their virtual reality-based planning images. PulmoVR was jointly developed and manufactured by Department of Cardiothoracic Surgery (Erasmus Medical Center), MedicalVR, EVOCS Medical Image Communication (Fysicon BV), and Thirona.
Please answer the following questions regarding correspondence of virtual reality-guided planning images and intraoperative anatomy findings:
Pulmonary segmental arteries: Intraoperative findings corresponded with the preoperative PulmoVR images
Agree/Disagree
Pulmonary segmental veins: Intraoperative findings corresponded with the preoperative PulmoVR images
Agree/Disagree
Pulmonary segmental bronchial anatomy: Intraoperative findings corresponded with the preoperative PulmoVR images
Agree/Disagree
Appendix E2
Questionnaire filled out by the surgeons to evaluate the ease of use, satisfaction, and attitude toward future use of virtual reality-guided surgical planning with PulmoVR (jointly developed and manufactured by Department of Cardiothoracic Surgery [Erasmus Medical Center], MedicalVR, EVOCS Medical Image Communication [Fysicon BV], and Thirona). CT, Computed tomography; N/A, not applicable; 3D, 3 dimensional.
PulmoVR review of CT scans creates more awareness of spatial orientation of anatomical structures and therefore contributes to the safer performance of segmentectomies.
Figure E1A schematic workflow on the preoperative workup carried out by the surgical team to prepare for segmentectomy. CT, Computed tomography; VR, virtual reality.
Virtual reality (VR)-based view of a patient-specific scan with a tumor in the right lower lobe. The review of the anatomy is performed by using VR controllers (Oculus Rift S [Oculus VR, Irvine, Calif]) that enable 360° rotation and zoom. Different segments of the right lower lobe are shown transparent in various colors. The bronchial anatomy (green) is also shown. Additionally, measurements can be performed in virtual reality to determine tumor size and expected surgical margins. Video available at: https://www.jtcvs.org/article/S2666-2507(21)00253-4/fulltext.
Virtual reality (VR)-based view of a patient-specific scan with a tumor in the right lower lobe. The review of the anatomy is performed by using VR controllers (Oculus Rift S [Oculus VR, Irvine, Calif]) that enable 360° rotation and zoom. Different segments of the right lower lobe are shown transparent in various colors. The bronchial anatomy (green) is also shown. Additionally, measurements can be performed in virtual reality to determine tumor size and expected surgical margins. Video available at: https://www.jtcvs.org/article/S2666-2507(21)00253-4/fulltext.
Virtual reality (VR) representation of a patient's scan with a tumor in the right upper lobe. Segmental arteries (blue), veins (red), and bronchial anatomy (green) are shown. The evaluation of computed tomography scans can be performed on a computer monitor in 3 dimensions as well as in VR with VR controllers. In addition. our PulmoVR software (jointly developed and manufactured by Department of Cardiothoracic Surgery [Erasmus Medical Center], MedicalVR, EVOCS Medical Image Communication [Fysicon BV], and Thirona) enables to take notes during review of the scan to determine the segmental branches of the arterial supply and venous drainage. Video available at: https://www.jtcvs.org/article/S2666-2507(21)00253-4/fulltext.
Virtual reality (VR) representation of a patient's scan with a tumor in the right upper lobe. Segmental arteries (blue), veins (red), and bronchial anatomy (green) are shown. The evaluation of computed tomography scans can be performed on a computer monitor in 3 dimensions as well as in VR with VR controllers. In addition. our PulmoVR software (jointly developed and manufactured by Department of Cardiothoracic Surgery [Erasmus Medical Center], MedicalVR, EVOCS Medical Image Communication [Fysicon BV], and Thirona) enables to take notes during review of the scan to determine the segmental branches of the arterial supply and venous drainage. Video available at: https://www.jtcvs.org/article/S2666-2507(21)00253-4/fulltext.
Video-assisted thoracoscopic surgery is a safe and effective alternative to thoracotomy for anatomical segmentectomy in patients with clinical stage I non–small cell lung cancer.
Interactive 3D reconstruction of pulmonary anatomy for preoperative planning, virtual simulation, and intraoperative guiding in video-assisted thoracoscopic lung surgery.
Effect of 3-dimensional virtual reality models for surgical planning of robotic-assisted partial nephrectomy on surgical outcomes: a randomized clinical trial.
Artificial intelligence-based fully automated per lobe segmentation and emphysema-quantification based on chest computed tomography compared with global initiative for chronic obstructive lung disease severity of smokers.
The past decade has brought a significant increase in the performance of pulmonary segmentectomies, including “complex” segmentectomies that involve the division of multiple intersegmental planes.1-4 With increasing use of lung cancer screening, this trend is not likely to abate. Successful completion of these operations can be challenging owing not only to their technical complexity but also to the difficulty in accurately identifying the correct anatomic segment on standard preoperative imaging.
00253-4&id=fx1.jpg){kind=link}
00253-4&id=fx2.jpg){kind=link}
00253-4&id=gr1.jpg){kind=link}
00253-4&id=gr2.jpg){kind=link}
00253-4&id=gr3.jpg){kind=link}
00253-4&id=gr4.jpg){kind=link}
00253-4&id=gr5.jpg){kind=link}
00253-4&id=fx5.jpg){kind=link}