Virtual reality and artificial intelligence for 3-dimensional planning of lung segmentectomies

Background There has been an increasing trend toward pulmonary segmentectomies to treat early-stage lung cancer, small intrapulmonary metastases, and localized benign pathology. A complete preoperative understanding of pulmonary anatomy is essential for accurate surgical planning and case selection. Identifying intersegmental divisions is extremely difficult when performed on computed tomography. For the preoperative planning of segmentectomies, virtual reality (VR) and artificial intelligence could allow 3-dimensional visualization of the complex anatomy of pulmonary segmental divisions, vascular arborization, and bronchial anatomy. This technology can be applied by surgeons preoperatively to gain better insight into a patient's anatomy for planning segmentectomy. Methods In this prospective observational pilot study, we aim to assess and demonstrate the technical feasibility and clinical applicability of the first dedicated artificial intelligence-based and immersive 3-dimensional-VR platform (PulmoVR; jointly developed and manufactured by Department of Cardiothoracic Surgery [Erasmus Medical Center, Rotterdam, The Netherlands], MedicalVR [Amsterdam, The Netherlands], EVOCS Medical Image Communication [Fysicon BV, Oss, The Netherlands], and Thirona [Nijmegen, The Netherlands]) for preoperative planning of video-assisted thoracoscopic segmentectomies. Results A total of 10 eligible patients for segmentectomy were included in this study after referral through the institutional thoracic oncology multidisciplinary team. PulmoVR was successfully applied as a supplementary imaging tool to perform video-assisted thoracoscopic segmentectomies. In 40% of the cases, the surgical strategy was adjusted due to the 3-dimensional-VR–based evaluation of anatomy. This underlines the potential benefit of additional VR-guided planning of segmentectomy for both surgeon and patient. Conclusions Our study demonstrates the successful development and clinical application of the first dedicated artificial intelligence and VR platform for the planning of pulmonary segmentectomy. This is the first study that shows an immersive virtual reality-based application for preoperative planning of segmentectomy to the best of our knowledge.

Results: A total of 10 eligible patients for segmentectomy were included in this study after referral through the institutional thoracic oncology multidisciplinary team. PulmoVR was successfully applied as a supplementary imaging tool to perform video-assisted thoracoscopic segmentectomies. In 40% of the cases, the surgical strategy was adjusted due to the 3-dimensional-VR-based evaluation of anatomy. This underlines the potential benefit of additional VR-guided planning of segmentectomy for both surgeon and patient.
Conclusions: Our study demonstrates the successful development and clinical application of the first dedicated artificial intelligence and VR platform for the planning of pulmonary segmentectomy. This is the first study that shows an immersive virtual reality-based application for preoperative planning of segmentectomy to the best of our knowledge. (JTCVS Techniques 2021;7:309-21) Artificial intelligence and virtual reality reconstruction of segmental pulmonary anatomy.

CENTRAL MESSAGE
PulmoVR, a novel and immersive virtual reality-based application, is a feasible and clinically applicable method for accurate and patient-tailored surgical planning of pulmonary segmentectomy.
our hospital, we have adopted segmentectomy for specific surgical procedures, including resections of small ( 2 cm) stage I non-small cell lung cancer (NSCLC), benign intrapulmonary disease limited to segments (eg, bronchiectasis), and deeply located or nonpalpable metastases of other primary malignancies.
Thoracoscopic lung segmentectomies are generally more complex than thoracoscopic lobectomies. 1 Identifying intraparenchymal planes and ligation of target branches of pulmonary arteries, veins, and bronchi can be more challenging, partly due to the existence of many anatomical variations and abnormalities in pulmonary vascular anatomy. 3,4 Conventional computed tomography (CT) imaging is the gold standard for preoperative planning of anatomical (sub) lobar resections. Unfortunately, the identification of segmental borders and segmental branches of arteries, veins, and bronchi is challenging when conventional 2-dimensional (2D) CT is used.
The success of pulmonary segmentectomies largely depends on the surgeon's preoperative understanding of the anatomy. Previous studies have already demonstrated the feasibility and added value of 3-dimensional (3D) reconstructions of pulmonary anatomy for preoperative and intraoperative planning of pulmonary segment resections. 5,6 Over the past few years, virtual reality (VR) devices and software are gaining popularity and have been shown to improve the understanding of patient anatomy by surgeons. [7][8][9] A VR platform has more features and functions than existing 2D or 3D planning software, including immersive and interactive manipulation, realistic in-depth perception, and visualization of complex relationships of anatomic structures that are ready at hand and can be applied by surgeons to gain a more realistic insight into a patient's anatomy. 8,10 Moreover, automated imaging algorithms could potentially create a more efficient planning by enabling automatic visualization of anatomic structures of interest. 11,12 By developing an artificial intelligence (AI)-based and immersive 3D-VR platform as a supplementary preoperative planning tool to traditional CT imaging, a novelty can be added to the thoracic surgeon's armamentarium.
To overcome most of the shortcomings of the currently available imaging modalities and to create an easy-to-use and dedicated platform for CT imaging review, we have developed a 3D VR-based digital imaging and communication in medicine (DICOM) viewer. In addition, we developed PulmoVR (jointly developed and manufactured by Department of Cardiothoracic Surgery [Erasmus Medical Center], MedicalVR, EVOCS Medical Image Communication [Fysicon BV], and Thirona), a dedicated immersive 3D-VR and AI-based segmentectomy surgical planning tool. We hypothesized that such a platform could provide a more accurate preoperative understanding of the patient-specific segmental and intersegmental anatomy.
The present prospective single-center pilot study shows our early results on the clinical application of our innovative immersive AI and VR-based segmentectomy planning tool (PulmoVR) in 10 consecutive patients. The main objectives are to assess the technical feasibility and clinical applicability of PulmoVR in performing 3D VR-guided video-assisted thoracoscopic surgery (VATS) segmentectomies.

METHODS Patient Selection
The medical ethical committee of the Erasmus Medical Center approved this study (MEC-2020-0702). A total number of 10 consecutive patients who were eligible and accepted for pulmonary segmentectomy at the Erasmus Medical Center, Rotterdam, The Netherlands, were enrolled in this prospective study. All patients were consecutively included after referral and confirmation of eligibility for segmentectomy by the institutional local thoracic oncology multidisciplinary team meeting (Figure 1, A). After approval of a patient's eligibility for surgery, patients were planned for VATS segmentectomy. Subsequently, written informed consent was obtained from all patients and patients were included prospectively for VR-guided planning of segmentectomy as a supplement to conventional CT-guided planning. The inclusion criteria were as follows: age >18 years and pulmonary pathology suitable for pulmonary segmentectomy (ie, confirmed/clinical suspicion of stage I NSCLC 2 cm, limited benign lesion, and intrapulmonary metastases of other primary malignancies).

Study Participants
Two of our dedicated lung surgeons and a surgery resident physician participated in the study. All participants received a brief ($15 minutes) audiovisual tutorial on how to use the software and hardware before the start of the study.

VR-and AI-based Surgical Planning Modalities
An immersive VR-based CT (grayscale) DICOM viewer was developed together with MedicalVR. 13 In addition, a dedicated 3D-VR and AI-based pulmonary segmentectomy planning tool called PulmoVR was successfully developed together with MedicalVR, EVOCS Medical Image Communication (Fysicon BV), and Thirona. Segmental anatomy was determined automatically using LungQ software (Thirona). The AIbased delineation of pulmonary segments was based on determination of the bronchial and arterial intersegmental planes. Based on this delineation, the intersegmental veins were located within the border of segments. The anatomical 3D segmentations were visually checked by trained analysts Abbreviations and Acronyms ¼ computed tomography DICOM ¼ digital imaging and communication in medicine NSCLC ¼ non-small cell lung cancer S ¼ segment VATS ¼ video assisted thoracoscopic surgery VR ¼ virtual reality for accuracy and corrected, as needed, within Thirona's ISO 13485certified image analysis service. There were no specific technical requirements for 3D immersive evaluation of CT scans in VR besides a maximum of 1000 CT images and at least an adequate image quality for 2D evaluation. The AI analysis was possible on both contrast and noncontrastenhanced inspiratory thoracic CT scans with a maximum slice thickness of 1.5 mm.

Preoperative Planning Workflow
A schematic overview of the preoperative planning workflow is outlined in Figure 1 and Figure E1. After referral by our local multidisciplinary team ( Figure 1, A), a surgeon independently identified and documented the target segment(s) on preoperative diagnostic CT scans. (Figure 1, B). Subsequently, the DICOM files were extracted from the patient archiving and communication system (Figure 1, C) and VR-based evaluation was carried out by a cardiothoracic surgery resident physician who was blinded to the study to independently evaluate the anatomy in our 3D-VR DICOM viewer (Figure 1, D). In this phase, the VR-based images were only a grayscale, 3D volume rendering of the CT scan without postprocessing edits (Figure 1, D, and Figure 2, A). The resident physician identified the lesion first, after which he identified the target segment based on anatomical landmarks (eg, intersegmental vein). Next, a surgeon also reviewed the anatomy in 3D-VR ( Figure E1). The DICOM files were then transferred to Thirona and the pulmonary arteries, veins, segments, and the bronchi were segmented using LungQ (Thirona) AI-based lung quantification software (Figure 2, F). These analyses were transferred to our hospital within 72 hours. Then, PulmoVR enabled 3D visualization of both CT images and AI-based segmentation (segmentation means digital labeling of anatomic structures) files in immersive VR (Figure 2, G). Reconstructions were created for all patients undergoing segmentectomy and the target segment and patient-specific anatomy was evaluated in PulmoVR for the final time by the surgeon (Figure 2, G).
During surgery, a VR workstation was located in the operating theatre and provided an additional review of the anatomy when necessary. Moreover, preoperatively acquired 3D reconstructions of anatomy were displayed on operating theatre monitors intraoperatively. (Figure 2, H).

Surgical Technique
Two dedicated cardiothoracic surgeons performed the 3D VATS segmentectomies independently (6 and 4 each). Surgical assistance was provided by a resident physician. All patients underwent 3D 3-port VATS under general anesthesia and double-lumen intubation and single lung ventilation were applied. Intraoperative intersegmental border demarcation was achieved using intravenous indocyanine green as described previously by Iizuka and colleagues. 14 Transection of parenchyma was performed with endostaplers.

Clinical End Point and Qualitative Assessment
All data were analyzed using Microsoft Office Excel 2015 (Microsoft, Redmond, Wash). This study aimed to study technical feasibility on the development of an AI-and VR-based tool for preoperative planning and performance of segmentectomy. Technical feasibility is defined as the ability to create an AI-based segmentation of patient-specific CT scans and to subsequently review pulmonary arteries, veins, bronchi, and segmental planes in an immersive VR application. The following clinical results were selected to serve as a clinical end point to demonstrate potential clinical safety: success of segmentectomy, resection margin (radical R0 resection), and intraoperative blood loss.
The participating cardiothoracic surgeons were asked to fill out a questionnaire (Appendix E1) after each procedure to assess intraoperative anatomy correspondence with their VR-based planning images. Additional questionnaires were filled out by the surgeons to evaluate the ease of use, satisfaction, and attitude toward future use of VR-guided surgical planning with PulmoVR (Appendix E2). These qualitative results served only as a subjective measurement to illustrate the potential usability of VR-based surgical planning.

Patient Characteristics
A total of 10 consecutive patients (5 men and 5 women; mean age, 51 AE 13.4 years) were referred for elective segmentectomy and were enrolled in this study (Table 1). All baseline patient characteristics, pathology results, and target segments are outlined in Table 1.

Technical Feasibility: 3D-VR and PulmoVR Reconstructions
In all patients, VR reconstructions were made successfully. As demonstrated in Figures 2 and 3, for all patients, the segmental arteries, segmental veins, bronchi, and segmental borders were visualized in PulmoVR. Segmentations of vasculature and segmental borders were made only in the lobe of interest (Figures 2 and 3). All other thoracic structures were automatically visualized in grayscale as well. Based on the questionnaire that was filled out by the surgeons postoperatively after each case (Appendix E1), in 100% of the cases, intraoperative anatomy corresponded entirely with preoperative VR images. Videos 1 and 2 demonstrate the VR-guided evaluation of the intrapulmonary segmental anatomy from various perspectives in 2 different cases.

Clinical Applicability: Perioperative Outcomes and Surgical Adjustments
All segmentectomy procedures were performed successfully through 3D VATS, assisted by preoperative VR planning. A complete resection (ie, R0) with adequate surgical margins was obtained in all cases (Table 1).  In 4 out of 10 cases, the VR-based evaluation resulted in a change in the selection of the target segment than the initially selected target segment by 2D-CT. We will briefly highlight these 4 cases. For all operations, initial surgical strategies and changes after review in VR are outlined in Table 1. Patient 1. A patient with colorectal metastasis in the right lower lobe was referred for segmentectomy of segment (S) 9. In this patient VR evaluation showed tumor expansion through the segmental borders of S6, S9, and S10. Consequently, surgical strategy after evaluation in VR was adjusted accordingly and en-bloc segmentectomy of S9 and S10 and a subsegmentectomy of S6 was performed (see Figures 2 and 3 for VR images of this patient). Patient 4. A patient with a tumor in the apico-dorsal part of S1 of the right upper lobe was referred for segmentectomy of S1. VR evaluation showed the relatively small tumor margin in relation with the S1/S2 border (Figure 4, A and B), which was not visible on preoperative CT (Figure 4, D). Consequently, the patient underwent an en-bloc resection of S1 and a subsegment of S2 to maintain safe surgical margins. Intraoperatively obtained indocyanine green-guided intersegmental border visualization corresponded with the preoperatively acquired VR reconstructions (Figure 4, C). Patient 7. Patient 7 had a history of pulmonary metastasized mixed germ cell tumor for which he underwent wedge resection of the right lower lobe during September 2019. During follow-up, there was a new pulmonary lesion (suspicious recurrent metastasis) for which he received chemotherapy. Chemotherapy did not fully resolve the lesion and the patient was accepted for redo VATS segmentectomy for the cystic lesion. After review in CT, the lesion was believed to be located in S6 (Figure 4, H). However, after evaluation in VR, the tumor seemed to be located clearly in S8 (Figure 4, E and F). A successful segmentectomy of S8 was performed (Figure 4, G).

Patient 9.
A patient with a tumor in the lingula (suspicious for metastasis/loco-regional recurrence of stage IV NSCLC) based on CT images was referred for lingulectomy (S4 and S5). However, VR evaluation showed that the lesion was located entirely within S5 and consequently resulted in a successful segmentectomy of S5. Table 2 depicts the experience of the surgeons. Overall, PulmoVR was rated to be easy to use, useful, and efficient for preoperative planning of segmentectomy.

DISCUSSION
In general, pulmonary segmentectomies are considered technically challenging and difficult procedures, especially when compared with lobectomy or other sublobar resections, such as wedge resection. Several factors have been identified that contribute to the technical difficulties of segmentectomy, including understanding of the individual pulmonary anatomy and its relation to the tumor localization. Accurate evaluation of segmental pulmonary anatomy remains a challenge when assessed on 2D-CT images. For this reason, we codeveloped an innovative AI and immersive VR-based segmentectomy planning tool (the PulmoVR). In this study we report our early results on the technical feasibility and clinical applicability of our PulmoVR application in patients undergoing VATS segmentectomy.
Recent studies have explored and highlighted the role of 3D-CT angiography and manual 3D virtual modeling of CT images to examine patient-specific (segmental) anatomy preoperatively. [15][16][17] Moreover, reports from other studies that have used custom-made 3D software to simulate a segmentectomy procedure have been published and underline the benefits that 3D modeling offers in the context of segmentectomy planning. 15,17 An important disadvantage of 3D modeling is the loss of data because most often these applications focus on the visualization of specifically selected and requested anatomic structures such as pulmonary vessels, tumor, bronchi, and parenchyma. 16 As a result, information on intrapulmonary lymph nodes, thoracic cage anatomy (eg, intercostal spaces and sternum), the heart, diaphragm, and other intrathoracic vessels is lost or needs to be selected and manually segmented separately. Our 3D-VR DICOM viewer, as described in the current study, enables direct VR visualization by virtually stacking the CT images and allowing evaluation in VR. This provides a fast method (<1 minute) to evaluate patient-specific CT scans in 3D and with reality-like in-depth perception in 360 without the need for postprocessing techniques and without loss of imaging data. 11,12 Furthermore, in currently available techniques 3D segmentation (which is the identification and selective coloring of an anatomic structure) is often performed manually (by technicians), which increases the risk for error, requires manual labor, is time-consuming, and for this reason is more expensive. 18 Another shortcoming of the available 3D modeling software is that it commonly requires contrast-enhanced CT scanning for accurate segmentation, which results in radiation and iodinecontrast agent exposure and consequently in additional health risks for patients. 18 In this study, we have demonstrated that our PulmoVR application provides a VIDEO 1. Virtual reality (VR)-based view of a patient-specific scan with a tumor in the right lower lobe. The review of the anatomy is performed by using VR controllers (Oculus Rift S [Oculus VR, Irvine, Calif]) that enable 360 rotation and zoom. Different segments of the right lower lobe are shown transparent in various colors. The bronchial anatomy (green) is also shown. Additionally, measurements can be performed in virtual reality to determine tumor size and expected surgical margins. Video available at: https://www.jtcvs.org/article/S2666-2507(21)00253-4/fulltext. unique environment to review patient-specific segmental anatomy by facilitating selective AI-based automatic labeling, visualization, and assessment of pulmonary structures, without the need for contrast-enhanced CT imaging. Moreover, automated imaging algorithms could potentially create more efficient planning.
In addition, 3D imaging review on a computer screen remains a 3D visualization on a 2D monitor and requires extensive (and separate) review of pulmonary structures from different angles to understand the spatial relations and in-depth anatomy of structures. Finally, obtaining accurate 3D images in this case also requires additional investment in hardware (3D monitors, high-performance computers, and glasses). In accordance to the reported data from the literature on VR applications, in this study we have demonstrated that our PulmoVR tool enables immersive interactive image manipulation, realistic in-depth perception, and immersive visualization of complex relationships of anatomic structures (eg, vascular, bronchial, tumor margins, and segmental borders) that can be applied by surgeons preoperatively (without the need to consult radiologists or imaging technicians) to gain a better and more realistic insight into a patient's specific anatomy. 8,10 Additionally, we have showed that PulmoVR tool is easy and ready to use after only a single demonstration and tutorial of 15 minutes as was concluded from the questioners filled out by the participating surgeons. The software can be installed on every computer or laptop with sufficient graphical and processing power and the off-the-shelf VR head mounted displays are widely available. There is no need for additional 3D screens, hardware, or glasses.

Future Outlook
In more comparative research we aim to study the application of PulmoVR in larger patient populations. In addition, our goal is to design a multiuser platform in which a patient's anatomy can be evaluated by more users at the same time. Hopefully, this would also pave the way for the development of surgical simulations for education and training purposes.
Other exciting and futuristic VR applications in surgical planning might be the application of VR or augmented reality to enhance intraoperative views in robotic procedures, such as robotic segmentectomies. In terms of preoperative planning, VR can assess the ideal thoracoscopic port placement locations. Also, the intraoperative 3D view could be merged with preoperatively acquired reconstructions. This could potentially enhance intraoperative navigation and also the identification of very small intrapulmonary lesions, lymph nodes, and small vascular branches. Other interesting examples of VR-based guidance and planning are navigational (ie, VR guided) bronchoscopy or augmented reality guided detection of small and deeply located lesions. 9,19 Further development and use of these novel technologies could potentially result in better preoperative diagnostics.  FIGURE 5. The identification of segmental anatomy for surgical planning of lung segmentectomy is quite challenging with conventional imaging. In the current study, a novel method is presented to create artificial intelligence based virtual reality reconstructions to prepare for thoracoscopic segmentectomies in 10 consecutive patients. CT, Computed tomography; VR, virtual reality. Usefulness and efficiency of PulmoVR 1. PulmoVR helps me to review segmental anatomy more accurately compared to reviewing CT scans in 2D 5 5 2. The assessment pulmonary segmental anatomy in PulmoVR takes less time than conventional CT scan review 4 5 3. Segmentation (highlighting arteries, veins, and bronchi in color) of the CT scans in PulmoVR is of added value for CT assessment 5 5 4. PulmoVR helps me to prepare a surgical strategy more efficiently compared with conventional methods (eg, CT scans).

5
5. PulmoVR review of CT scans creates more awareness of spatial orientation of anatomical structures and therefore contributes to the safer performance of segmentectomies 5 5 Attitude toward future use of PulmoVR in preoperative planning of segmentectomy 1. I would like to work with PulmoVR in the future 5 5 2. The future, I prefer using PulmoVR for preoperative planning as a supplement to conventional methods 5 5 3. In the future, I prefer using PulmoVR for preoperative planning instead of conventional methods 2 1 4. I believe, in the future, PulmoVR-like technologies will become a standard of care imaging modality for preoperative planning of segmentectomies 5 5 5. I would recommend PulmoVR as a preoperative planning tool to my colleagues 5 5 CT, Computed tomography; 3D, 3 dimensional; 2D, 2 dimensional. *Scale was 1 ¼ strongly disagree to 5 ¼ strongly agree. yPulmoVR was jointly developed and manufactured by Department of Cardiothoracic Surgery, MedicalVR, EVOCS Medical Image Communication (Fysicon BV), and Thirona.

Limitations
We acknowledge that the current pilot study has some limitations in terms of technology and study setup. First, only a small number of patients were included, and only 2 cardiothoracic surgeons have used and evaluated the platform. To draw definite conclusions to structurally implement such technology, more prospective, extensive, and comparative analyses are required. Also, cost-effectiveness studies are important to evaluate implementation in the clinic. In addition, in the current situation, only segmental branches of blood vessels are visualized in PulmoVR. To further improve on the software, the entire pulmonary artery should ideally be segmented to ease evaluation in PulmoVR. Ideally, also (semi)automatic segmentation of intrapulmonary and mediastinal lymph nodes could ease surgical planning and performance of lymph node dissection.
Another important limitation of PulmoVR is the lack of the possibility to simulate lung deflation, manipulation, and virtual resection/transection. As a result, PulmoVR cannot yet be used to create augmented virtual projections of anatomy within the physical surgical field. Future development of augmented reality and digital navigation systems should hopefully overcome this challenge within the next decade.

CONCLUSIONS
This study presents the first artificial intelligence and immersive VR-based visualization method for segmentectomy planning ( Figure 5). We have demonstrated technical feasibility and clinical applicability in 10 consecutive patients undergoing thoracoscopic segmentectomy. Notably, the application of PulmoVR resulted in essential changes in surgical strategy in 40% of the cases. All segmentectomy procedures were performed successfully, and all oncologic resections adequately. More extensive clinical trials are needed to study safety and clinical benefits and to provide clinical evidence for the application of virtual reality in segmentectomy planning.

Conflict of Interest Statement
Drs Sadeghi and Mahtab are coinventors of the virtual realitybased technology presented in this article. All other authors reported no conflicts of interest. This work was supported by Koers23, Erasmus Medical Center, The Netherlands.
The Journal policy requires editors and reviewers to disclose conflicts of interest and to decline handling or reviewing manuscripts for which they may have a conflict of interest. The editors and reviewers of this article have no conflicts of interest. 4. I believe, in the future, PulmoVR-like technologies will become a standard of care imaging modality for preoperative planning of segmentectomies