If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Prolonged air leak is the most common complication of thoracic surgery. Intraoperative leak site detection is the first step in decreasing the risk of leak-related postoperative complications.
Methods
We retrospectively reviewed the surgical videos of patients who underwent lung resection at our institution. In the training phase, deep learning-based air leak detection software was developed using leak-positive endoscopic images. In the testing phase, a different data set was used to evaluate our proposed application for each predicted box.
Results
A total of 110 originally captured and labeled images obtained from 70 surgeries were preprocessed for the training data set. The testing data set contained 64 leak-positive and 45 leak-negative sites. The testing data set was obtained from 93 operations, including 58 patients in whom an air leak was present and 35 patients in whom an air leak was absent. In the testing phase, our software detected leak sites with a sensitivity and specificity of 81.3% and 68.9%, respectively.
Conclusions
We have successfully developed a deep learning-based leak site detection application, which can be used in deflated lungs. Although the current version is still a prototype with a limited training data set, it is a novel concept of leak detection based entirely on visual information.
We developed novel deep learning-based air leak detection software to automatically estimate potential leak sites from intraoperative images. We retrospectively assessed the usefulness of this software. Although the current version is in a preliminary phase and works with a limited training data set, we still obtained good sensitivity and specificity.
Prolonged air leak is the most common complication of pulmonary resection and one of the factors that prolong hospital stay.
To find the leak point, a leak test is performed by immersing the thoracic cavity in water and observing air leakage from the lung surface before closing the wound. However, in these days of minimally invasive surgeries, such as video-assisted thoracoscopic surgery and robot-assisted thoracoscopic surgery (RATS), the problems of poor visibility due to the inflated lung itself and damage of nonphysiologic positive pressure ventilation to the lung parenchyma during the conventional leak test have been recognized. Detecting a spot that requires extra treatment to prevent or treat air leaks could reduce medical costs, patients' burden of prolonged air leak-related complications, and hospital stay.
Among recent advances in computer vision studies, J. Redmon developed a real-time object detection framework using deep learning methods.
Redmon J, Farhadi A. YOLO9000: better, faster, stronger. Paper presented at: IEEE Conference on Computer Vision and Pattern Recognition; July 21-26, 2017; Honolulu.
“Object detection” is a computer algorithm for locating and recognizing specific objects in images or videos. This framework has been previously applied to detect bone fractures and tumors in radiography.
However, it has not been applied to detect specific anatomical structures or organs during surgery. The algorithm is an open-source system that can show the existence and range of an object to be detected from an unknown image by examining the visual information. The processing speed is high, indicating that the method can be used as a real-time detection method for static images as well as for videos.
The procedures in RATS, which have no tactile sense, are primarily performed on the basis of the fine visual image. Moreover, with the development of vision systems, such as 4K and 3D endoscopic cameras, more detailed images are helping surgeons perform more complex procedures in recent minimally invasive surgery.
Acquiring basic and advanced laparoscopic skills in novices using two-dimensional (2D), three-dimensional (3D) and ultra-high definition (4K) vision systems: a randomized control study.
The importance of visual information in thoracic surgery is gradually increasing.
A leak site is a lung area where the lung surface is missing or damaged. We attempted to develop an algorithm to automatically detect a “potential leak site” that has the objective appearance of the surface of the lung that is associated with an elevated risk of air leak. In this study we aimed to evaluate the clinical applicability of the prototype application.
Methods
Development of Our Novel Application
This study comprised 2 major phases: training phase and testing phase (Figure 1). Different data sets without overlapping data were used during the testing and training phases. The testing data set included 109 detected sites containing 64 leak-positive sites and 45 leak-negative sites. The testing data set was obtained from 93 operations, including 58 patients in whom an air leak was present and 35 patients in whom an air leak was absent. During the training phase, for each leak site, we prepared the original image and labeled image with the leak area marked in color (Figure 2). Our proposed software obtains an endoscopic image as the input and outputs leak area if a leak site is detected in the image. The input endoscopic image was divided into S × S nonoverlapped grid cells, and each cell detects the potential leak site belonging to the grid cell, as applied in Redmon.
Redmon J, Farhadi A. YOLO9000: better, faster, stronger. Paper presented at: IEEE Conference on Computer Vision and Pattern Recognition; July 21-26, 2017; Honolulu.
The output image is presented in Figure 1. The red rectangular box (ie, predicted box) shows the algorithm-based predicted leak area by our developed software. The confidence score is the representative value of the probability that a leak site exists in the predicted box. This confidence score is a value that can range from 0 to 1, where a score of 0 represents a grid cell that does not contain any “potential leak site: lung parenchyma that has the objective appearance associated with an elevated risk of an air leak” and 1 represents 100% confidence that a “leak site” is present in the predicted grid cell.
Figure 1Schematic diagram of this study, consisting of training and testing phases. The 110 pairs of the original and leak site marked images were used in the training data set, whereas another set of thoracoscopic images and videos were used to evaluate the performance of the developed software. YOLO, You Only Look Once.
We retrospectively reviewed the surgical videos of patients who underwent pulmonary resection at our institution. In collecting leak-positive images, we defined the inclusion criteria according to the following 3 conditions: cases in which the presence of an intraoperative leak using the conventional leak test was described in the surgical record; cases in which surgical videos were preserved and intraoperative leak tests were recorded; and cases in which the leak area can be visually confirmed. During the pilot development phase, training data sets were often collected in hundreds,
Redmon J, Farhadi A. YOLO9000: better, faster, stronger. Paper presented at: IEEE Conference on Computer Vision and Pattern Recognition; July 21-26, 2017; Honolulu.
Localisation of colorectal polyps by convolutional neural network features learnt from white light and narrow band endoscopic images of multiple databases.
so we aimed to collect data from 100 leak sites for the prototype application. One surgeon (Y.K.) identified the true leak site from the conventional air leak test in the surgical videos. One frame containing the leak site in the video was screen-captured per single leak site, and the image with the leak area was marked with color (training phase; Figure 1). We collected 110 images of leak sites from 70 surgeries. These 110 labeled and originally captured images were randomly translated, clipped, and scaled. This preprocess creates various patterns of local images and improves stability of learning. Hence, we used clipped images with 416 × 416 pixels as the training data set, 32 for each training batch and 300 for the total number of training epochs to train the application.
In the testing phase, we evaluated our application using another data set. We prepared cases with a confirmed air leak site and cases without air leak using the traditional leak test. The leak-positive test data consisted of images selected from consecutive cases between January 2021 and September 2021, on the basis of the same inclusion criteria in the training data set. For the leak-positive cases, the captured image was trimmed in the following conditions: there was a large amount of saline remaining in the thoracic cavity, the text was displayed, and the leak site was biased toward the edges of the screen. In collecting leak-negative images, we defined the inclusion criteria as the following 2 conditions: cases in which the intrathoracic saline had been sufficiently removed after a conventional leak test, and cases in which the hilar and residual lung parenchymas were recorded for >5 seconds in the surgical video. Considering these criteria, leak-negative surgeries were randomly selected. The target number of images was set at approximately half of the cases of leak-positive images between May 2019 and September 2021.
Only 1 frame from the surgical video was screen-captured per each leak site, and we did not recapture another frame of the same leak site after evaluation by the application. This study was conducted on a PC Intel Core i7-10700 with 32 GB RAM, clock speed or frequency of CPU at 2.90 GHz, and GPU of NVIDIA GeForce RTX 2070 SUPER.
Performance Evaluation Measures
In our developed software, multiple predicted boxes may be displayed in a single output. Therefore, we categorized each predicted box into 4 categories: true positive (TP), false positive (FP), true negative (TN), and false negative (FN). We objectively evaluated the performance in terms of sensitivity and specificity (Figure 3). The area under the receiver operating curve and Youden index were calculated to assess the optimal cutoff value for the confidence score. Specifically, this method determines the optimal cutoff point as the tangent-based point that is closest to the point of the (0, 1) coordinates in Figure 4. This model was also evaluated using precision and recall scores to compare its usefulness with that of previously reported models that use deep learning algorithms. The precision score (%) was calculated as TP/(TP + FP) × 100, which is the same index as the so-called positive predictive value. The recall score (%) was calculated as TP/(TP + FN) × 100.
Figure 4Receiver operating characteristic (ROC) curve of the confidence score to predict the leak site. The optimal cutoff value was 0.09 for the confidence score of each bounding box.
The study protocol was approved by the institutional review board of Nagoya University School of Medicine (2015-0458, March 8, 2016). The requirement for informed consent was waived because of the retrospective study design.
Results
Figure 5 presents 2 cases of our application's ability to detect the leak site on the intraoperative endoscopic images, with Figure 5, A and D showing the actual leak site. To compare the actual leak site and predicted site in the testing phase, the leak site was colored in green in advance when the testing data set was constructed, as shown in Figure 5, B and E. Figure 5, C and F show the leak sites detected by our developed model. Of the 238 surgeries with lung resection performed in the testing phase, 90 (37.8%) had leaks in the conventional leak test. We obtained 64 leak-positive sites and 11 leak-negative sites from 58 (64%) of the surgeries. We obtained 35 leak-negative sites from 35 of 397 (8.8%) leak-negative surgeries. Clinical information of the testing data set is shown in Table E1. Overall, the testing data set contained 64 leak-positive sites and 45 leak-negative sites. As noted in the Methods section, each predicted box has a confidence score representing the probability that the leak site is in the predicted box. When the cutoff value of the confidence score was set at 0, the number of TP, FP, TN, and FN boxes were 52, 14, 31, and 12, respectively. As a result, the sensitivity and specificity were 81.3% and 68.9%, respectively. The precision was 78.7%, and the recall was 81.3%. The receiver operating characteristic curve is presented in Figure 4, and the area under the receiver operating characteristic curve was 0.689. There was an optimal cutoff value of 0.09, with a sensitivity of 62.5% and specificity of 69.2%.
Figure 5Sample cases of leak site detection. A and D, The original captured image. B and E, The actual leak site detected using the traditional leak test. C and F, The output images from the application. The lower case had a large amount of saline remaining in the cavity, and the part of the background was trimmed as shown in (F) before running the application.
Our developed software simultaneously predicts multiple bounding boxes, and we can process streaming video almost in real time. Although it is still the preliminary stage, we verified the results with a video clip of a scene after a traditional leak test, and our software detected the same as a still image (Video 1).
Discussion
In this study, we developed deep learning-based software that detects the presence or absence of leaks in the locations of potential air leak sites on endoscopic surgical images. If an air leak is known to be present at the time of postresection visual inspection, additional therapy could be applied to reduce the severity or duration of that air leak. Prolonged air leak after lung resection is the most common complication and is related to long hospital stay and high cost.
Prolonged air leak also leads to other complications, such as empyema and wound infection. There is no definitive definition of prolonged air leak, but several studies defined it as a condition lasting more than 5 to 10 days.
Therefore, detection of the air leak site is the first step in reducing air leak-related postoperative complications, shortening hospital stays, and reducing medical costs.
Recently, an automatic diagnostic system on the basis of deep learning algorithms has been proposed, and some clinical applications have been initiated.
However, this trend has been limited to pathology and radiology, where human experts have originally made visual decisions, and it is a substitute for that process.
Redmon J, Farhadi A. YOLO9000: better, faster, stronger. Paper presented at: IEEE Conference on Computer Vision and Pattern Recognition; July 21-26, 2017; Honolulu.
In contrast, the leak site detection procedure has not traditionally used visual detection. The conventional submersion test is conducted by inflating the lung within the saline-filled pleural cavity.
Surgeons do not find the leak site by direct visual inspection but by following the bubbles that leak from the damaged lung surface. Recently, with the increased application of minimally invasive surgeries, the inflated lung parenchyma itself might block the surgeon's view and prevent a high-quality leak test.
Although there have been attempts to develop new leak test methods, a new technique that does not require inflation has not been introduced so far.
The precision and recall of the software were 78.7% and 81.3%, respectively. Because there is no previous deep learning-based application in the field of leak testing, it is difficult to compare the performance of our proposed application. In the previously reported outcomes in the prototype application in radiology and pathology, precision and recall are approximately 87% to 93% and 65% to 88%, respectively.
Compared with previous studies, the precision score of the current model is slightly lower. This indicates that it is detecting regions other than the leak sites, creating FP cases. For example, our software detected areas, such as the mediastinal fat and surgical instruments (Figure E1). In contrast, FN cases were also apparent, in which the appearance of the leak site was similar to the color of the background lung (Figure E2). These FP and undetected cases are expected to decrease in the future as the amount of training data increase.
This software has 3 strengths. First, it is on the basis of a deep learning algorithm, which is expected to provide a smooth transition from images to videos. Second, this software would not require lung inflation, which allows us to have sufficient working space, even in minimally invasive surgery. Last, we found the deep learning-based algorithm to be good at detecting small, damaged areas that are sometimes difficult to detect with the human eye (Figure 5, D-F).
This study has several limitations. First, the captured surgical endoscopic images were retrospectively extracted from recorded videos. In future prospective clinical trials, the method of how we screened the lung surface would be of importance. Without appropriate manipulation, there might be areas of leakage that might not be visible. This would influence the ability to detect “unsuspected” areas of leakage. Second, in leak-negative cases, when the conventional leak test did not reveal a leak, the residual lung surface was not observed. Therefore, it is unclear how this application would be effective when manipulating the lung surface without knowing whether there is a leak. Third, in cases featuring leakage from the automatic suturing line, surgeons sometimes cannot find the leak sites even after careful visual inspection. In this study, we excluded cases in which the leak site was not visible in either the training data set or validation data set, so detection of leakage of this type is not expected at present. If the resolution of endoscopic cameras is improved so that they can detect small deficits more effectively than the human eye, it might be possible to train the detection of these types of leaks. Fourth, the current version is still in a preliminary phase and works with extremely limited training data. When the predicted box is more precise, this application would be more useful. In the future, we would like to conduct a multicenter prospective study to upgrade this application, making it more useful and contributing to the reduction in prolonged and unexpected postoperative lung leaks after lung resection. Finally, this study was conducted on the basis of the latest artificial intelligence theory, and we acknowledge a certain difficulty to understand for the general clinicians. The 2 publications that introduced machine learning related to the medical field might help this discrepancy.
We have developed a pulmonary air leak site detection application on the basis of still images of deflated lung tissue obtained from a robotic or thoracoscopic camera immediately after pulmonary resection. The concept of our application is on the basis of visual information only, which could change the concept of conventional leak tests and allow appropriate intervention to reduce or mitigate air leaks.
Conflict of Interest Statement
The authors have obtained a patent for this leak detection technique. At the time of writing this report, this technology had not been commercialized and no financial ties had yet been planned. The authors reported no conflicts of interest.
The Journal policy requires editors and reviewers to disclose conflicts of interest and to decline handling or reviewing manuscripts for which they may have a conflict of interest. The editors and reviewers of this article have no conflicts of interest.
The authors thank all staff members of the Department of Thoracic Surgery, Nagoya Graduate School of Medicine.
Figure E1False detection of leak sites. A and D, The captured original image. B and E, The actual leak site detected using the conventional leak test. C and F, The output images from the application. White arrows indicate incorrectly detected mediastinal fat and surgical instruments.
Figure E2Sample cases of false-negative results. A and C, The captured original image. B and D, The actual leak site detected using the traditional leak test. There is only a slight color difference between the background tissue and leak site.
Redmon J, Farhadi A. YOLO9000: better, faster, stronger. Paper presented at: IEEE Conference on Computer Vision and Pattern Recognition; July 21-26, 2017; Honolulu.
Acquiring basic and advanced laparoscopic skills in novices using two-dimensional (2D), three-dimensional (3D) and ultra-high definition (4K) vision systems: a randomized control study.
Localisation of colorectal polyps by convolutional neural network features learnt from white light and narrow band endoscopic images of multiple databases.
00365-0&id=fx1.jpg){kind=link}
00365-0&id=fx2.jpg){kind=link}
00365-0&id=gr1.jpg){kind=link}
00365-0&id=gr2.jpg){kind=link}
00365-0&id=gr3.jpg){kind=link}
00365-0&id=gr4.jpg){kind=link}
00365-0&id=gr5.jpg){kind=link}
00365-0&id=fx4.jpg){kind=link}
00365-0&id=fx5.jpg){kind=link}