Skip to main content

Advertisement

SpringerLink
Log in

Systematic Review of Generative Adversarial Networks (GANs) for Medical Image Classification and Segmentation

  • Review
  • Published:
Journal of Digital Imaging Aims and scope Submit manuscript

Abstract

In recent years, generative adversarial networks (GANs) have gained tremendous popularity for various imaging related tasks such as artificial image generation to support AI training. GANs are especially useful for medical imaging–related tasks where training datasets are usually limited in size and heavily imbalanced against the diseased class. We present a systematic review, following the PRISMA guidelines, of recent GAN architectures used for medical image analysis to help the readers in making an informed decision before employing GANs in developing medical image classification and segmentation models. We have extracted 54 papers that highlight the capabilities and application of GANs in medical imaging from January 2015 to August 2020 and inclusion criteria for meta-analysis. Our results show four main architectures of GAN that are used for segmentation or classification in medical imaging. We provide a comprehensive overview of recent trends in the application of GANs in clinical diagnosis through medical image segmentation and classification and ultimately share experiences for task-based GAN implementations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Availability of Data and Material

Not applicable.

Code Availability

Not applicable.

References

  1. Goodfellow I, Pouget-Abadie J, Mirza M, et al.: Generative adversarial nets, In Advances in neural information processing systems, 2014.

  2. Zhang X, Zhu X, Zhang N, et al.: Seggan: Semantic segmentation with generative adversarial network, In 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM), 2018, IEEE.

  3. Li C, and Wand M: Precomputed real-time texture synthesis with markovian generative adversarial networks, In European conference on computer vision, 2016, Springer.

    Book  Google Scholar 

  4. Wang X, and Gupta A: Generative image modeling using style and structure adversarial networks, In European conference on computer vision, 2016, Springer.

    Book  Google Scholar 

  5. Tzeng E, Hoffman J, Saenko K, et al.: Adversarial discriminative domain adaptation, In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.

  6. Wang K, Gou C, Duan Y, et al.: Generative adversarial networks: introduction and outlook, IEEE/CAA Journal of Automatica Sinica, 2017, 4, 588–598.

  7. Kurach K, Lucic M, Zhai X, et al.: The gan landscape: Losses, architectures, regularization, and normalization, 2018.

  8. Pan Z, Yu W, Yi X, et al.: Recent progress on generative adversarial networks (GANs): A survey, IEEE Access, 2019, 7, 36322–36333.

  9. Deng J, Dong W, Socher R, et al.: Imagenet: A large-scale hierarchical image database, In 2009 IEEE conference on computer vision and pattern recognition, 2009, Ieee.

  10. Liu Z, Luo P, Wang X, et al.: Deep learning face attributes in the wild, In Proceedings of the IEEE international conference on computer vision, 2015.

  11. Chuquicusma MJ, Hussein S, Burt J, et al.: How to fool radiologists with generative adversarial networks? a visual turing test for lung cancer diagnosis, In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), 2018, IEEE.

  12. Lau F, Hendriks T, Lieman-Sifry J, et al.: Scargan: chained generative adversarial networks to simulate pathological tissue on cardiovascular mr scansDeep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, 2018, Springer, pp. 343–350.

  13. Lee J-H, Tae Kim S, Lee H, et al.: Feature2mass: Visual feature processing in latent space for realistic labeled mass generation, In Proceedings of the European Conference on Computer Vision (ECCV), 2018.

  14. Yi X, Walia E, and Babyn P: Generative adversarial network in medical imaging: A review, Medical image analysis, 2019, 58, 101552.

  15. Gui J, Sun Z, Wen Y, et al.: A review on generative adversarial networks: Algorithms, theory, and applications, arXiv preprint arXiv:200106937, 2020.

  16. Wang Z, She Q, and Ward TE: Generative adversarial networks in computer vision: A survey and taxonomy, arXiv preprint arXiv:190601529, 2019.

  17. Moher D, Liberati A, Tetzlaff J, et al.: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement, PLoS med, 2009, 6, e1000097.

  18. Harzing A-W: The publish or perish book, 2010, Tarma Software Research Pty Limited.

  19. Mirza M, and Osindero S: Conditional generative adversarial nets, arXiv preprint arXiv:14111784, 2014.

  20. Li H, Chen D, Nailon WH, et al.: Signed Laplacian Deep Learning with Adversarial Augmentation for Improved Mammography Diagnosis, In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2019a, Springer.

  21. Nie D, and Shen D: Adversarial Confidence Learning for Medical Image Segmentation and Synthesis, International Journal of Computer Vision, 2020, 1–20.

  22. Radford A, Metz L, and Chintala S: Unsupervised representation learning with deep convolutional generative adversarial networks, arXiv preprint arXiv:151106434, 2015.

  23. Han C, Rundo L, Araki R, et al.: Infinite brain MR images: PGGAN-based data augmentation for tumor detectionNeural approaches to dynamics of signal exchanges, 2020, Springer, pp. 291–303.

  24. Isola P, Zhu J-Y, Zhou T, et al.: Image-to-image translation with conditional adversarial networks, In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.

  25. Zhu J-Y, Park T, Isola P, et al.: Unpaired image-to-image translation using cycle-consistent adversarial networks, In Proceedings of the IEEE international conference on computer vision, 2017.

  26. Ning Y, Han Z, Zhong L, et al.: Automated pancreas segmentation using recurrent adversarial learning, In 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 2018, IEEE.

  27. Xing J, Li Z, Wang B, et al.: Lesion Segmentation in Ultrasound Using Semi-pixel-wise Cycle Generative Adversarial Nets, IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2020.

  28. Siddiquee MMR, Zhou Z, Tajbakhsh N, et al.: Learning fixed points in generative adversarial networks: From image-to-image translation to disease detection and localization, In Proceedings of the IEEE International Conference on Computer Vision, 2019.

  29. Huo Y, Xu Z, Moon H, et al.: Synseg-net: Synthetic segmentation without target modality ground truth, IEEE transactions on medical imaging, 2018, 38, 1016–1025.

  30. Kwon G, Han C, and Kim D-s: Generation of 3D brain MRI using auto-encoding generative adversarial networks, In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2019, Springer.

  31. Ossenberg-Engels J, and Grau V: Conditional Generative Adversarial Networks for the Prediction of Cardiac Contraction from Individual Frames, In International Workshop on Statistical Atlases and Computational Models of the Heart, 2019, Springer.

  32. Islam J, and Zhang Y: GAN-based synthetic brain PET image generation, Brain Informatics, 2020, 7, 1–12.

  33. Yu Z, Xiang Q, Meng J, et al.: Retinal image synthesis from multiple-landmarks input with generative adversarial networks, Biomedical engineering online, 2019b, 18, 1–15.

  34. Zhang T, Fu H, Zhao Y, et al.: SkrGAN: Sketching-rendering unconditional generative adversarial networks for medical image synthesis, In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2019, Springer.

  35. Xu Z, Wang X, Shin H-C, et al.: Tunable CT lung nodule synthesis conditioned on background image and semantic features, In International Workshop on Simulation and Synthesis in Medical Imaging, 2019, Springer.

  36. Zhou Y, He X, Cui S, et al.: High-resolution diabetic retinopathy image synthesis manipulated by grading and lesions, In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2019, Springer.

  37. Li H, Paetzold JC, Sekuboyina A, et al.: Diamondgan: unified multi-modal generative adversarial networks for mri sequences synthesis, In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2019b, Springer.

  38. Falotico R, and Quatto P: Fleiss’ kappa statistic without paradoxes, Quality & Quantity, 2015, 49, 463–470.

  39. Chi Y, Bi L, Kim J, et al.: Controlled synthesis of dermoscopic images via a new color labeled generative style transfer network to enhance melanoma segmentation, In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2018, IEEE.

  40. Shen T, Gou C, Wang F-Y, et al.: Learning from adversarial medical images for X-ray breast mass segmentation, Computer methods and programs in biomedicine, 2019, 180, 105012.

  41. Yang J, Liu S, Grbic S, et al.: Class-aware adversarial lung nodule synthesis in CT images, In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), 2019, IEEE.

  42. Doman K, Konishi T, and Mekada Y: Lesion Image Synthesis Using DCGANs for Metastatic Liver Cancer DetectionDeep Learning in Medical Image Analysis, 2020, Springer, pp. 95–106.

  43. Bhattacharya D, Banerjee S, Bhattacharya S, et al.: GAN-Based Novel Approach for Data Augmentation with Improved Disease Classification. Advancement of Machine Intelligence in Interactive Medical Image Analysis, 2020, Springer, pp. 229–239.

  44. Frid-Adar M, Diamant I, Klang E, et al.: GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification, Neurocomputing, 2018, 321, 321–331.

  45. Kaur S, Aggarwal H, and Rani R: MR Image Synthesis Using Generative Adversarial Networks for Parkinson’s Disease Classification, In Proceedings of International Conference on Artificial Intelligence and Applications, 2020, Springer.

  46. Sedigh P, Sadeghian R, and Masouleh MT: Generating Synthetic Medical Images by Using GAN to Improve CNN Performance in Skin Cancer Classification, In 2019 7th International Conference on Robotics and Mechatronics (ICRoM), 2019, IEEE.

  47. Jin D, Xu Z, Tang Y, et al.: CT-realistic lung nodule simulation from 3D conditional generative adversarial networks for robust lung segmentation, In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2018, Springer.

  48. Jafari MH, Girgis H, Abdi AH, et al.: Semi-supervised learning for cardiac left ventricle segmentation using conditional deep generative models as prior, In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), 2019, IEEE.

  49. Liao H, Tang Y, Funka-Lea G, et al.: More knowledge is better: Cross-modality volume completion and 3d+ 2d segmentation for intracardiac echocardiography contouring, In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2018, Springer.

  50. Chaitanya K, Karani N, Baumgartner CF, et al.: Semi-supervised and task-driven data augmentation, In International conference on information processing in medical imaging, 2019, Springer.

  51. Rezaei M, Harmuth K, Gierke W, et al.: A conditional adversarial network for semantic segmentation of brain tumor, In International MICCAI Brainlesion Workshop, 2017, Springer.

    Google Scholar 

  52. Gaj S, Yang M, Nakamura K, et al.: Automated cartilage and meniscus segmentation of knee MRI with conditional generative adversarial networks, Magnetic Resonance in Medicine, 2020, 84, 437–449.

  53. Saffari N, Rashwan HA, Herrera B, et al.: On Improving Breast Density Segmentation Using Conditional Generative Adversarial Networks, In CCIA, 2018.

  54. Sandfort V, Yan K, Pickhardt PJ, et al.: Data augmentation using generative adversarial networks (CycleGAN) to improve generalizability in CT segmentation tasks, Scientific reports, 2019, 9, 1–9.

  55. Liu Y, Khosravan N, Liu Y, et al.: Cross-Modality Knowledge Transfer for Prostate Segmentation from CT ScansDomain Adaptation and Representation Transfer and Medical Image Learning with Less Labels and Imperfect Data, 2019, Springer, pp. 63–71.

  56. Dong N, Kampffmeyer M, Liang X, et al.: Unsupervised domain adaptation for automatic estimation of cardiothoracic ratio, In International conference on medical image computing and computer-assisted intervention, 2018, Springer.

    Book  Google Scholar 

  57. Shin H-C, Tenenholtz NA, Rogers JK, et al.: Medical image synthesis for data augmentation and anonymization using generative adversarial networks, In International workshop on simulation and synthesis in medical imaging, 2018, Springer.

  58. Xue Y, Xu T, Zhang H, et al.: Segan: Adversarial network with multi-scale l 1 loss for medical image segmentation, Neuroinformatics, 2018, 16, 383–392.

  59. Enokiya Y, Iwamoto Y, Chen Y-W, et al.: Automatic Liver Segmentation Using U-Net with Wasserstein GANs, Journal of Image and Graphics, 2018, 6.

  60. Tu W, Hu W, Liu X, et al.: DRPAN: A novel Adversarial Network Approach for Retinal Vessel Segmentation, In 2019 14th IEEE Conference on Industrial Electronics and Applications (ICIEA), 2019, IEEE.

  61. Neff T, Payer C, Štern D, et al.: Generative adversarial networks to synthetically augment data for deep learning based image segmentation, In Proceedings of the OAGM workshop, 2018.

  62. Shi Z, Hu Q, Yue Y, et al.: Automatic Nodule Segmentation Method for CT Images Using Aggregation-U-Net Generative Adversarial Networks, Sensing and Imaging, 2020c, 21, 1–16.

  63. Decourt C, and Duong L: Semi-supervised generative adversarial networks for the segmentation of the left ventricle in pediatric MRI, Computers in Biology and Medicine, 2020, 123, 103884.

  64. Lei B, Xia Z, Jiang F, et al.: Skin Lesion Segmentation via Generative Adversarial Networks with Dual Discriminators, Medical Image Analysis, 2020, 101716.

  65. Shi H, Lu J, and Zhou Q: A Novel Data Augmentation Method Using Style-Based GAN for Robust Pulmonary Nodule Segmentation, In 2020 Chinese Control And Decision Conference (CCDC), 2020a, IEEE.

  66. Hamghalam M, Wang T, and Lei B: High tissue contrast image synthesis via multistage attention-GAN: Application to segmenting brain MR scans, Neural Networks, 2020a.

  67. Zhao M, Wang L, Chen J, et al.: Craniomaxillofacial bony structures segmentation from MRI with deep-supervision adversarial learning, In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2018, Springer.

  68. Yu F, Zhao J, Gong Y, et al.: Annotation-free cardiac vessel segmentation via knowledge transfer from retinal images, In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2019a, Springer.

  69. Shi X, Du T, Chen S, et al.: UENet: A Novel Generative Adversarial Network for Angiography Image Segmentation, In 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 2020b, IEEE.

  70. Kugelman J, Alonso-Caneiro D, Read SA, et al.: Constructing synthetic chorio-retinal patches using generative adversarial networks, In 2019 Digital Image Computing: Techniques and Applications (DICTA), 2019, IEEE.

  71. Hamghalam M, Wang T, Qin J, et al.: Transforming Intensity Distribution of Brain Lesions Via Conditional Gans for Segmentation, In 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), 2020b, IEEE.

  72. Dai W, Dong N, Wang Z, et al.: Scan: Structure correcting adversarial network for organ segmentation in chest x-raysDeep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, 2018, Springer, pp. 263–273.

  73. Xu C, Xu L, Ohorodnyk P, et al.: Contrast agent-free synthesis and segmentation of ischemic heart disease images using progressive sequential causal GANs, Medical Image Analysis, 2020, 101668.

  74. Goodfellow I: NIPS 2016 tutorial: Generative adversarial networks, arXiv preprint arXiv:170100160, 2016.

  75. Cortes C, Kabongo L, Macia I, et al.: Ultrasound image dataset for image analysis algorithms evaluationInnovation in Medicine and Healthcare 2015, 2016, Springer, pp. 447–457.

  76. Jaeger S, Candemir S, Antani S, et al.: Two public chest X-ray datasets for computer-aided screening of pulmonary diseases, Quantitative imaging in medicine and surgery, 2014, 4, 475.

  77. Suárez PL, Sappa AD, and Vintimilla BX: Colorizing infrared images through a triplet conditional dcgan architecture, In International Conference on Image Analysis and Processing, 2017, Springer.

  78. Cardenas CE, Yang J, Anderson BM, et al.: Advances in Auto-Segmentation, Seminars in Radiation Oncology, 2019, 29, 185–197.

  79. Flanders AE, Prevedello LM, Shih G, et al.: Construction of a Machine Learning Dataset through Collaboration: The RSNA 2019 Brain CT Hemorrhage Challenge, Radiology: Artificial Intelligence, 2020, 2, e190211.

  80. Heusel M, Ramsauer H, Unterthiner T, et al.: Gans trained by a two time-scale update rule converge to a local nash equilibrium, In Advances in neural information processing systems, 2017.

  81. Klambauer G, Unterthiner T, Mayr A, et al.: Self-normalizing neural networks, In Advances in neural information processing systems, 2017.

  82. Wu J, Zhang C, Xue T, et al.: Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling, In Advances in neural information processing systems, 2016.

  83. Pitaloka DA, Wulandari A, Basaruddin T, et al.: Enhancing CNN with preprocessing stage in automatic emotion recognition, Procedia computer science, 2017, 116, 523–529.

  84. Paszke A, Gross S, Massa F, et al.: Pytorch: An imperative style, high-performance deep learning library, In Advances in neural information processing systems, 2019.

  85. Thomas SJ: Relative electron density calibration of CT scanners for radiotherapy treatment planning, Br J Radiol, 1999, 72, 781–786.

  86. Ganesan P, Xue Z, Singh S, et al.: Performance Evaluation of a Generative Adversarial Network for Deblurring Mobile-phone Cervical Images, In 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2019, IEEE.

  87. Mahapatra D, Bozorgtabar B, and Garnavi R: Image super-resolution using progressive generative adversarial networks for medical image analysis, Comput Med Imaging Graph, 2019, 71, 30–39.

  88. Song TA, Chowdhury SR, Yang F, et al.: PET image super-resolution using generative adversarial networks, Neural Netw, 2020, 125, 83–91.

  89. You C, Li G, Zhang Y, et al.: CT Super-Resolution GAN Constrained by the Identical, Residual, and Cycle Learning Ensemble (GAN-CIRCLE), IEEE Trans Med Imaging, 2020, 39, 188–203.

Download references

Funding

The work is partially supported by the Winship Cancer Institute Pilot$ grant.

Author information

Authors and Affiliations

  1. Department of Biomedical Informatics, Emory School of Medicine, Atlanta, USA

    Jiwoong J. Jeong, Amara Tariq & Imon Banerjee

  2. Department of Radiology, Emory School of Medicine, Atlanta, USA

    Tobiloba Adejumo, Hari Trivedi, Judy W. Gichoya & Imon Banerjee

Authors
  1. Jiwoong J. Jeong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Amara Tariq
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tobiloba Adejumo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hari Trivedi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Judy W. Gichoya
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Imon Banerjee
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Not applicable.

Corresponding author

Correspondence to Jiwoong J. Jeong.

Ethics declarations

Ethics Approval

Not applicable.

Consent to Participate

Not applicable.

Consent for Publication

Not applicable.

Conflict of Interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 482 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jeong, J.J., Tariq, A., Adejumo, T. et al. Systematic Review of Generative Adversarial Networks (GANs) for Medical Image Classification and Segmentation. J Digit Imaging 35, 137–152 (2022). https://doi.org/10.1007/s10278-021-00556-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-021-00556-w

Keywords

Search

Navigation