Deep Learning of Entity-Guided Representations in Digital Pathology

Abstract

Pathological examination is the gold standard for cancer diagnosis, prognosis, and therapeutic response predictions. Advancements in scanning technologies and an increased focus on precision medicine have paved the way for developing digital-pathology-based assessments. Digital pathology has enabled the digitization of microscopy slides into high-resolution whole-slide images and opened up opportunities for computational pathology (CP). CP aspires to alleviate the cumbersome and time-consuming routine workflow of pathologists by introducing computer-aided assistive tools. To this end, CP leverages computational techniques for automated exploration and extraction of meaningful information from histopathology images. The demand for CP has recently gained even more attention due to the growing incidence rate of diagnostic cases per year. The basis of a typical CP system is artificial intelligence, in particular, deep learning (DL) due to its recent large-scale success. Capability of DL to automatically extract and utilize informative representations from complex histopathology images in a data-driven manner have popularized its adoption in CP. Several DL methods have been developed to address various histopathology tasks, such as nuclei detection and characterization, tumor delineation, tissue grading and staging, and survival estimation. However, the clinical adoption of DL methods is inhibited by several challenges, including: (1) infeasibility of acquiring large high-quality annotated histopathology datasets for training models; (2) requirement of prohibitive computational resources for processing large whole-slide images; and (3) a lack of transparency and interpretability of DL decisions. Further, most DL models in CP are built based on convolutional neural networks (CNNs), which treat an image as a composition of multisets of pixels, to perform analyses in a pixel-paradigm. However, operating in pixel-paradigm induces several crucial bottlenecks, such as: (i) not being able to easily utilize tissue composition and well-established prior pathological knowledge, due to a disregard for histological entities, e. g., nuclei, cells, glands; (ii) an inability to simultaneously capture both local cell microenvironment and global tissue microenvironment; (iii) intensive computational requirements for operating on large whole-slide images; and (iv) non-straight-forward model interpretations due to the trained models not making diagnostic decision explicitly based on well-defined histological entities. This thesis aims to address the aforementioned challenges and limitations concerning DL methods in CP. The motivation herein is that the analysis of tissues should rely on the phenotype and topological distribution of their constituting histological entities. Therefore, the analytical paradigm is proposed to be shifted from conventional pixels to entities. A histopathology image is first transformed into an entity-guided representation, specifically an entity-graph. The nodes and edges of the graph denote comprehensible histological entities and entity-to-entity interactions, respectively. The local entity-level phenotypical properties are embedded in the nodes and the global tissue-microenvironment is captured by the graph topology. Subsequently, the advancements of DL techniques on graph-structured data, in particular Graph Neural Networks (GNNs), are leveraged to efficiently construct a relation-aware entity-graph-representation for addressing downstream histopathology tasks. Operating in the entity-paradigm enables the incorporation of task-relevant entity-level prior knowledge for comprehensive tissue modelling. Entity-graphs being more flexible and memory efficient, compared to pixel-based counterparts, can scale to images of arbitrary shapes and sizes. Further, interpreting an entity-graph-based model can highlight salient entities and interactions for model decisions, which the pathologists can directly comprehend. Relevance and superiority of learning on entity-guided tissue representations are established for a variety of histopathology tasks across several tissue types. The proposed entity-graphs encode different entity types, i. e., nuclei, tissue regions, and both; and include different graph topologies, i. e., uni-level, multi-level, hierarchical. Further, various entity-guided GNNs are proposed herein to tackle the challenges of: (1) learning from weak supervision and limited annotations; (2) processing histopathology images of arbitrary sizes; and (3) interpretability and explainability of model decisions in pathologist-friendly terminologies. Specifically, the proposed methodologies are applied for the following histopathology tasks: (a) supervised subtyping breast carcinoma tumor regions, (b) weakly-supervised simultaneous classification and semantic segmentation of prostate cancer needle biopsies, and (c) generating qualitative and quantitative interpretations of breast subtyping model decisions. The proposed methods achieve state-of-the-art performance for these tasks, and have been validated by domain-expert pathologists. The generalization ability of the proposed methods is also substantiated by classifying and segmenting prostate cancer biopsies from multiple data sources. In addition, a flexible open-source python library, HistoCartography, has been developed to facilitate effective graph analytics in digital histopathology.