In the modern medical industry, highly-skilled experts such as doctors or diagnosticians are required to perform diagnoses/analyses using images of tissue samples. We attempt to automate this process by using the CLIP framework to create an image-captioning tool that can be used to retrieve useful textual pathology notes given histology images. We implement a modular CLIP framework and experiment with candidate choices of encoders. We conduct qualitative and quantitative analyses of the efficacy of our trained models. We demonstrated the ability of our model to identify the pathology of a given tissue image from unstructured data.

report