Affinity-based color enhancement methods for contrast enhancement in hyperspectral and multimodal imaging

This work proposes separating data analysis from hyperspectral enhancement or editing, providing a robust, context-independent, fully-tunable framework for biomarker-based contrast in wide-field imaging with a series of reliable properties that could enable its use in guided surgery. Some applications of this method powered by deep learning diagnostics will be discussed and shown.


Introduction
In this particular context, the term hyperspectral enhancement refers to the alteration of natural spectral properties in a hyperspectral image (HSI) so that their reconstruction to a visualizable domain highlights or enhances pixels with specific molecular or spectral properties. Current state-of-the art methods for image enhancement apply Principal Components Analysis (PCA), which is based on the Singular Value Decomposition (SVD), as part of their workflow [1][2][3]. The SVD is an operator with a provable unique solution. This unfortunately means that imaging contrast will be significantly dependent on what is present in the hyperspectral, fluorescence or multimodal images, which may hinder their translation into a clinical domain.
In this contribution, some examples of how the SVD could potentially damage image contrast are shown, and an improved method, inspired by previous algorithms, is demonstrated. Instead of extracting what should be enhanced with PCA, a relevant magnitude of interest must be first found. Then, a colorimetric method with tunable enhancement gain, contrast, and threshold is provided, essentially mimicking the previous methods but without this sensitivity to nonessential information in the image. Its resilience and robustness may prove its potential applicability in the field of guided surgery [4].

Materials and methods
Imaging systems Two different imaging systems were used to produce molecular imaging data, namely (1) a hyperspectral rotating mirror system [5] and (2) a Perkin-Elmer IVIS CT machine with a retrofitted Spatial Frequency Domain Imaging (SFDI) device [6]. Both systems are wide-field molecular imaging devices, with the following fundamental properties: • The HSI system is a far-field rotating mirror scanning system capable of an FOV of about 15 × 15 cm, a resolution of 1024 × 1024 pixels, and 218 channels in the Vis-NIR range (400-1000 nm, about 3 nm per channel). The device is used in two experiments, namely (a) the acquisition of HSI data of a healthy subject's hand, and (b) closed-loop quantification of Intralipid vs. red ink concentrations.
• The SFDI machine allows for 1024 × 1024-pixel images, an FOV of circa 10 × 10 cm, and a total of four spatial frequencies (0.0, 0.1488, 0.6053, 1.3736 mm −1 ) and four wavelengths (490.0, 550.0, 600.0, and 700.0 nm). The objective of this device is to enable surgical margin assessment in Breast Conserving Surgery (BCS) specimens. Hyperspectral enhancement Hyperspectral enhancement consists on editing spectra prior to spectral reconstruction. By spectral reconstruction we refer to the transformation of Vis-NIR spectral data to representable RGB data, by means of Color Matching Functions, which attempt to imitate the spectral sensitivity of the human retina [7]. This is usually approximated by Riemannian sums: where r = (r 1 , . . . , r l ) T is a vector of reflectance data, s = (s 1 , . . . , s l ) T is the power spectral density of a CIEstandardized illuminant, andx,ȳ,z are the Color-Matching Functions (CMFs) for a secondary light source (i.e. reflected information). Finally, k is a normalization constant, k = 100/(∑ l i=1 s iȳi ∆λ ). The constant ∆λ is the spectral resolution.
Hyperspectral enhancement attempts to exaggerate the specific spectral properties of a particular spectrum r by a particular function r = f (r; θ ). Here, we denote r as the enhanced spectrum, achieved via an enhancement method f (r; θ ), with θ a series of specific parameters. In essence, if θ is not an objective quantity and/or depends on elements of the image that are non-essential, then its ability to highlight specific spectral signatures will be hindered.
SVD-based enhancement A common method for spectral anomaly detection uses the SVD, or Singular Value Decomposition, of the spectra present in a given HSI image. These methods obtain an enhanced spectrum via the expression [1,3]: and here r i represents the original spectrum (as a column vector), r i is the enhanced spectrum and s i is an m-rank approximation of spectrum r i : with {u j } the left singular vectors of matrix R = (r 1 | . . . , |r n ) with n spectra as column vectors, α i j = r i −r i , u j is the projection of r i −r onto the j-th left singular vector, andr = 1 N ∑ n j=1 r i is the average spectrum. The matrix W is a diagonal matrix with all elements set to zero except for the wavelengths of interest. Essentially, the m-rank approximation of the spectrum is subtracted to the original data so that the differences that deviate from the most frequent spectra at specific wavelengths can be exaggerated. This limits the method to only consider statistical anomalies, which are largely dependent on the number of similar or dissimilar spectra present on a given image or, as can be demonstrated, on background elements not pertaining to the specimen of interest [4]. Figure 1. Three types of spectra 1.(a) are synthetically generated: Types A, B, and C. We will try to enhance the reflectance valley about 580 nm of Type C spectra with the traditional method. In the first scenario, 200, 200, and 50 spectra with AWGN noise are generated for each type, respectively (as shown in 1.(b)). The 'rare' spectrum for such scenario is Type C. The result of applying the SVD to these 450 spectra results in 1.(c), where the first singular vector partly explains the peaks of Type A and B spectra (at 420 and 480 nm), but not the peaks of Type C. Considering m = 1 for the m-rank approximation, we obtain that, as a result, the most different spectrum is Type C 1.(d), which is then properly enhanced 1.(e). However, this contrast is significantly reduced when we provide the SVD with 50, 200, and 200 spectra of Types A, B, and C, respectively. For this second scenario (1.(f) through 1.(h)), the least common spectrum is Type A, which affects enhancement at 580 nm for Type C spectra. This dependence on the relatives ratios of spectral signatures within an image may hinder its applicability in an environment with varying context and in constant movement.

Context-dependence of SVD-based methods Consider the examples shown in
Affinity-based color enhancement (ACE) To achieve a flexible and robust hyperspectral enhancement method, quantification or detection of anomalies must be achieved separately from the modification of the spectral data. Therefore, prior to application of the method, a proper affinity function or mapping must be found. An affinity map {a i } is simply a continuous function that translates the provided data to a map of what should be highlighted. For the experiments showcased below, deep neural networks are trained to obtained the desired diagnostic probability maps. More specifically: • Vein enhancement. Vein enhancement required a 6 × 100 ELU multilayer perceptron (MLP) network, with a 218 × 1 input layer and a 3 × 1 sigmoidal output layer. The three output categories are: (1) Background, (2) Skin and (3) Veins. A reduced dataset of 600 spectra is produced, while the rest of the pixels (and other images) are left as test data.
• Chromophore quantification. A network of identical dimensions is set up, but with an 8 × 1 sigmoidal output layer. Closed-loop quantification is achieved by preparing a training set of cuvettes with known concentrations, and a test set with 8 different concentrations: 0%, 10%, 20%, . . . , 70%. The concentration c i of a given spectrum is interpolated via c i = ∑ 7 j=0ŷ i × (i/10), withŷ 0 , . . . ,ŷ 7 representing the eight output units, which are made to activate for red-ink-to-Intralipid ratios of 0% through 70%, respectively. A specific concentration ratio can be targeted with an expression such as a i = σ (− log 10 (h 2 (c i − c ob j ) 2 )), with c obj the objective concentration, h > 0 a desired bandwidth and σ (.) the sigmoid function [4].
• Margin delineation in SFDI This application requires an ensemble of convolutional neural networks, each of which provides a map of tumor probability and/or malignancy. The ensemble produces an average map of tumor probability, which is then used as the affinity metric of interest.
Then, the spectral information is edited via applying r e,i = r i (1 + K · c e · tanh (k (â i + b))), where r e,i is the enhanced output, r i is the original spectrum, K will be is the enhancement gain of the method, c e is the desired output color of the enhancement, tanh(.) is the hyperbolic tangent function, k is a contrast gain, b is a threshold value, andâ i is the affinity measure value of the spectrum when all the spectra are normalized in the range [0, 1], namelyâ i = a i −a min a max −a min . The values a min and a max should be known beforehand, by selecting an appropriate affinity metric. Figure 2 provides a general description of the various applications achievable with this method. In the top row, reflectance at 850 nm allows for the acquisition of spectral signatures typical of veins and skin. This data can be used for training, obtaining a probability map for veins in the rest of the image (center). This vein map shall remain constant regardless of the properties of the background and/or hand orientation, as long as the scatter signatures of veins and skin remain constant, which is not the case for SVD-based traditional methods [4].

Experimental results
The center row shows the second experiment, pertaining to closed-loop inspection of specific chromophore quantification, demonstrating how deep-learning-powered quantification combined with ACE can provide custom-color contrast for only cuvettes with 40% red ink-to-Intralipid concentrations. While a trained human may be able to manually detect imprecise or incorrect chromophore or fluorophore concentrations, an automated model can provide an aid for untrained individuals, or enhance low-SNR fluorescence signals. The output color, contrast and gain of the enhancement can be set at will, in real time, depending on the needs of the end user. Finally, the bottom row presents an example for margin assessment and delineation of a lumpectomy (left) with an ensemble of deep neural networks. This last example is much simpler, and the affinity function (center) used for enhancement is just a tumor probability map.
Each application has a different configuration for K, k and b, so that the representation can be dependent on the user's objective. Those parameters can be modified in real time, providing additional flexibility. With these particular traits, the method could perhaps be useful in a variety of applications related to hyperspectral imaging, modulated imaging and fluorescence-guided surgery, improving contrast as long as an adequate, clinically-relevant affinity function can be found.  Fig. 2. Examples of ACE for contrast enhancement. Top row: vein detection algorithms can be merged with the original spectral data to provide an RGB reconstruction with better contrast. Middle row: chromophore quantification estimates can also be used; here, 40% ink-to-Spectralon cuvettes (fifth cuvette from the left in the validation set) are highlighted. Bottom row: enhancing margin contrast of a Breast Conserving Surgery sample by processing modulated imaging data with a convolutional neural network.

Conclusions
In this contribution, some issues regarding current hyperspectral enhancement methods have been discussed. The proposition of a novel method [4] that provides identical results with consistent output shows promise in terms of applicability to clinically useful scenarios such as vein delineation, chromophore concentration detection, and surgical margin assessment of lumpectomies.