Author : Mr. Chen Hongjun
Last Updated : 17 July 2007

Soft Tissue Tracking

Introduction

Advanced Mammotome System and Margin Tracking

One of the most critical and challenging problems in the Advanced Mammotome System project is to track the deformation of tumor margin during the minimal invasive surgery (MIS) lumpectomy.  

In this vacuum-assisted surgery, the only way for the surgeon to monitor the surgical needle tip and the cutting process is through a real-time 3D ultrasound machine. However, due to similar ultrasonic properties of tumor margin and the surrounding healthy tissue, the margin can not be perceived in the ultrasound images.  

The objective of this research is to design an effective algorithm to be integrated into AMS, which is being developed in out lab to give the surgeon enough margin deformation information throughout the whole lumpectomy procedure. 

Here are the basic facts about this MIS lumpectomy procedure:

·        Breast tumor and margin are removed step by step, with a Mammotome needle inserted into the proper position in the breast.

·        The must-cut margin cannot be detected by the ultrasound machine. It is usually a virtual margin delimited by the surgeon according to the shape of the tumor.

·        The surgery requires one insertion, but multiple cutting steps (varying from patient to patient).

·        The doctor requires the tumor should be first removed, then the margin for the safety consideration. Thus the procedure is divided into two stages: tumor removing and margin removing.

·        In the tumor removing stage, the tumor deforms after each cutting, so does the margin. The margin still can be delimited according to the tumor shape, so it’s easy to locate the margin.

·        It comes to the margin removing stage after the entire tumor is removed. In this stage, the margin also deforms while being cut. But it cannot be delimited any more, since the tumor has gone. It is the most challenging job to find the position of the margin in this algorithm research.

Margin Tracking Approach via Image Registration

Our current approach to track the margin deformation is through image registration method. All the information used to locate the margin position is within the ultrasound image volumes. 

·        Before the surgery, the margin can be delimited according to the tumor shape. After the Mammotome needle is properly positioned (under the tumor and inside the margin, see Fig. 1). The margin should still be observed. It can be called the initial margin.

Figure 1.   Needle positioning

·        The surgeon starts to cut. His objective is to cut the tumor and the margin clearly and keep healthy tissue as much as possible. So whatever he cuts will be constrained inside the initial margin surface.

·        During the surgery, certain quantity of tumor/margin tissue is removed after each cutting. The margin keeps deforming due to the cutting and the internal/external pressure. However, the tissue outside of the initial margin won’t be cut or removed.

·        For the two image volumes before and after one cutting (say, the first cutting), there should be some kind of correspondent relation for the tissue outside of the initial margin. They stand for the same volume of tissue and the margin after the cutting actually is the deformed version of the one before the cutting.

·        We can try to find some kind of TRANSFORMATION that can transform the image volume of the tissue outside of the initial margin before the cutting to the one after the cutting. Then we can apply this transformation to the margin and obtain the position of the deformed margin (see Fig. 2).

            Figure 2.   Transformation (registration) according to the tissue outside of the margin

·        The procedure of finding the transformation essentially is image registration. For our project, it’s non-rigid image registration for inner structure position tracking from image volume series.

·        The advantages of this approach include: (1) the position tracking is totally based on the image information of the particular patient, no need to worry about the number of patient cases. (2) No need to consider the correspondence problem for statistical analysis. (3) The variance caused by the ultrasound probe, stabilizer, and Mammotome needle has no influence to the result. In one word, the finial objective is simplified and concentrated to one problem. That is intra-object intra-modality non-rigid image registration for inner structure (surface) position tracking.

Medical Image Registration 

Medical image registration is an important pre-processing step for many image related medical applications. It is defined as the process of aligning images so that the shape, structure, size, and spatial relationships of corresponding anatomical structures in two images can easily be matched or related, as shown in Fig 3. This procedure is a spatial mapping. 

Figure 3.  Image registration: the correspondence between 

point A and point B is found after registration.  

A generic medical image registration problem is solved in three steps:

·        Defining registration transformation according to the expected nature of tissue motion;

·        Defining registration basis (similarity measure);

·        Finding optimal/sub-optimal transformation parameters.

Accordingly, there exist at least three classification criterions for registration algorithms:

·        The first criterion is the type of transformation. There are rigid, Affine, projective, curved transformations.

·        The second criterion is similarity measure. There are non-image markers such as external fiducial landmarks, image-based similarity measures and intensity-based measures.

·        The third criterion is the method used to find the optimal/sub-optimal transformation parameters. 

In medical applications, the difference between the source image and the target image can be due to variable reasons. The deformation can be large and nonlinear. In this paper, a new framework for medical image registration with large nonrigid tissue deformations is proposed. Registration problem is formulated as to recover the deformation process from the source image to the target image. 

REGISTRATION USING VIRTUAL FRAMES

Image Registration vs. Optical Flow

Image registration is to align two images so that corresponding structures can be related. The objective is to find the structure correspondence or pixel correspondence. Optical flow is to study the motion of image contents within a sequence. The procedure of determining optical flow can be regarded as the nonparametric image registration.

The most important assumption for optical flow is that the intensity of corresponding pixels in the sequence does not change. Image content change is only the results of geometrical transformations. 

Registration Considered as Recovering Deformation Process

For intra-subject mono-modality image registration, the source image and target image are usually from a sequence. It is reasonable to relate them by deformations caused by certain forces. Therefore, image registration can be regarded as to recover the deformation process with source image as the initial state and target image as the final state.

By taking the time parameter into consideration, a 2D image in can also be modeled as a discrete but differentiable brightness function, . 

The source image and the target image are respectively denoted as: and . 

The geometric transformation can be represented in a nonparametric form, , 

where vector is the displacement in x and y directions.

In an ideal registration, there should be .

Therefore energy function

for each pixel should be minimized to find the optimal registration. It is under-determined because there are two unknowns in this single equation.

This is consistent with the idea of considering registration as to a process of recovering deformation. Given only initial state and final state, there are infinite paths leading the source to the target, as illustrated in Fig 4.  To solve such an ill-posed problem, constraints must be specified to find the optimal path from the source to the target.

Figure 4.   Transformation that maps source image to target image is not unique

CONSTRUCTING VIRTUAL FRAMES WITH AFFINE TRANSFORMATION

To register images with large deformation, virtual frames are proposed to be inserted between the source image and the target image to simulate the deformation process as illustrated in Fig 5. These virtual frames serve as the milestones of the deformation.  

Local Affine Transformation for Registration

Instead of assuming free pixel displacements, it is favorable to constrain pixel motion to be affine,, where is the pixel position in homogenous coordinates;and A is the affine transformation matrix.Therefore, the source image and target image can be denoted as: and , respectively.

By choosing sum of squared error as the similarity measure, the energy function to be minimized is,, where stands for all the pixels in the image. For an image with N pixels, N local affine transformations are to be found, which leads to a dense deformation field with the ability to manipulate pixels locally.

Smoothness Constraints for Image Registration

A reasonable assumption on tissue deformation is the smoothness and continuousness of the deformation. The motion of neighbouring pixels should be similar and change should be gradual. This leads to another energy function to be minimized,.

Thus the registration solution can be uniquely found by minimizing the sum of energy function of similarity measure and that of smoothness constraint. By differentiating energy function E=E₁+E₂ with respect to affine parameters and letting the result be zero, the parameters can be iteratively found.

Figure 5.   Source image, target image and virtual frames  

In practice, the parameters are first initialized. After the first iteration, the parameters are updated and applied to the source image to produce the first virtual frame. The parameters are updated again according to the difference between this virtual frame and the target image. By repeating this procedure an image sequence containing the source and target images and the virtual frames can be obtained, as shown in Fig 5.

In summary, image registration is regarded as to recover the actual deformation process from the source to the target under our framework. Virtual frames are of key importance in this procedure. The adapted optical flow technique is used to generate the virtual frames. To solve the optical flow equation, this paper requires the affine parameter to be spatially smooth. In this way, the inherent iteration of optical flow is used as a technique to uniquely generate virtual frames between the source and the target.

EXPERIMENTS AND RESULTS

Currently, three kinds of images are conducted to demonstrate the effectiveness of the proposed algorithm.  

The first image pair to be registered is synthetic images with large nonlinear deformation. The initial SSD between the source and the target is 0.286. During the registration process, the SSD decreases to 0.152, 0.109 and 0.039 at the 12^th, 24^th and 36^th virtual frame, respectively. The gradual deformation process is shown in Fig 6.  

Figure 6.   Experiment on sythetic images with large deformation: 

a-source, a’-target; b, c and d are the 12^th 24^th and 36^th virtual frames respectively

Figure 7.   Experiment on MRI images: a-source, a’-target; b and b’ are the 100^th virtual frame and its deformation field; c and c’ are registered source and the final deformation field (red ‘x’ stands for feature in source image; green ‘+’ stands for the corresponding feature in target image)

The other two experiments are conducted with brain MRI images and breast ultrasound images. Synthetic nonrigid deformations are used to quantitatively evaluate the accuracy. The deformed images are to be registered to their original versions. The correspondence of salient structures is determined manually. The results are evaluated with two criterions, intensity sum of squared difference (SSD) between the transformed source image and the target image, and the pixel SSD between the transformed feature points and their ground truth. 

In the MRI brain image experiment, the corresponding feature points are marked in the source image (red ‘x’) and the target image (green ‘+’). In the registered source image, these feature points are related well as shown in Fig 7, together with the deformation field. One virtual frame is also presented. The intensity SSD, feature point SSD of before, during and after registration are listed in Table 1. 

In the ultrasound image experiment, the feature points are labelled as the boundary of one fibroadenoma in favour of the potential application of registration techniques in image segmentation. The results are shown in Fig 8 and listed in Table 1. It can be seen that as the number of virtual frames increase, the registration performance improves and the intensity SSD, feature point SSD keep decreasing from source image to target image through virtual frames.

Figure 8.   Ultrasound images (the legends are the same with those in Fig 7)

          Table 1. Intensity SSD and feature point SSD before, during and after registration

The number of virtual frames needed in the registration procedure depends on the complexity of the images to be registered. In our experiments on the synthetic images, about 40 virtual frames are enough to obtain good results. For the real images, the number is about 100. The sizes of the images used are 96×96, 256×256 and 256×192 for the three experiments respectively. The computation time used is about 42s, 215s and 316s respectively in the three experiments (Dell workstation, Pentium IV 2.4GHz, RAM 512 M, WinXP Pro and Matlab 7.0 R14).

DISCUSSION

In this research, a new framework for medical image registration with large nonrigid tissue deformations is proposed, in which registration problem is formulated as to recover the deformation process from the source image to the target image. A time parameter is introduced into in this procedure. To model large nonlinear deformation, the adapted optical flow is used to generate virtual frames that serve as the milestones in the deformation process. In the image sequence, the deformation between any two consecutive virtual frames is modeled with local affine transformation. To ensure the uniqueness of solution, the transformation parameters are required to be spatially smooth. Experimental results demonstrate that this framework is effective. 

Compared with the popular multi-scale methods in space domain, the procedure of inserting virtual frames can be regarded as to increase time resolution in time domain. The future work will be focused on stability problem and the smoothness constraints of the parameters in time domain. The deformation parameters are actually also function of time in the image sequence. With the temporal smoothness constraints when constructing virtual frames, better registration results are expected. In the future we are going to integrate this algorithm into our AMS and further experiments will be conducted and examined thus to help this system come into clinical trial for benefit of both patients and doctors. 

Status

The project is funded by National Medical Research Council (NMRC) and started in August 2002. The participants include Tan Tock Seng Hospital and Nanyang Technological University.

Publications related to Soft Tissue Tracking.

We would be glad if you could sign our guest book.

For more information, please contact the principal investigator:

A/P Ng Wan Sing
School of Mechanical & Aerospace Engineering
Nanyang Technological University
Nanyang Avenue, Singapore 639798
Fax:(65) 6791 1859