Detection of facial areas in ThERMAL IMAGES

Summary

This article deals with non-contact measurement of temperature in a human faces and describes program for the evaluation of temperature changes. It describes the algorithm of the program, the possibility of using and further deals with the possibilities of the segmentation of thermal images. Variety of image processing methods were used to design this algorithm including registration of images, segmentation using k-means clustering, Hough transformation, thresholding and others. The aim is to distinguish a human face from the hair and background. It also describes the possibility of detection of individual facial details. The functionality of those procedures was tested on experimental data.

Keywords:
Non-contact temperature measurement, thermal imager, thermogram registration of the images, image segmentation

Introduction

The current trend in medicine is increasing amount of the information that is obtained for diagnostic purposes. With this fact are associated higher demands that are placed on the technique use to support the evaluation of the obtained data. Especially in the field of radiology and image processing was made the great progress in recent years and physicians in general practice have a variety of automatic or semiautomatic tools that help ensure high reliability and efficiency of diagnostic centres.

An important part of the diagnostic apparatus are imaging systems, especially active imaging systems which transmit some energy to the organism, and according its interaction with the tissue is obtained information about the status of this tissue. Active systems usually represent a health risk to the patient, or are associated with large financial costs. Infrared imaging systems are the contrary passive imaging systems, their use does not pose any risk to the patient and are relatively inexpensive compared to other imaging systems. They only scan infrared radiation, which is emitted by the patient and provides information about surface temperature. The temperature is one of the important diagnostic indicators. A number of pathological processes is associated with changes in temperature, whether local or global. Therefore, infrared imaging systems represent a promising tool for the diagnosis. Although good results have been already achieved in certain applications, their practical use is substantially limited (for example breast cancer diagnosis, monitoring restoring limb perfusion after vascular interventions).

This article describes a program that could serve as an auxiliary diagnostic tool for detecting temperature changes in human faces. This program could be useful for evaluating certain types of allergic reactions, the onset of fever and monitoring of time characteristics of such processes.

The program description

The input of the program is a sequence of thermal images capturing faces in different time moments. Sampling takes several minutes and in total is obtained about ten images. Thermal imager is fixed on a tripod, but the person is not fixed. Despite the fact that a person is asked to do not move, some movements can occur in a scene. This move would cause serious inaccuracies in the final analysis, so first of all is necessary to perform the precise registration of each image. First image in the series serves as a reference image and all the others will be aligned to the reference one.

It is necessary to select area in which the temperature changes will be monitored across all images. However, it was not found any specific area where all subjects showed a change in temperature if some intervention occurs (e.g. physical work) and therefore it is followed the temperature change of the whole faces. It is necessary to segment the images for the analysis. It has to be distinguished face from the hair and background. Then it is necessary to remove areas of faces, which have a significantly different temperature than the rest of the face and also the temperature change in these areas are not so much influenced by physiological processes. It means especially eyes, nose and eventually mouth. The temperature of these zones is individual for each subject.

Experimental images were used for implementation and testing of the program. Simulation of temperature change was made by the icing of faces.

Image registration

The task of registration is thus to eliminate motion in a scene. The open source software Elastix was used for registration [1]. The input is a fixed image, moving image, and the file with setting of the parameters of registration. Fixed frame serves as a pattern according which all others images are transformed so that the corresponding area is located at the same positions in all transformed images. Fig. 1 shows an example of a successful registration. Picture a) shows the overlap of the two images before the registration. It is possible to see that the face in each frame are shifted relative to each other, it is shown by the presence of purple and green colors in the image.

Picture b) shows the overlay of images after registration. It is seen that in this case green and violet colour appears substantially only in the area of the face. This is caused by temperature change, not by the movement of examinated person. Also there is the presence of violet color in the peripheral part of the image that is caused by deformation field of the registration. This region is outside the region of interest and can be easily removed from the further processing.

Segmentation

It is necessary to separate as best as possible the area of interest for the purposes of the program. In this case, the face of the person distinguish from the background and hair. Segmentation is done by using the k-means clustering [2]. The pixels in the image are clustered based on their similar properties. The pixels are first converted to the colour space L*a*b*, where L* is the lightness, a* is the location in the red-green axis, b* is the location in the blue-yellow axis [3]. The face may be in comparison with hair considered as homogeneous area, therefore, was determined for each pixel even local entropy. It can be described as the level of information contained in a message, in this case in the pixel. Local entropy is for a given pixel calculated from the surrounding pixels. Entropy is high in the parts of image where there are edges and some minor texture, and contrary entropy is small in uniform homogeneous areas.

Each pixel is thus described by the vector of properties that contains information about positions in colour space. Clustering has been empirically set into seven clusters, while the area of interest contains only clusters representing the human face. These clusters are usually two and are found thus that their centroids are close together and close to the centre of the image. The average value of each clustering parameters are close. They differ significantly just for the value of local entropy. These obtained clusters are shown in Fig. 2. Segmented face is obtained after merging these clusters, see Fig. 3a). This cluster is further processed thus that pixels, that are not associated with facial parts, are removed. Furthermore, the part inside the face, which had a significantly different temperature, but may represent a region of interest, are returned back into the region of interest, for example cold cheeks. Compact segmented face is obtained after these adjustments, see Fig. 3b).

 

Detection of the facial details

It is necessary to detect some facial parts, whose temperature is significantly different than the rest of the faces and remove them from further processing at this moment. These are eyes, nose, and optionally mouth. Eyes must be removed in any case. Eye corners are generally the warmest place in the face, and the temperature does not change significantly within a short period of time. Also, a person may blink during the experiment, causing that algae are more visible. This would be manifested as sharp decreasing of the temperature in the region, which has nothing to do with the physiological manifestations.

The nose is often found at much lower temperatures, especially for girls and women, as is shown in Fig. 3b). In addition, this region may have different time characteristics than the rest of the face. For example cold nose can heat after coming into a warm room slower than the rest of the face. For these reasons, the nose is removed from the area of interest only in the case that has significantly different temperature at the beginning of the experiment.

Similar reason applies to the mouth. These were often at obtained experimental frames significantly at higher temperature, see Fig. 3b). Again, a person can slightly open mouth, resulting in a false positive reduction of temperatures in the monitored area.

Eyes detection

Eyes are usually detected first for basic orientation in the image. Presence of almost perfectly circular shape of iris and cornea in a concentric position can be used in the ordinary photographs, but these structures are not visible in thermo images and thus a different approach has been performed.

The eyebrow has shape of circular segment is clearly visible and in thermo images and therefore Hough transform (HT) of semicircles can be used for eyebrow detection. First, edge detection is performed by using of Canny detector, see Fig. 4a) and then the HT detecting eyebrows. The result of HT with the rendering of a half-circle that best approximates searched eyebrows is shown in Fig. 4b).

The problem remains physiological asymmetry of eyebrows. Moreover, both eyebrows may be due mimic in different shapes and some persons have an upper eyebrow line in more circular shape or contrary some of them lower eyebrow line. For these reasons, even if the algorithm for detection of eyebrows is reliable, does not provide for all people sufficiently accurate information about eye position. Therefore, this detection of the eyebrows is only used to find the frame, which certainly include sought eyes. Cutout area where further search of eye will be performed, was chosen based on information about the position of the centre half-circle approximating eyebrows and its radius.

The fact, that the inner corners of the eyes are generally the warmest place in the face, is used for more accurate detection of the eye and therefore can be found by simply thresholding of the image. The advantage is that the heating of these corners is symmetric and therefore point representing the corner of the eyes can be determined as the centroid of the area exceeding the specified threshold. It is necessary to find also outer corners of the eye for further purposes of this work. Again, it was found that even in the outer corner of eye there is a place, which has higher temperature than surroundings to some extent, and thus the outer corner can be found analogously to the interior. The results are four points representing eyes.

As already mentioned, it is necessary to mask the eyes and eyebrows, so that it does not enter into the analysis. Each eye is masked as shown in the picture Fig. 4b) by the ellipse. Since the position of the inner and outer corner is known, it is possible to calculate the centre of the eye. Centre of eye represents a cross in Fig. 4a).

Size of masking major axis of the ellipse is determined by the distance between the corners, minor axis size is calculated based on the size of the radius of the circle approximating eyebrows.

Detection of other facial details

Fitting the points representing the inner and outer corners of the eyes by the line is obtained horizontal axis of the face. Vertical axis of the head can be obtained by finding perpendicular line to the horizontal axis, which goes through the calculated point, which lies between two eyes, and thus represents the root of the nose. This axis respects tilt of the head of the person and passes through an additional searched structures nose and mouth.

As already mentioned, these areas must be removed only if they have substantially different temperatures. If this is indeed the case, these places were already included during the initial clustering to another cluster than the one representing the face, see Fig. 3a). Position of inactive pixels located inside the facepiece have been saved and now is tested whether facial axis goes through such area. In case that yes, these areas are identified and removed from further processing. The result of these modifications shows Fig. 4b). There is further marked axis of the head and eyes masked by using ellipses. Thus processed image is the result of preprocessing and allows the analysis itself.

Data

The test set consists of thermal images taken with a thermal imager FLIR B200. Obtained images have the resolution 320x240 px. Temperature change in the face was simulated by cooling faces by using ice cubes.

It was measured in a total nineteen time series, each time series contained on average 13 shots and time of monitoring was about 2 min.

Analyses

The output is a software that can be used for detection of sudden temperature changes over time in the face. It would be possible to apply this software in case of allergic reactions to certain drugs from a practical point of view. Allergies may exhibit dilation of vessels and thus increasing the blood flow in the monitored area, causing a temperature increase. Another possible use is the evaluation of cold test which helps assess the overall status of the cardiovascular system. It monitors the dynamics of cardiovascular system, with which the body copes with temperature changes and can therefore reveal nonspecific pathology of the heart and blood vessels. In case of pathologies can be affected in fact not only the dynamics with which the area has been re-warming, as for example a significantly longer time, which the body needs to be returned to the original temperature, but also symmetry, for example, if one face warms much faster than the other. By the using of the software you can evaluate the development of fever during illness, but the problem would be to ensure stable environmental conditions because measurement should be carried out in this case after several hours. The software could be useful also in demonstrating sinusitis or inflammation of teeth, since inflammation is also reflected in the local temperature change, and therefore the affected area in the images seemed warmer.

The temperature of the face as a whole area may change slightly, but when selecting specific subareas, change may be better detectable.

In view of the above facts software is designed so that it allows four types of analyses. Allows to monitor the temperature over all images across the whole face then, to track the development of temperature for each half of the face. The face is divided according found face axis. Another option is to mark in the image of either one or two smaller squared sub-region, where will be carried out analysis.

It is possible to view the minimum, maximum or average temperature in the selected area.

Discussion

Part of the image segmentation seems to be the most problematic from the point of view of automatic processing. The main problem is the large variety of characters for all people, it makes difficult to find a general process applicable to all individuals. It is necessary to distinguish between face and hair. Since hair prevents the escape of IR radiation from the body surface and they can accumulate sweat that resulting temperature is distorted at this location. The same applies to the eyebrow area. Also hair represent today some individuality of man and so there are many hairstyles, which may complicate segmentation to some extent, since the dimensions of the face (eg. height, width) cannot be used for the orientation in the image, eg. ear was covered by hair only on one the side or hairs were unevenly distributed only over one half of the face and thus the highest point of the head was not located at the centre, as it would correspond to the anatomy of the head (see person to Fig. 4).

Another problem may be caused by variability of eyebrows, whether between individuals or even a single person. People with strong brows have present both arches of an eyebrow as the top and bottom in edge representation. In this case, the detection is easy, problems arise in the case that it is visible only one arc, or in addition one eyebrow has visible upper line and second lower line and, in extreme cases, especially for females may not be visible to any arc.

Asymmetry of both eyebrows causes that the size of diameters of approximating circle between individual eyes may vary significantly, resulting in that the y-coordinates of the centre of the circles are different. However detection of eyebrows was used only for a basic orientation in the image and narrowing the area in which eyes are certainty located, and for this purpose the algorithm is robust enough.

To verify the proposed algorithm was available only one type of thermal imager (FLIR B200) and therefore the algorithm is optimized for images with very low resolution. Nowadays, market offers already significantly better devices and exists thermal imagers that provide high-resolution images that capture more details and it is possible to detect in the face more important characteristic points.

Conclusion

This paper focuses on the possibility of non-contact temperature measurement of the faces and automatic processing of the obtained thermal images. The aim of this paper is to describe the possibilities of infrared imaging and also deals with its potential use in medical practice. The article describes a software that allows analysis of the evolution of temperature in a given area of the face over time. It focus on the segmentation of the thermogram and design of the algorithm of described software. It also describes the types of analyses that the user can choose.

Although, the algorithm was tested on images with low resolution, great results were achieved, and the purpose of this paper is mainly to draw attention to the automatic processing of the thermal images and possibilities for its use in clinical practice, where it could show new details for better diagnosis and also reduce the time required to examination.

Dana Kolářová

Department of Biomedical Engineering,

Faculty of Eletricial Engineering and Communications,

Brno University of Technology,

Antonínská 548/1, Brno 601 90

And

Department of Biophysics,

Faculty of Medicine,Masaryk University,

Kamenice 3, 625 00, Brno,

Czech Republic

E-mail: dana.kolarova@hotmail.com

tel: +420 728 536 858