This paper presents comparison of two image processing algorithms used for eye blink detection. The motivation of this research work is the need of disabled persons who are unable to move their body parts except eyes. The process of blink detection is divided into three parts viz.face localization, eye pair localization and template matching method. In method 1, YCbCr color model and morphological operations are used for the face and eyes localization. In method 2 face and eyes pair localization is performed by using Viola Jones method. After eye pair localization, the concept of template matching is applied for blink detection, in both the methods. A performance comparison is made for both the methods based upon detection accuracy and processing time. It is observed that method 1, gives better accuracy (80.75%) with low processing time (0.38sec.). The overall success rate of method 1 and method 2 is 71% and55% respectively.
Keywords |
Face localization, Eye localization, Eye blink detection, YCbCr Color models, Viola Jones,
Morphological operations and Template matching. |
INTRODUCTION |
Paralysis is the complete loss of muscle function for one or more muscle groups. Paralysis can cause loss of feeling or
mobility in the affected areas. Paralysis is most often caused by damage to the nervous system, especially the spinal
cord [2]. Fully paralysed patients require 24 hour support. But, it is not possible for anyone to be available at all times.
In this paper an improved efficiency of eye blink is given. Before starting work on eye blink we have to go through
from some steps as shown in fig.1. There are some techniques devised for blink detection as well. This research work is
to improve the efficiency of eye blink detection rate. Before doing work on eye blink we have to go through a survey as
explained below. |
LITRATURE SURVEY |
Ijaz Khan, Hadi Abdullah et al [2013]: In this paper the author presents improved algorithms for face, eyes and
mouth detection in an image. Viola Jones and skin color pixel detection as face detection techniques are widely used.
Viola Jones gives accurate face detection but consumes more time whereas skin color pixel detection technique
consumes less time but lacks in accuracy. |
Atish Udayashankar et al [2012]: The main aim of this paper is to design a real time interactive system that can assist
the paralyzed to control appliances such as lights, fans etc. or by playing pre-recorded audio messages, through a
predefined number of eye blinks. Image processing techniques have been implemented in order to detect the eye blinks. |
K Takahashi et al [2012]: This paper proposed a practical method for eye blink detection using a monocular system.
Eye blink detection is an important technology in many situations such as facial action analysis and signal processing.
However, automatic eye blink detection is quite difficult since the eye blink is occurred fast |
Kohei Aai and Ronny Mardiyanto et al [2011]: In this paper blink detection is used for a variety of applications such
as Human-Computer Interaction, wearable computing, etc. Blink detection method with Gabor filter is proposed to
realize high accurate blink detection. The blink detection accuracy is evaluated by several people and compared to the
other existing conventional methods. Through the comparison, it is found that the proposed blink detection method
with Gabor filter is superior to the other methods. |
![image](https://www.rroij.com/articles-images/IJAREEIE-1358-g001.gif) |
A. Human Face Localization |
For human face localization Viola Jones and skin color pixel detection as face detection techniques are widely used.
Viola Jones gives accurate face detection but consumes more time whereas skin color pixel detection technique
consumes less time
but lacks in accuracy. In this research both the techniques which are to be used will increases accuracy while
consuming less time. Viola Jones and other methods can accurately detect faces but in case of facial features detection
their accuracy decreases [1, 6, 14, 16]. |
B. Eye Localization |
The localization of eyes in facial images has many applications in computer vision such as gaze estimation, pose
estimation, face detection, face recognition, human computer interaction, eye blink detection etc. Eyes can be extracted
from facial images using different eye features such as color, illumination, shape, geometry, edges etc.[10,12,14].Once
two eye blocks appear from the segmented image and spread to a certain size; they will be detected by the
determination criterion of eye region[3,19]. However, sometimes, it is found that in the detection process the two
detected eyes might not be real eyes, furthermore, it is difficult to seek precise centre of eye [31]. |
C. Eye Blink Detection |
As the users eye closes during the process of a blink, its similarity to the open eye template decreases. Likewise, it
regains its similarity to the template as the blink ends and the user’s eye becomes fully open again. This decrease and
increase in similarity corresponds directly to the correlation scores returned by the template matching procedure [1, 3,
17]. |
PROBLEM FORMULATION |
Need of Proposed Research Work |
The main need of eye blink detection system is to help those people who are unable to move any part of their body
excepting their eyes. Fully paralysed patients require 24 hour support. But in present day this is not possible for anyone
to be available full time. There are some techniques devised for blink detection as well [2]. The detection of blinking
and analysis of blink detection will support the people who are not able to move their body parts. |
Significance of Proposed Research Work |
Significance of this research work is to improve detection time of eye blink as we know that Human face image
analysis, detection and recognition have become some of the most important research topics in the field of image
processing and pattern classification. The potential applications involve topics such as face detection, face
identification and recognition, and facial expression analysis. Among these research topics, one fundamental but very
important problem to be solved is automatic eye detection. The eye is the most significant and important feature in a
human face, as extraction of the eyes are often easier as compared to other facial features. |
METHODOLOGY/PLANNING OF WORK |
Two methods are used for eye blink detection in this research. In both the methods different techniques are applied to
localize human face and eye and a common method (Template Matching) is used for eye blink detection. Finally,
processing time and accuracy of both the methods is compared. Because, processing time is very important parameter
when eye blink detection is to be implemented in real-time (in future). Steps we followed in both the methods are as: |
Method 1 |
a) Human Face Localization and Eye Localization using YCbCr Color Model. |
b) Template Matching for Blink Detection. |
Method 2 |
a) Haar - like features for Face Localization and Eye Localization. |
b) Template Matching for Blink Detection. |
YCbCr COLOR MODEL |
YCbCr is a family of color spaces used as a part of the color image pipeline in video and digital photography systems.
Y is the luma component and CB and CR are the blue-difference and red difference chrome components. Y (with
prime) is distinguished from Y which is luminance, meaning that light intensity is non-linearly encoded using gamma
correction. YCbCr color model wedge the digital image into two parts, luma or luminance component and chrome or
chrominance component. Due to luminance and chrominance effect, it becomes most important in digital video camera
to handle video information [9, 23, 25] The YCbCr color space is widely used for digital video. In this paper,
luminance information is stored as a single component (Y), and chrominance information is stored as two colordifference
components (Cb and Cr). Cb represents the difference between the blue component and a reference value Cr
represents the difference between the red component and a reference value. |
VIOLA-JONES METHOD |
The Viola Jones object detection framework proposed in by Paul Viola and Michael Jones is the first object
detection framework to provide competitive object detection rates in real-time .Viola Jones and skin color pixel
detection as face detection techniques are widely used.[1,21,27] They proposed a face detection method based on the
adaboost learning algorithm using haar features that detected the face successfully with high accuracy.[26] Viola Jones
algorithm are: |
A. Integral images |
B. Haar Features |
C. Cascade of boosted classifier |
A. Integral images |
The simple rectangular features of an image are calculated using an intermediate representation of an image, called the
integral image [9]. The integral image is an array containing the sums of the pixels’ intensity values located directly to
the left of a pixel and directly above the pixel at location (x, y) inclusive. So if A[x, y] is the original image and AI[x,
y] is the integral image. |
B. Haar Classifier |
The core basis for Haar classifier object detection is the Haar-like features. These features, rather than using the
intensity values of a pixel, use the change in contrast values between adjacent rectangular groups of pixels. The contrast
variances between the pixel groups are used to determine relative light and dark areas. Two or three adjacent groups
with a relative contrast variance form a Haar-like feature. Haar-like features, as shown in Fig.3 are used to detect an
image [8, 16, 27]. Haar features can easily be scaled by increasing or decreasing the size of the pixel group being
examined. This allows features to be used to detect objects of various sizes. |
![image](https://www.rroij.com/articles-images/IJAREEIE-1358-g002.gif) |
C. Cascade of boosted classifier |
The overall form of the detection process is that of a degenerate decision tree, what we call a “cascade”. A positive
result from the first classifier triggers the evaluation of a second classifier which
has also been adjusted to achieve very high detection rates. A positive result from the second classifier triggers a third
classifier, and so on Stages in the cascade are constructed by training classifiers using AdaBoost and then adjusting the
threshold to minimize false negatives as shown in fig.4 has also been adjusted to achieve very high detection rates. A
positive result from the second classifier triggers a third classifier, and so on Stages in the cascade are constructed by
training classifiers using AdaBoost and then adjusting the threshold to minimize false negatives as shown in fig.4.A
series of classifiers are applied to every sub-window. The initial classifier eliminates a large number of negative
examples with very little processing. Subsequent layers eliminate additional negatives but require additional
computation. After several stages of processing the numbers of sub-windows have been reduced radically. |
![image](https://www.rroij.com/articles-images/IJAREEIE-1358-g003.gif) |
MORPHOLOGICAL OPERATIONS |
Morphological techniques are used for boundary detection. Dilation, followed by erosion and the calculation of
differences between the two produces an image with boundaries. The structuring element used in dilation and erosion
has a large matrix, so that clear and thick boundaries are detected [5, 25]. |
TEMPLATE MATCHING |
For eye blink detection template matching is used here.In the template matching method, segments of an input image
are compared to previously stored images, to evaluate the similarity of the counterpart using correlation values[2, 5]. |
Diff; Image |
D = (Tc-Ts) |
s = sum (sum (d)); |
Tc = current frame |
Ts = stored frame |
*Threshold is set by empirical method. |
Frame differencing is applied to find correlation between current template and stored template. Hence the value of ‘s’
less than threshold value indicates template matching. |
Method 1 |
(a) Human Face and Eye Localization using YCbCr Color Model: In this part following steps are to be followed: |
A. I/P frame |
B. Convert RGB to YCbCr |
C. Morphological operations |
D. Face localization |
E. Eye pair localization |
F. Eye blink detection(Template Matching Method) |
A. I/P frame: This is the very first step in which an input frame are to be given to the YCbCr Color Model for the purpose of
whole procedure. |
B. Conversion RGB to YCbCr: As we know that RGB frame is acquired for the process but we have to convert RGB frame
to YCbCr because this transformation is useful in detecting blobs and reduces the computational complexity. |
C. Morphological operations |
Dilation: The value of the output pixel is the maximum value of all the pixels in the input pixel's neighbourhood. In a binary image,
if any of the pixels is set to the value 1, the output pixel is set to 1. |
Erosion: The value of the output pixel is the minimum value of all the pixels in the input pixel's neighbourhood. In a binary image,
if any of the pixels is set to 0, the output pixel is set to 0. |
Structuring Elements: An essential part of the dilation and erosion operations is the structuring element used to probe the input
image. |
D. Face localization: After applying morphological operation on an input frame at the end face is detected as shown. |
E. Eye pair localization: This is the next step i.e. eye pair detection and edge detection is involved in it. Morphological
techniques are used for boundary detection. Dilation, followed by erosion and the calculation of differences between the two
produces an image with boundaries. Diamond structuring element is used. |
![image](https://www.rroij.com/articles-images/IJAREEIE-1358-g004.gif) |
Template matching is a technique for finding areas of frame that match (are similar) to a template frame (patch) it has
two primary components: |
Source frame (I): The frame in which we expect to find a match to the template frame as shown below: |
Template frame (T): The patch frame which will be compared to the template frame as shown below: |
![image](https://www.rroij.com/articles-images/IJAREEIE-1358-g005.gif) |
Method 2 |
(a) Haar - like features for Face Localization and Eye Localization using Viola Jones Algorithm. |
A. I/P frame |
B. Face localization |
C. Eye pair localization |
D. Eye blink detection (Template Matching Method) |
![image](https://www.rroij.com/articles-images/IJAREEIE-1358-g006.gif) |
Template matching is a technique for finding areas of frame that match (are similar) to a template frame (patch) it has
two primary components: |
Source frame (I): The frame in which we expect to find a match to the template frame as shown below: |
Template frame (T): The patch frame which will be compared to the template frame as shown |
![image](https://www.rroij.com/articles-images/IJAREEIE-1358-g007.gif) |
RESULTS AND DISCUSSION |
Detection Accuracy (DA) =TP/TP+FN…………. (1) |
False Alarm Rate (FAR) =FP/TP+FP……………… (2) |
Success Rate (%) =DA/DA+FAR…… (3) |
TP: Correctly detected blinks |
FP: False detection of blinks |
FN: Missed blink |
![image](https://www.rroij.com/articles-images/IJAREEIE-1358-t001.gif) |
Average success rate: 71% |
Average Accuracy: 80.75% |
Processing Time (Per blink): 0.38 sec. |
![image](https://www.rroij.com/articles-images/IJAREEIE-1358-t002.gif) |
Average success rate: 55% |
Average Accuracy: 60.37% |
Processing Time (Per blink): 0.65 sec |
![image](https://www.rroij.com/articles-images/IJAREEIE-1358-t003.gif) |
CONCLUSION |
Two different methods are implemented in this research for Blink Detection, which follows face and eye pair
localization in a frame and template matching to check the status of eyes(opened or closed).In Method 1, face and eyes
pair localization is performed usingYCbCr skin color model and morphological operations. Then template matching is
applied for blink detection. In Method 2, face and eyes localization is performed using Viola Jones Computer-Vision
Toolbox and template matching for blink detection. From result analysis, it is observed that, average detection accuracy
and processing time for Method 1 and 2 are 80.75%, 60.37% and 0.38 sec., 0.65sec., respectively. Therefore, it can be
concluded that the accuracy of Method 1 is better as compared to Method 2 and method one require less processing time too. The fall in accuracy is due to inability of the method to detect face and eye pair area, especially is case of
tilted faces. Further, the overall success rate is 71% and 55% for method one and two, respectively. The fall in the
success rate is due to large no. of “False Positives” i.e. false detection of blinks. Finally, it can be concluded that,
Method 1 is more suitable choice for blink detection for subjects with tilted faces too. |
References |
- Aleksandra Krolak, PawelStrumillo âÃâ¬ÃÅEye-Blink Detection System for Human-Computer InteractionâÃâ¬Ã Institute of Electronics, Technical University of Lodz, Wolczanska.pp.90-924, Lodz, Poland,2012.
- A. Yuille, P. Hallinan, D. Cohen, âÃâ¬ÃÅFeature extraction from faces using deformable templatesâÃâ¬ÃÂ, International Journal of Computer Vision, pp. 99-111, 1992.
- Bhaskar, T.N., Foo TunKeat ,Ranganath, S., Venkatesh, Y.V., 2003. âÃâ¬ÃÅBlink Detection and Eye Tracking for Eye LocalizationâÃâ¬ÃÂ.Conference on Convergent Technologies for Asia-Pacific Region,10oct 2003.
- CihanTopal, ÃÆÃâmerNezihGerek and Atakan âÃâ¬ÃÅA Head- Mounted Sensor-Based Eye Tracking Device: Eye Touch SystemâÃâ¬ÃÂ. Proeedings of the Symposium on Eye tracking research & applications. Savannah, GA, USA, pp. 87-90, 2008.
- CC.Hen, H.Y.M.Liao, G.J.Yu, L.H.Chen, âÃâ¬ÃÅFast face detection via morphology based pre processing. Pattern RecognitionâÃâ¬Ã pp.1701-1712, National Science Council of Taiwan 2000.
- Cai J, Goshtasby.A. and Yu C, âÃâ¬ÃÅDetecting Human Faces in Color ImagesâÃâ¬Ã Proceedings of International Workshop on Multi-Media Database Management Systems, pp. 124-131,1998.
- Constantine KotropoulosIoannis Pitas âÃâ¬ÃÅRule-Based Face Detection In Frontal ViewsâÃâ¬Ã (IEEE) International conference, volume-4,pp. 2537-2540,1997.
- Duan-Sheng Chen ,Zheng-Kai Liu, Aug 2011. âÃâ¬ÃÅGeneralized Haar-like features for Fast Face DetectionâÃâ¬ÃÂ. Conference on Machine Learning and Cybernetics, Hong Kong, pp. 2131-2135, 2007.
- Hsu, R.L., Abdel-Mottaleb, M. & Jain, A,âÃâ¬ÃÂFace detection in color imagesâÃâ¬ÃÂ, IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 24, no. 5, pp. 696-706,2002.
- Ijaz Khan, Hadi Abdullah and MohdShamian Bin Zainal âÃâ¬ÃÅEfficient eyes and mouth detection algorithm use combination of Viola Jones and skin color pixel detectionâÃâ¬Ã International Journal of Engineering and Applied Sciences. June 2013.
- Inho Choi, Seungchul Han, and Daijin Kim, âÃâ¬ÃÅEye Detection and Eye Blink Detection using Adaboost learning and groupingâÃâ¬Ã computer communication and networks (ICCCN), proceeding of 20th international conference .,pp.1-4 July31,2011.
- J.Huang ,H.Wechsler, âÃâ¬ÃÅVisual routines for eye localization using learning and evolutionâÃâ¬ÃÂ.IEEETrans.Evol.Comput. 73-82,2000.
- K.M. Lam, H. Yan, âÃâ¬ÃÅAn analytic-to-holistic approach for face recognition based on a single frontal viewâÃâ¬ÃÂ, IEEE Trans. Pattern Anal. Mach. Intell. 673- 686 20, 1998.
- KoheiAai and Ronny Mardiyanto , âÃâ¬ÃÅComparative Study on Blink Detection and Gaze Estimation Methods for HCI, in Particular, Gabor Filter Utilized Blink Detection MethodâÃâ¬Ã .Proceedings of 18th International Conference on Information Technology: New Generations. Las Vegas, USA, pp. 441-446,2011.
- Kwok-Wai Wong, Kin-Man Lam, Wan-Chi Siu âÃâ¬ÃÅAn efficient algorithm for human face detection and facial feature extraction under different conditionâÃâ¬Ã pp.1-12,The Journal of The Pattern Recognition 2004.
- Lae-Kyoung Lee, Su-Yong An , Se-Young Oh, Aug 2011. âÃâ¬ÃÅEfficient. Face Detection and Tracking with extended camshift and haar-like featuresâÃâ¬ÃÂ. International Conference on Mechatronics and Automation (ICMA), Pohang, South Korea, pp. 507-513, 2011.
- Liting Wang ; Xiaoqing Ding Chi Fang ; Changsong Liu and Kongqiao Wang "Eye blink detection based on eye contour extraction".2009, Proc. SPIE 7245, Image Processing: Algorithms and Systems VII, 72450R ,February 10, 2009.
- Michael Chau and MargritBetke, .âÃâ¬ÃÅReal Time Eye Tracking and Blink Detection with USB CamerasâÃâ¬ÃÂ2005. Boston University, USA. May 12, 2005.
- Muhammad Shafi and Paul W. H.ChungâÃâ¬ÃÅA Hybrid Method for Eyes Detection in Facial ImagesâÃâ¬Ã World Academy of Science, Engineering and Technology.pp.1-6, 2008.
- M. Takagi, K. Mohri, M. Katoh and S. Yoshino.âÃâ¬ÃÂMagnet-Displacement Sensor Using Magneto-Inductive Elements for Sensing Eyelid MovementâÃâ¬ÃÂ. IEEE Translation Journal On Magnetics In Japan, Vol. 9, No.2, pp .78-83, 1994.
- Paul Viola âÃâ¬ÃÅRobust Real-Time Face DetectionâÃâ¬Ã International Journal of Computer Vision 57(2), 137âÃâ¬Ãâ154, 2004.
- S Saravanakumar, and N Selvaraju, âÃâ¬ÃÅEye Tracking and Blink Detection for Human Computer InterfaceâÃâ¬ÃÂ, International Journal of Computer Application, VOl. 2, No. 2, pp 7-9, May 2010.
- Sanjay Kr. Singh, D. S. Chauhan, MayankVatsa, Richa Singh âÃâ¬ÃÅA robust skin color based face detection algorithmâÃâ¬Ã Tamkang Journal of Science and Engineering, Vol. 6, No. 4, pp. 227-234, 2003.
- Takahashi, K., Mitsukura, Y. âÃâ¬ÃÅEye blink detection using monocular system and its applicationsâÃâ¬Ã Grad. Sch. of Sci. & Technol., Keio Univ., Yokohama, Japan. , pp 743 âÃâ¬Ãâ 747, 9-13 Sept. 2012
- TanmayRajpathak,Ratnesh Kumar and Eric Schwartz âÃâ¬ÃÅEye Detection Using Morphological and Color Image ProcessingâÃâ¬ÃÂFlorida Conference on Recent Advances in Robotics, FCRAR.pp.1-6,2009.
- Udayashankar, A. ;Kowshik, A.R. ; Chandramouli, S. ; Prashanth, H.S. âÃâ¬ÃÅ Assistance for the Paralyzed Using Eye Blink DetectionâÃâ¬Ã Fourth International Conference,Digital Home (ICDH), Pp 104 âÃâ¬Ãâ 108, 2012.
- Viola, P. and Jones, M., âÃâ¬ÃÅRapid Object Detection using a Boosted Cascade of Simple FeaturesâÃâ¬Ã Proc. of the Conf. On Computer Vision and Pattern Recognition (CVPR), Hawaii, USA, Vol. 1, pp. 511-518, December 9-14, 2001.
- W. Zhang,âÃâ¬ÃÅPrecise eye localization with AdaBoost and fast radial symmetry,âÃâ¬Ã Computational Intelligence and Security, pp. 1068-1077, 2006.
- YihuYi ,DaokuiQu , Fang Xu âÃâ¬ÃÅA Hybrid Algorithm of Eyes Localization in Color Facial RegionâÃâ¬ÃÂInternational Conference on Artificial Intelligence and Computational Intelligence.pp.1-6,2009.
- Yoav Freund and Robert E. Schapire âÃâ¬ÃÅA Decision-Theoretic Generalization of On-Line Learning and an Application to BoostingâÃâ¬Ã journal of computer and system sciences 55, pp.119-139, 1997.
- ZhihengNiu, Shiguang Shan, Shengye Yan, Xilin Chen, and Wen Gao, âÃâ¬ÃÅ2D Cascaded AdaBoost for Eye LocalizationâÃâ¬Ã ICPR (International Conference on Pattern Recognitition),pp.1216-1219,(IEEE)2006.
|