International Journal of Advanced Research in Electrical, Electronics and Instrumentation Engineering
MenuISSN ONLINE(2278-8875) PRINT (2320-3765)
ISSN ONLINE(2278-8875) PRINT (2320-3765)
Jilfy James1, Jobi Jose2
|
| Related article at Pubmed, Scholar Google |
Visit for more related articles at International Journal of Advanced Research in Electrical, Electronics and Instrumentation Engineering
This paper mentions a method for the detection and correction of films damaged by tear. The method includes estimation of motion vectors between two frames in an image sequence, delineating the tear boundary and restoring the damaged frame using suitable missing data treatment. Thus loss of data along the tear boundary is also corrected. The paper also includes the comparison of performance of different motion estimation methods like exhaustive search, three step search, four step search, diamond search and adaptive rood pattern search. Among these adaptive rood pattern search is found to be the best in performance based on the study of quality measurement and the execution time required.
Keywords |
| motion estimation, displaced frame difference, image segmentation, image restoration |
I. INTRODUCTION |
| Good quality is the demand of today’s world in each and every field. This urge is more in the digital videos, therefore digital restoration of archives have gained much attention now-a-days. This paper deals with automatic correction of a film tear, wherein we compare the performance of different motion estimation techniques. |
| An image is a collection of pixels. A set of pixels is called a block. The motion of such different blocks within an image generates the next image in a video sequence. Here a tear that occurs in such an image sequence is considered. The tear affecting the clarity of an image is shown in figure 1 .The presence of tear is awkward .It occurs due to aging or due to the careless handling of the film. It occurs in some frames of an image sequence. Tear on the frame creates disturbance for the viewers and results in the loss of information along the tear. Thus the aim of this paper is to detect such a frame in the image sequence with tear and its correction. |
| Previous works related with film tear are very less. The only work done is given in [1] which is efficient but complex and time consuming. Here a method which is simple and compares with different motion estimation techniques to find minimum time required is adopted. |
 |
| The automated correction of tear in a film material is done through mainly four steps such as motion estimation, tear delineation, boundary detection and missing data treatment. The block matching method of motion estimation in combination with a simplified optical flow is carried out in motion estimation step. An additional comparison of different block matching techniques is also done. The quality and execution time of these methods is checked. |
II. BLOCK DIAGRAM |
| The block diagram shown in figure 2, shows different steps involved in the detection and treatment of torn films. The procedure involves mainly four steps. The initial step is motion estimation, followed by tear delineation, boundary detection and finally missing data treatment. Initially the motion estimation is done, where the relative motion or the transformation from one frame to another in an image sequence is determined. Tear delineation is the stage where the tear that occurred in a frame is detected. Displaced frame difference, segmentation and morphological operations are carried out in this process. Boundary detection is the stage where the torn portion is separated along the tear boundary. In missing data treatment the restoration of image is done. |
III. MOTION ESTIMATION |
| The relative motion between two frames in a sequence is determined by the algorithm in [3]. It uses block matching combined with simplified optical flow. The transformation between consecutive frames is obtained by proper motion estimation methods. It results in motion vectors. Motion vector indicates how much the current block is displaced compared to the reference block. In motion estimation two consecutive frames are selected. The first frame is selected as the current frame and the next frame is taken as the reference frame. A block which is a set of pixels is selected from the current frame and is searched for in the reference frame. The best matched block is then found and its displacement in position both in horizontal and vertical direction is noted. This results in forward motion vector. This motion vector consists of two values indicating the displacement in horizontal and vertical directions. According to the algorithm mentioned in [3] the motion vector resulting from the block matching is an approximate estimate of true displacement. In order to refine the search, Taylor series approximation is adopted. |
| If the motion estimation of three frames is to be found and assume that the second frame is affected by a tear then the result of motion estimation will look as given in figure 3. The Figure 3 (a) finds the relative motion between the first and second frame, where due to the presence of tear in the second frame, a set of pixels have displaced to downward direction. It happens since a block of pixels in the torn portion have undergone displacement in position in relation to the second frame. The Figure 3 (b) finds the relative motion between the second and third frame, where since there is no tear on the third frame those set of pixels have their positions in upward direction. Thus it is concluded that if there exits such motion vectors which opposes in direction in adjacent frames, then it indicates the presence of a tear in the middle frame. |
 |
 |
IV. TEAR DELINEATION |
| Tear delineation focuses on detecting the tear on a frame in an image sequence. The various steps involved are displaced frame difference, segmentation and morphological operations that refine the segmentation. |
| A) Displaced Frame Difference |
| Applying the motion vectors determined by motion estimation method to an image in order to synthesize the next frame in a sequence, results in motion compensation. The compensated frame thus generated is almost similar to the next frame. The difference between two frames results in frame difference. But displaced frame difference given in [2] is more efficient. It takes the difference between the compensated frame and the next frame as given in equation (1). The residue or the unwanted noise content will be less for displaced frame difference than the simple difference. This helps in reducing the burden of the morphological operations to remove the noise from the images. The displaced frame difference Δf(x) is given below. |
 |
| B) Segmentation |
| Through segmentation the set of pixels sharing certain similar visual characteristics such as color, intensity are assigned same value. In thresholding based segmentation technique, an image is segmented based on a given threshold value. Global thresholding is adopted and threshold value is taken as 0.3 by trial and error method. Pixels below this value become black and above become white. This thresholding naturally creates a binary image. |
| C) Tri Map Refinement |
| The resulting sequence from previous stage contains some unwanted or unassigned pixels. Thus the image consists of a definite background, a definite foreground and a blended region. This forms trimap and it has to be refined in order to get the image clearly. Here the region of interest or object is the tear. The blended region consists of pixels that cannot be precisely considered as part of foreground or background. Tri map refinement is done by area opening [6]. This is nothing but erosion followed by dilation. This is aimed at noise removal and also helps in finding specific shapes in an image. Thus the tear on the image can be found. |
 |
V. BOUNDARY DETECTION |
| An image will have an outer boundary and it can have objects within the image. If so they too have certain boundaries. Thus the task of the boundary detection section includes finding the exteriorand boundaries within a frame. Here the boundaries of entire frame and that formed due to tear are found. In order to achieve it, initially certain regional properties of the image are taken. The regional properties of each connected component or object is found. The connected components will be assigned to an array variable. The value 0 of this variable denotes the background, value 1 corresponds to first object, value 2 corresponds to second object and so on. The function can be performed only on binary images. Binary images are generated after thresholding based segmentation. For performing this function, in addition to the binary image, the row and column coordinates of the starting point, and the direction of the first step need to be mentioned. Only the area and the centroid of the frames in the image sequence is calculated from the many available regional properties. Some other examples of regional properties are bounding box, perimeter, eccentricity etc. A frame with tear will have two values of area and centroid. The centroids of the two regions separated by a tear on a frame are shown in figure 5. It shows two centroids located at two regions having two different areas. Therefore by analyzing the index of the area and centroid variable, the presence of tear can be ensured. For boundary detection to be executed following steps has to be performed. |
| 1. Read image and display it. |
| 2. Convert the image to a binary image. |
| 3. The row and column coordinates of a pixel on the border of the object that has to be traced is determined and is considered as the starting location for the boundary tracing. |
| 4. Direction of trace is also needed to start from the starting point. |
| 5. Display the original gray scale image and use the coordinates returned to plot the border on the image. It returns a cell array, where each cell contains the row/column coordinates for an object in the image. |
 |
 |
VI. MISSING DATA TREATMENT |
| The tear in the frame results in loss of data. The missing data can appear as dark or bright blotches which can occur due to dirt, scratch etc. All these occur due to aging of film material or careless handling. Many applications demand the resynthesizing of lost information in image regions. The loss of data at certain regions can affect any file. Blotches or scratches can lead to loss of data and these will cover up the information underlying them. This has to be resynthesized. Since the defect here is a tear, the region of the tear is initially masked. A mask has pixel value as that of either black or white. It covers the region of interest. It can be viewed as a filter too, since it restricts the application of operations to certain regions only.In order to restore the lost data, information should be taken from previous or next frame. Same method is adopted here. The tear is initially masked. Mean of the binary image is taken. If there is a tear or a missing region on an image the mean value would be greater than 0.5.If there is no tear the mean value would be 0.5. Later the information is taken from the previous or next frame, if there is no masking in that frame. Thus the fully restored sequence can be obtained. |
VII. RESULTS |
| Two inputs are tested namely old London street sequence and caltrain sequence. Tear is made manually in the sequenceand is taken for further stages. Initially motion estimation is done. There exits different block matching methods. Among them the performance of diamond search, adaptive rood pattern search, three step search, four step search and exhaustive search are found. The performance is checked based on the quality of result and the execution time required. The studies of performance of these block matching methods are mentioned in Table I. The comparison of the same method based on the execution time is also done. The exhaustive search is found to take more time. Adaptive rood pattern search takes the least time. The execution times for different block matching methods are mentioned in Table II. The time required for the execution of thirty selected frames are analysed for each method.After analysing the results the optimized block matching method is found. The three step search and adaptive rood pattern search have the quality measurement almost close to exhaustive search, taking less time. |
 |
 |
VIII.CONCLUSION |
| This paper deals with a method for digital restoration of torn films Here the tear is detected and the destructed image data along the tear boundary is corrected. The goal is achieved by doing the four steps like motion estimation, tear delineation, boundary detection and missing data treatment. In motion estimation, the relative motion between two frames in an image sequence is determined. In tear delineation tear is detected through displaced frame difference, segmentation and morphological operations. In boundary detection the torn portion is separated along the tear boundary after finding the different regional properties of the image. Through proper missing data treatment the restoration of image is done. |
References |
|