An adaptive video-stream's scene boundaries detector and method of its learning based on the video stream's content characteristics: dark/light, slow/quick

The problem of scene boundaries detection are considered. A scene is defined as an unbroken meaningful sequence of frames taken from one camera. There are two basic types of scene transitions: abrupt and gradual. This research is focused on abrupt transitions only. The aim of the project is to develop a detector (classifier) which every frame of the video-stream classify either to class 1 (first frame of the scene) or to class 0 (internal frame of the scene). To evaluate quality of the detector sensitivity, specificity, precision and F-score are used: „ TP „ TN „ TP „ ,, TP Precision • Recall Se =-, Sp =-, Precision =-, Recall =-, F 1 = 2-. P N TP + FP TP + FN Precision + Recall The proposed detector is based on using metrics and the frame partitioning into non-overlapping blocks. The metrics are the functions that calculate measure of similarity between frames. If the metric value for two consecutive frames is small, it means the frames are similar and, probably, belong to one scene. If the metric value is large, it means the frames are dissimilar and, probably, belong to different scenes. Splitting of the frame into blocks is used to improve the quality of the detector: a block is signaling about scene change if the metric value for the corresponding blocks of two successive frames exceeds a prespecified threshold value Tm; the detector make a decision "scene change detected" in case portion of signaling blocks is above a prespecified threshold value T_b. Type of the metrics, number of the blocks and threshold values influence on detector's quality. It is discovered that optimal threshold values depend strongly on the video-stream's content. This fact makes harder to develop a fully automatic scene boundaries detector. To overcome this problem this work is dedicated. It was found that the optimal threshold values depend on such characteristics of the video stream content, as dark/light and slow/quick. Methods for qualitative estimation of these characteristics for local fragments of the video-stream were developed. The algorithm for adaptive scene boundaries detector, which for each frame of the video-stream use optimal thresholds values based on data of nearby frames, was proposed. A method for estimating the parameters of threshold's dependency models was proposed. To verify the developed algorithm and method of its learning the experiment on real data was conducted. Twenty fragments of popular movies was loaded from youtube.com. Twenty seven video-streams contained abrupt scene transitions only were prepared such that every source video-stream can be classified to one of the next four classes: dark-slow, light-slow, dark-quick, light-quick. The length of each video-stream was 1499 frames. Total number of scenes was 1098. One thousand experiments was conducted. In each experiment the source video-stream sample was divided into three part: learning, validation and test samples. Learning and validation samples were used to estimate parameters of adaptive detector, test sample was to evaluate its performance. Next mean values of sensibility, specificity, precision and F-score were gained: Se = 0.92, Sp = 0.99, Precision = 0.94 и F1 = 0.93. In comparison to fixed threshold's values detector with T_b = 0.6 и T_m = 0.06, adaptive detector has a higher classification precision and close values of other quality's measures. In comparison to the hypothetically best detector, adaptive detector has a little bit lower value of sensitivity and close values of other quality's measures. It must be noted, that obtained results were received on the sample of movies, the quality of the adaptive detector on video-streams with other types of content is subject for another research.

Keywords

Authors

References