Introduction to Video Search Engines (eBook)

David Gibbon joined Bell Laboratories in 1985 and is currently a Lead Member of Technical Staff in the Video and Multimedia Services Research Department at AT&T Labs - Research. His research interests include multimedia processing for searching and browsing of video databases and real-time video processing for communications applications. David has written book chapters and encyclopedia articles as well as numerous technical papers; he has 40 US patent filings and holds 14 US patents in the areas of multimedia indexing, streaming, and video analysis; and he is a member of the ACM, and a senior member of the IEEE. David contributes to IPTV industry standards for metadata and in 2007 he was awarded the AT&T Science and Technology Medal for outstanding technical leadership and innovation in the field of Video and Multimedia Processing and Digital Content Management.Zhu Liu joined AT&T Labs - Research in 2000, and he is currently a Principal Member of Technical Staff in the Video and Multimedia Services Research Department. His research interests include multimedia content processing, multimedia databases, pattern recognition, and machine learning. Zhu holds 7 US patents and he is the inventor of more than 20 pending patents in the areas of multimedia service and content analysis. He has published more than 40 refereed papers in international leading journals and at key conferences in the areas of multimedia. He is a member of ACM and Tau Beta Pi, and a senior member of the IEEE.

Preface 5
Who should read this book? 7
How is this book organized? 7
Acknowledgements 8
Contents 9
1 Video Search 16
1.1 Introduction 16
1.2 Addressing the Opportunity 17
1.3 Classification of Web Video Sites 20
1.3.1 Content Originators and Traditional Broadcasters 20
1.3.2 Aggregators 21
1.3.3 Download 21
1.3.4 Sharing 21
1.3.5 Application Specific 22
1.3.6 Other Video Systems 22
1.4 Classification of Video Sources 23
1.4.1 Webcams / Security 24
1.4.2 Video Telephony / Teleconferencing 24
1.4.3 Industrial / Academic / Medical 24
1.4.4 User Generated Content 25
1.4.5 Public Access and Government (PEG) Content 25
1.4.6 Enterprise Content 25
1.4.7 Rushes, Raw Footage 26
1.4.8 News 26
1.4.9 Advertising 26
1.4.10 Episodic TV Programming 26
1.4.11 Feature Films 27
1.4.12 Content Value 27
1.5 Challenges of Video Search 28
1.5.1 Acquisition 29
1.5.2 Media File Formats 30
1.5.3 Data Transport 31
1.5.4 Browsing 31
1.5.5 Duplication 32
1.5.6 Ranking and Indexing 32
1.6 Advantages of Video Search over Text 33
1.6.1 Applications 33
1.6.2 Metadata 34
1.7 Metadata vs. Content 34
1.7.1 Content-based retrieval 34
1.8 Conclusion 35
References 36
2 Video Data Sources and Applications 38
2.1 Introduction 38
2.1.1 Evolution of Digital Media Metadata 38
2.1.2 Consumer Video Metadata 39
2.1.3 Metadata Loss 39
2.1.4 Metadata Standards 40
2.1.5 Dublin Core 41
2.1.6 MPEG-7 42
2.1.7 MPEG-21 42
2.2 Essential Media Metadata 44
2.2.1 Embed Global Metadata 44
2.2.2 Elementary Metadata 44
2.3 Metadata for Personal Media Collections 46
2.3.1 Consumer Media Libraries 46
2.3.2 UPnP Forum 48
2.3.3 MP3 ID3 48
2.3.4 3GP / QuickTime / MP4 49
2.3.5 Metadata Services 49
2.3.6 Content Identification 51
2.3.7 Recorded Television 52
2.4 Media Syndication: RSS Content Description 54
2.4.1 Content Syndication 54
2.4.2 Media Enclosures 54
2.4.3 Podcasts 56
2.4.4 RSS for Content Ingest 57
2.4.5 MediaRSS 58
2.5 Metadata for Broadcast Television 58
2.5.1 Electronic Programming Guide (EPG) 59
2.5.2 Extended Data Service (XDS) 61
2.5.3 Program and System Identifier Protocol (PSIP) 62
2.6 Metadata for Video on Demand 62
2.6.1 Introduction 62
2.6.2 Cable Labs 64
2.7 Production Metadata 65
2.8 Timed Text Formats 66
2.8.1 Introduction 66
2.8.2 Synchronization Precision and Resolution 67
2.8.3 Transcripts 68
2.8.4 Closed Captions 69
2.8.5 Synchronized Accessible Media Interchange 70
2.8.6 Metadata from Social Sources 70
2.8.7 Metadata Issues 70
2.9 Conclusion 71
References 71
3 Internet Video 74
3.1 Introduction 74
3.2 Digital Video 74
3.2.1 Aspect Ratio 74
3.2.2 Luminance and Chrominance Resolution 76
3.2.3 Video Compression 77
3.3 Internet Protocol Media Systems 81
3.3.1 Transport 81
3.3.2 Searching VoD vs. Live 82
3.3.3 IPTV 83
3.3.4 Rights Management 85
3.3.5 Redirector Files 85
3.3.6 Layered Encoding 88
3.3.7 Illustrated Audio 88
3.4 Media Captioning 89
3.5 Conclusion 90
References 91
4 Video Search Engine Systems 92
4.1 Introduction 92
4.2 Content Acquisition 93
4.2.1 Metadata Normalization 93
4.2.2 User Contributed 94
4.2.3 Syndicated Contribution 95
4.2.4 Broadcast Acquisition 96
4.3 Content Processing 97
4.3.1 Asset Management 97
4.4 Retrieval 99
4.5 User Perspectives 100
4.5.1 Interaction States 100
4.5.2 Granularity of Search Results Representation 102
CID1 103
4.6 Factors Concerning Scalability 103
4.6.1 Introduction 103
4.6.2 Acquisition 104
4.6.3 Processing 104
4.6.4 Storage 105
4.6.5 Retrieval 106
4.7 Retrieval Interfaces 107
4.8 Typical System Features 108
4.9 Conclusion 109
References 109
5 Media Processing 112
5.1 Introduction 112
5.2 Feature Extraction 114
5.3 Media Segmentation 115
5.4 Clustering, Structure Generation 116
5.5 Real-Time Processing 118
5.6 Systems Issues and Architectures 118
5.7 Conclusion 119
References 120
6 Video Processing 122
6.1 Introduction 122
6.2 Shot Boundary Determination 123
6.2.1 Feature Extraction 125
6.2.2 Shot Boundary Detectors 126
6.2.3 Fusion of Detector Results 132
6.2.4 Evaluation Results 132
6.3 Representative Image Selection 133
6.4 Face Detection 136
6.5 Face Recognition 141
6.6 Video Optical Character Recognition 144
6.7 Concept Detection 146
6.7.1 Color Feature 148
6.7.2 Texture Feature 148
6.7.3 Edge Feature 150
6.8 Video Browsing 150
6.9 Conclusion 155
References 156
7 Audio Processing 160
7.1 Introduction 160
7.2 Audio Signal and Its Representation 161
7.3 Audio Features 163
7.3.1 Frame-Level Features 163
7.3.2 Clip-Level Features 169
7.4 Audio Segmentation 171
7.4.1 Speaker Segmentation 172
7.4.2 Audio Scene Segmentation 173
7.5 Audio Content Categorization 175
7.5.1 Speaker Recognition 175
7.5.2 Audio Scene Detection 177
7.5.3 Music Genre Classification 178
7.6 Speech Recognition 179
7.7 Audio Query and Browsing Techniques 181
7.7.1 SpeechLogger 182
7.7.2 Query by Example 186
7.8 Conclusion 187
References 188
8 Text Processing 192
8.1 Introduction 192
8.2 Story Segmentation 193
8.2.1 Cue Phrases 193
8.2.2 Cosine Similarity 194
8.2.3 Dynamic Programming 196
8.2.4 Topic Classification 198
8.3 Named Entity Extraction 198
8.3.1 Rule Based NEE 199
8.3.2 Data Driven NEE 200
8.3.3 NEE Tools 201
8.4 Part-of-Speech Tagging 202
8.5 Capitalization 204
8.5.1 Linguistic Processing Architecture 206
8.5.2 Web Document Collection 206
8.5.3 Text Capitalization Algorithm 207
8.6 Information Retrieval 209
8.6.1 Stemming 209
8.6.2 Term Weighting 210
8.6.3 Ranking 211
8.7 Text Summarization 212
8.7.1 Keyword Extraction 214
8.8 Conclusion 216
References 216
9 Multimodal Processing 218
9.1 Introduction 218
9.2 Case Studies 220
9.2.1 Closed Caption Alignment 220
9.2.2 Multimodal News Story Segmentation 224
9.2.3 Major Cast Detection 229
9.3 Conclusion 232
References 232
10 Research Systems 236
10.1 Introduction 236
10.2 Academic and Industrial Research 237
10.3 Early Internet Deployments 241
10.3.1 SpeechBot 241
10.3.2 StreamSage 242
10.3.3 SingingFish 242
10.4 Selected Commercial Systems 243
10.4.1 Virage and Convera 243
10.4.2 Nexidia (FastTalk) 243
10.5 Resources: Datasets, Evaluations, Conferences 244
10.6 Media Monitoring Deployments 246
10.7 Case Study: AT& T MIRACLE
10.7.1 Introduction 247
10.7.2 System Architecture 247
10.7.3 Collections 248
10.7.4 Data Organization 250
10.7.5 Acquisition / Ingest 251
10.7.6 Content Processing 253
10.7.7 Real-time processing 254
10.7.8 Query Engine 254
10.7.9 Applications 255
10.7.10 Performance 255
10.8 Conclusion 257
References 257
11 Current Trends in Video Search 262
11.1 Introduction 262
11.2 Video Production 263
11.2.1 Metadata Retention 263
11.2.2 Multiple Distribution Channels 263
11.2.3 Mobisodes and Webisodes 264
11.3 Video Distribution 264
11.3.1 Streaming Protocols 265
11.3.2 Electronic Sell Through 265
11.3.3 Peer-to-peer Delivery 266
11.3.4 Managed Download 266
11.3.5 Syndication 267
11.4 The Video Web and User Interaction 267
11.4.1 Web-Based Editing 267
11.4.2 Media Browsing 267
11.4.3 Social Tagging 268
11.4.4 Dynamic Interfaces 268
11.4.5 Video Blogs (vlogs) 269
11.4.6 Integrated Collections 269
11.5 Television Technology and Consumption 269
11.5.1 Proliferation of Channels 270
11.5.2 Live to Time Shifted 270
11.5.3 Mobile Consumption 270
11.6 Trends in Media Devices 271
11.6.1 Increased Media Capabilities 271
11.6.2 Increasing Accessibility 272
11.6.3 DRM 272
11.6.4 Home Media Systems 272
11.7 Media Processing Research 272
11.8 Deployments 275
11.9 Conclusion 276
References 276
Glossary 280
Index 286