Content-based Image Sequence Representation

Three dimensional video representations: We developed algorithms that process monocular video sequences and extract 3D models of rigid objects present in the scene. The shape of the object is described by patches, e.g., planar patches, or, more generally, polynomial patches. Our algorithms factor a rank one matrix to obtain the rigid 3D shape and the rigid 3D motions of the objects. Besides several papers describing this work, see for example Aguiar and Moura [2001,2003], part of this work was patented, see “System and Method for Generating a Three-dimensional Model from a Two-Dimensional Image Sequence,” (allowed by the the United States patent Office in February 2004). Earlier work fused texture information from a monocular video sequence with range laser measurements to build textured 3D models of objects, see Martins and Moura.

Modeling human motion: We extended generative video (see below) to capturing and modeling video sequences with human walkers, see our papers Cheng and Moura [1999,1998].

Generative video: Our work is concerned with developing representations for video sequences based on their content. These representations differ from those developed for MPEG/H.26X coding standards in that sequences are described in terms of extended images instead of collections of frames. We describe how these extended images, e.g., mosaics, are generated by basically the same principle: the incremental composition of visual photometric, geometric, and multi-view information into one or more extended images. Different outputs, e.g., from single 2-D mosaics to full 3-D mosaics, are obtained depending on the quality and quantity of photometric, geometric, and multi-view information. In particular, we developed a framework–generative video–that is well suited to the representation of scenes with independently moving objects. Content-based video representations can potentially provide compression ratios that are in the range of 1000:1 with acceptable quality. This work is the subject of several papers, in particular a comprehensive description is in the CRC Chapter [2004], and in the papers [1996,1995a,1995b]. Part of the work is patented, see “Generative Video: Very Low Bit Rate Video Compression.”

Video over wireless: We demonstrated the high compression at good quality provided by generative video with a very early (in 1995) demonstration of transmitting video over wireless. The wireless network interfaced two highly heterogeneous wireless networks – a ‘fast’ 2 Mbps local area network (CMU’s wireless andrew) and a slow metropolitan area ‘slow’ 19.6 Kbps wireless network. The results were reported in [1996], in an invited paper published by the IEEE Personal Communications Magazine.

Predictive lossy compression: Our work developed noncausal random field models to describe the image texture and then developed novel predictive coding algorithms that compared very favorably with transform based coders, see for example the patent “Noncausal Predictive Image Codec,” or the papers Balram and Moura [19961993], Moura and Balram [1992], and Asif and Moura [1996].

Book Chapters (additional chapters in Book Chapters)

  • Pedro M. Q. Aguiar, Radu Jasinschi, José M. F. Moura, and Charnchai Pluempitiwiriyawej, “Content-based Image Sequence Representation,” ed. Todd Reed, in Digital Image Sequence Processing: Compression and Analysis, CRC Press Handbook, in press, 2004 (61 pages). Invited Chapter.
  • José M. F. Moura and Nikhil Balram, “Statistical Algorithms for Noncausal Markov Random Fields,” in Handbook of Statistics, edts. N. K. Bose and C. R. Rao, Chapter 15, North Holland, Amsterdam, The Netherlands, July 1993. Invited Chapter.

Selected Journal Papers (additional papers in  Publications)

Selected Conference Papers (additional papers in Conference Publications)

  • Pedro M. Q. Aguiar and José M. F. Moura, “3D Rigid Structure from Video: What Are “Easy” Shapes and “Good” Motions?,” 5th IEEE Workshop on Multimedia Signal Processing, September 2002.
  • Amir Asif and José M. F. Moura, “Fast Inversion of L-Block Banded Matrices and their Inverses,” ICASSP’02, IEEE International Conference on Signal Processing, vol. 2, pp. 1369-1372, Orlando, FL, 12-17 May 2002.
  • Pedro M. Q. Aguiar and José M. F. Moura, “Image Motion Estimation – Convergence and Error Analysis,” ICIP ’01, IEEE Proceedings of International Conference on Image Processing, vol. 2, pp. 937-940, Greece, 7-10 October, 2001.
  • Pedro M. Q. Aguiar and José M. F. Moura, “Maximum Likelihood Estimation of the Template of a Rigid Moving Object,” EMMCVPR’01, Energy Minimization Methods in Computer Vision and Pattern Recognition, July 2001.
  • Pedro M. Q. Aguiar and José M. F. Moura, “Weighted Factorization,” ICIP ’00, IEEE Proceedings of International Conference on Image Processing, Vancouver, British Columbia Canada, October 2000.
  • Amir Asif and José M. F. Moura, “Block Banded Matrix Approximations: Application to Kalman-Bucy filter,” ICASSP’00, IEEE International Conference on Signal Processing, Istanbul, Turkey, June 2000.
  • Pedro M. Q. Aguiar and José M. F. Moura, “Fast 3D Modeling from Video,” 3th IEEE Workshop on Multimedia Signal Processing, Copenhagen, Denmark, December 1999.
  • Pedro M. Q. Aguiar and José M. F. Moura, “A Fast Algorithm for Rigid Structure from Image Sequences,” ICIP ’99, IEEE International Conference on Image Processing, Yokoama, Japan, October 1999.
  • Pedro M. Q. Aguiar and José M. F. Moura, “Maximum Likelihood Inference of 3D Structure from Image Sequences,” EMMCVPR’99, Energy Minimization Methods in Computer Vision and Pattern Recognition, July 1999.
  • Pedro M. Q. Aguiar and José M. F. Moura, “Factorization as a Rank 1 Problem,” CVPR’99, Computer Vision and Pattern Recognition Conference, June 1999.
  • Jia-Ching Cheng and José M. F. Moura, “Capture and Synthesis of Human Motion in Video Sequences,” 2nd IEEE Workshop on Multimedia Signal Processing, Santa Monica, CA, December 7-9, 1998.
  • Pedro. M. Q. Aguiar and José M. F. Moura, “Video Representation via 3D Shaped Mosaics.” ICIP ’98, IEEE Proceedings of International Conference on Image Processing, Chicago, Illinois, October 1998.
  • Pedro M. Q. Aguiar and José M. F. Moura, “Robust 3D Structure From Motion Under Orthography,” IEEE Image and Multidimensional Digital Signal Processing Workshop’98, Vienna, Austria, July 1998.
  • Pedro Aguiar and José M. F. Moura, “Detecting and Solving Template Ambiguities in Motion Segmentation,” ICIP’97,IEEE Proceedings of International Conference on Image ProcessingSanta Barbara, CA, October 1997.
  • Jia-Ching Cheng and José M. F. MouraTracking Human Walking in Dynamic Scenes.” ICIP’97, IEEE Proceedings of International Conference on Image Processing, Vol. I, pp. 137-140, Santa Barbara, CA, October 1997.
  • Jia-Ching Cheng and José M. F. Moura, “Model Based Recognition of Human Walking in Dynamic Scenes.” 1 st IEEE Workshop on Multimedia Signal Processing, pp. 268-273, Princeton, NJ, June 1997.
  • Amir Asif and José M. F. Moura, “Fast Recursive Reconstruction of Large Time Varying Multidimensional Fields,” ICASSP ‘97IEEE International Conference on Signal ProcessingVolume IV, pages 3037-3039, Munich, Germany, April 1997.
  • Fernando Martins, Hirohisa Shiojiri and José M. F. Moura, “3D-3D Registration of Free Formed Objects Using Shape and Texture,” VCIP ‘97Conference on Visual Communications and Image Processing, SPIE Symposium on Electronic Imaging, Science and Technology, San Jose, CA, February 1997.
  • Pedro Aguiar and José M. F. Moura, “Incremental Motion Segmentation in Low Texture,” ICIP’96, IEEE International Conference on Image Processing, Lausanne, Switzerland, September 1996.
  • Radu S. Jasinschi and José M. F. Moura, “Nonlinear Editing by Generative Video,” in ICASSP’96, IEEE International Conference on Acoustics, Speech, and Signal Processing, Vol. II, pp. 1220-1223, Atlanta, GA, May 1996. Special Session on Video Processing by Contents.
  • José M. F. Moura, R. S. Jasinschi, H. Shiojiri, and J-C. Lin, “Video Over Heterogeneous Networks,” in Proceedings of SPIE “Wireless Personal Communications Technologies and Services, vol. SPIE-2603, Invited paper in Session “PCS Data and Multimedia Applications.” Philadelphia, PA, October 1995.
  • F. Martins and José M. F. Moura, “3-D Video Compositing: Towards a Compact Representation for Video Sequences,” in ICIP’95, Proceedings of International Conference on Image Processing, vol. 1, pp. 550-553. IEEE, October 1995.
  • R. S. Jasinschi and José M. F. Moura, “Content-Based Video Sequence Representation,” in ICIP’95, Proceedings of International Conference on Image Processing, vol. 2, pp. 229-232. IEEE, October 1995.
  • R. S. Jasinschi, J. M. F. Moura, J-C. Cheng, and A. Asif, “Video Compression Via Constructs,” in ICASSP ’95, IEEE International Conference on Acoustics, Speech, and SignaProcessing, vol. IV, pp. IV-2165-2168, Detroit, MI, May 8-12, 1995.
  • A. Asif and José M. F. Moura, “Assimilation of Satellite Data in Beta-PlaneOcean Circulation Models,” ICASSP ’95IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. V, pp. V-2789-2792, Detroit, MI, May 1995. Special Session on Signal Processing in the Ocean Environment.
  • Nikhil Balram and Jose´ M. F. Moura, “Predictive Coding Using Noncausal Models,” ICASSP ‘93, IEEE International Conference on Acoustics, Speech, and Signal Processing, April 1993.
  • José M. F. Moura and Nikhil Balram, “2D Linear Optimal Statistical Signal Processing on Finite Lattices,” in Acoustic Signal Processing for Ocean Exploration, edts. José M. F. Moura and Isabel M. G. Lourtie, pp. 413-432, D. Reidel, Amsterdam, The Netherlands, January 1993.
  • José M. F. Moura and Nikhil Balram, “Optimal Prdecitive Coding of 2D Fields,” in ISIT’93IEEE International Symposium on Information Theory, San Antonio, TX, January 1993.
  • H. Tokuda, Y. Tobe, S. T.-C Chou, and J. M. F Moura. “Continuous media communication with dynamic QOS control using artss with an FDDI network.” In Proceedings of the ACM Symposium on Communications Architectures and Protocols (ACM SIGCOM/ 92 ), pages 88-98, Baltimore, October 1992.
  • Nikhil Balram and José M. F. Moura, “Parameter Estimation in 2D Fields,” ICASSP ‘92, IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 3, pp. 345-348, March 1992.
  • Nikhil Balram and José M. F. Moura, “Rapid Enhancement and Compression of Image Data,” SPIE VCIP ’91, Boston, MA, November 1991.
  • Nikhil Balram and José M. F. Moura, “Recursive Enhancement of Noncausal Images,” ICASSP ’91IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 4, pp. 2997-3000, May 1991.
  • Nikhil Balram and José M. F. Moura, “Parameter Estimation for Noncausal Gauss Markov Random Fields,” CISS’91, 25 th Annual Conference on Information Sciences and Systems, pp. 365-370, Baltimore, MD, March 1991.

Patents and disclosures

  • Generative Video: Very Low Bit Rate Video Compression,” José M. F. Moura and Radu S. Jasinschi, US Patent and Trademark Office, S.N. 5,854,856, issued December 29, 1998.
  • Noncausal Predictive Image Codec,” Nikhil Balram and José M. F. Moura, US Patent and Trademark Office, S.N. 5,689,591, issued November 18, 1997.
  • “System and Method for Generating a Three-dimensional Model from a Two-Dimensional Image Sequence,” Pedro M. Q. Aguiar and José M. F. Moura, provisional patent filed July 1999; patent filed with US Patent and Trademark Office, Serial Number 09/614,841, July 12, 2000. Notice of allowance 2/23/2004.

Lab Members

  1. Amir Asif
  2. Fernando Martins
  3. Pedro M. Q. Aguiar
  4. Jia-Ching Cheng
  5. Radu S. Jasinschi
  6. Nikhil Balram