SVCL - Query by Semantic Example

Home People Research Publications Demos

News Jobs Prospective
Students About Internal

Semantic Image Represenation (Demo)

Figure 1: An illustration of image representation on the semantic space. An image is represented as a semantic multinomial which is a weight vector obtained using an array of appearance based classifiers.

Semantic image representation is a novel image representation, that brings a paradigm shift in the way image are represented. Under semantic image representation, instead representing the images on the space of low level appearance features derived from the image, a semantic space --- a space where each dimension represents a meaningful visual concept --- is introduced, upon which the images are represented and all recognition decisions are performed. To obtain the semantic representation of an image, first a vocabulary of visual concepts is defined and statistical models are learned for all concepts in the vocabulary with existing appearance modeling techniques. Next, the outputs of these appearance classifiers are then interpreted as the dimensions of the semantic space. This is illustrated in Figure 1, where an image is represented by the vector of its posterior probabilities under each of the appearance models. This vector is denoted as a semantic multinomial (SMN) distribution as the image features themselves define a multinomial distribution over the semantic concepts.

The benefits of semantic image representation is illustrated through design of two different visual recognition systems:

Query by Semantic Example: Query-by-semantic-example (QBSE), that addresses the semantic gap. QBSE leverages on the semantic image representation by extending the query-by-example paradigm into the semantic domain, whereby the nearest neighbor retrieval operation is performed directly on the semantic space. This is shown to have two main properties of interest, one mostly practical and the other philosophical. From a practical standpoint, because QBSE has a higher level of abstraction, it enables retrieval systems with higher generalization ability that are more accurate than what was previously possible. Philosophically, because it allows a direct comparison of visual and semantic representations under a common query paradigm, QBSE enables the design of experiments that explicitly test the value of semantic representations for image retrieval.

Scene Classification with Low-dimensional Semantic Spaces: A novel approach to scene categorization is proposed. An intermediate space is introduced, based on a low dimensional semantic "theme" image representation, learned with weak supervision from casual image annotations.

Publications: Bridging the Gap: Query by Semantic Example
Rasiwasia, N., Moreno, P. L., Vasconcelos, N.
Multimedia, IEEE Transactions on,
Vol. 9(5), pp. 923-938, Aug 2007.� IEEE,[ps][pdf]

Query By Semantic Example
Nikhil Rasiwasia, Nuno Vasconcelos, Pedro J Moreno
Proceedings of the International Conference on Image and Video Retrieval LNCS 4071, pp. 51-60
Phoenix, Arizona, 2006. [ps][pdf]

Scene Classification with Low-dimensional Semantic Spaces and Weak Supervision
N. Rasiwasia and N. Vasconcelos.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
Anchorage, June 2008. �IEEE, [ps][pdf]

Contact: Nuno Vasconcelos, Nikhil Rasiwasia

Home	People	Research	Publications	Demos

News	Jobs	Prospective Students	About	Internal

Semantic Image Represenation (Demo)

Figure 1: An illustration of image representation on the semantic space. An image is represented as a semantic multinomial* which is a weight vector obtained using an array of appearance based classifiers.*
Semantic image representation is a novel image representation, that brings a paradigm shift in the way image are represented. Under semantic image representation, instead representing the images on the space of low level appearance features derived from the image, a semantic space --- a space where each dimension represents a meaningful visual concept --- is introduced, upon which the images are represented and all recognition decisions are performed. To obtain the semantic representation of an image, first a vocabulary of visual concepts is defined and statistical models are learned for all concepts in the vocabulary with existing appearance modeling techniques. Next, the outputs of these appearance classifiers are then interpreted as the dimensions of the semantic space. This is illustrated in Figure 1, where an image is represented by the vector of its posterior probabilities under each of the appearance models. This vector is denoted as a semantic multinomial (SMN) distribution as the image features themselves define a multinomial distribution over the semantic concepts.
The benefits of semantic image representation is illustrated through design of two different visual recognition systems:
Query by Semantic Example: Query-by-semantic-example (QBSE), that addresses the semantic gap. QBSE leverages on the semantic image representation by extending the query-by-example paradigm into the semantic domain, whereby the nearest neighbor retrieval operation is performed directly on the semantic space. This is shown to have two main properties of interest, one mostly practical and the other philosophical. From a practical standpoint, because QBSE has a higher level of abstraction, it enables retrieval systems with higher generalization ability that are more accurate than what was previously possible. Philosophically, because it allows a direct comparison of visual and semantic representations under a common query paradigm, QBSE enables the design of experiments that explicitly test the value of semantic representations for image retrieval.

Scene Classification with Low-dimensional Semantic Spaces: A novel approach to scene categorization is proposed. An intermediate space is introduced, based on a low dimensional semantic "theme" image representation, learned with weak supervision from casual image annotations.

Publications:	Bridging the Gap: Query by Semantic Example Rasiwasia, N., Moreno, P. L., Vasconcelos, N. Multimedia, IEEE Transactions on, Vol. 9(5), pp. 923-938, Aug 2007.� IEEE,[ps][pdf]
	Query By Semantic Example Nikhil Rasiwasia, Nuno Vasconcelos, Pedro J Moreno Proceedings of the International Conference on Image and Video Retrieval LNCS 4071, pp. 51-60 Phoenix, Arizona, 2006. [ps][pdf]
	Scene Classification with Low-dimensional Semantic Spaces and Weak Supervision N. Rasiwasia and N. Vasconcelos. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Anchorage, June 2008. �IEEE, [ps][pdf]
Contact:	Nuno Vasconcelos, Nikhil Rasiwasia