EVALUATION OF IMAGE QUALITY METRICS

Chief investigator: Martin Čadík, email: cadikm@centrum.cz, homepage: http://cadik.posvete.cz/

Project web page: http://cadik.posvete.cz/iqm/

Project description:

The goal of the project is the evaluation of state-of the art image quality metrics. We conduct both the subjective and objective comparisons of metrics.

FULL-REFERENCE IMAGE QUALITY METRICS

Full-reference image quality metrics

IMPLEMENTED IMAGE QUALITY METRICS

Original image	Distorted image	Map of differences	Method
			S. Daly The Visible Differences Predictor: An Algorithm for the Assessment of Image Fidelity
			Z. Wang, A. C. Bovik A Universal Image Quality Index http://www.cns.nyu.edu/~zwang/files/papers/uqi.html Matlab code available
			Z. Wang, A. C. Bovik, H. R. Seikh, E. P. Simoncelli Image quality assessment: From error visibility to structural similarity Matlab code available
			A. Shnayderman, A. Gusev, A. M. Eskicioglu An SVD-Based Gray-Scale Image Quality Measure for Local and Global Assessment http://www.sci.brooklyn.cuny.edu/~eskicioglu/papers Published also in [An SVD-Based Gray-Scale Image Quality Measure for Local and Global Assessment]
DONE	DONE	DONE	Tunc Ozan Aydin, Rafal Mantiuk, Karol Myszkowski, Hans-Peter Seidel Dynamic Range Independent Image Quality Assessment 3 types of distortion: loss of visible contrast, amplification of invisible contrast, reversal of visible contrast, project page (online implementation)
DONE	DONE	DONE	Rafal Mantiuk, Kil Joong Kim, Allan G. Rempel, Wolfgang Heidrich HDR-VDP-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions
DONE	DONE	DONE	R. Mantiuk, S. Daly, K. Myszkowski, H.-P. Seidel Predicting Visible Differences in High Dynamic Range Images - Model and its Calibration Extends VDP for HDR, also published in [Visible Difference Predicator for High Dynamic Range Images] http://www.mpi-inf.mpg.de/resources/hdr/vdp/
DONE	DONE	DONE	Kuang, J., Johnson, G.M., Fairchild M.D. iCAM06: A refined image appearance model for HDR image rendering Extends iCAM, MatLab code available

METHODS TO IMPLEMENT (i.e. possible student projects/master's thesis topics)

TODO	J. L. Mannos, D. J. Sakrison The Effects of a visual fidelity criterion on the encoding of images
TODO	H. Rushmeier, G. Ward, C. Piatko, P. Sanders Comparing Real and Synthetic Images: Some Ideas About Metrics Model after [Gervais84]
TODO	Patrick C. Teo, David J. Heeger Perceptual image distortion
TODO	J. Lubin A visual discrimination model for imaging system design and evaluation
TODO	C.C. Taylor, Z. Pizlo, J.P. Allebach, C.A. Bouman Image Quality Assessment with a Gabor Pyramid Model of the Human Visual System
TODO	M.R.M. Nijenhuis, F.J.J. Blommaert Perceptual-error measure and its application to sampled and interpolated single-edged images
TODO	L. Neumann, K. Matkovic, W. Purgathofer Perception based color image difference
TODO	A. P. Bradley A Wavelet Visible Difference Predictor
TODO	J. Oh, S. I. Wooley, T. N. Arvanitis, J. N. Townend A multistage perceptual quality assessment for compressed digital angiogram images
TODO	X. Tong, D. Heeger, C. J. van den B. Lambrecht Video quality evaluation using ST-CIELAB
TODO	Alakuijala, Jyrki; Obryk, Robert; Stoliarchuk, Ostap; Szabadka, Zoltan; Vandevenne, Lode Guetzli: Perceptually Guided JPEG Encoder
TODO	Abasi, Saeedeh; Amani Tehran, Mohammad; Fairchild, Mark D. Distance metrics for very large color differences
TODO	Zhang, Richard; Isola, Phillip; Efros, Alexei A.; Shechtman, Eli; Wang, Oliver The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
TODO	Nilsson, Jim; Akenine-Möller, Tomas Understanding SSIM
TODO	Prashnani, Ekta; Cai, Hong; Mostofi, Yasamin; Sen, Pradeep PieAPP: Perceptual Image-Error Assessment Through Pairwise Preference
TODO	Andersson, Pontus; Jim Nilsson, Nvidia; Tomas Akenine-möller, Nvidia; Magnus Oskarsson, Nvidia; Åström, Kalle; Fairchild, Mark D; Nilsson, Jim; Akenine-Möller, Tomas; Oskarsson, Magnus FLIP: A Difference Evaluator for Alternating Images
TODO	Wolski, Krzysztof; Giunchi, Daniele; Ye, Nanyang; Didyk, Piotr; Myszkowski, Karol; Mantiuk, Radosław; Seidel, Hans Peter; Steed, Anthony; Mantiuk, Rafał K. Dataset and metrics for predicting local visible differences
TODO	Banterle, Francesco; Artusi, Alessandro; Moreo, Alejandro NOR-VDPNET : A NO-REFERENCE HIGH DYNAMIC RANGE QUALITY METRIC TRAINED ON HDR-VDP 2
TODO	Artusi, Alessandro; Banterle, Francesco; Carra, Fabio; Moreno, Alejandro Efficient evaluation of image quality via deep-learning approximation of perceptual metrics

OUR PUBLICATIONS

*Author*	*Title*	*Publishing* *details*
Čadík, M., Herzog, R., Mantiuk, R., Mantiuk, R., Myszkowski, K., Seidel, H.P.	Learning to Predict Localized Distortions in Rendered Images [Paper (pdf)] [Supplementary Material (html)] [Presentation slides (pdf)]	Computer Graphics Forum Vol. 32, Num. 7 (proc. of Pacific Graphics'13), pp. 401-410, 2013.
Čadík, M., Herzog, R., Mantiuk, R., Myszkowski, K., Seidel, H.P.	New Measurements Reveal Weaknesses of Image Quality Metrics in Evaluating Graphics Artifacts [Paper (pdf)] [Supplementary materials (pdf)]	ACM Transactions on Graphics (SIGGRAPH Asia 2012), vol. 31, no. 6, DOI: 10.1145/2366145.2366161, ACM, 2012.
Herzog, R., Čadík, M., Aydin, T. O., Kwang I. K., Myszkowski, K., Seidel, H.P.	NoRM: No-Reference Image Quality Metric for Realistic Image Synthesis [Paper (pdf)] [Supplementary materials (pdf)] [Dataset and User Study]	Computer Graphics Forum, vol. 31, issue 2 (2012), (Proceedings of Eurographics'12, Cagliari / Italy, 13–18 May), 2012.
Čadík, M., Aydin, T.O., Myszkowski, K., Seidel, H.P.	On Evaluation of Video Quality Metrics [Paper (pdf)] [Presentation slides (pdf)] [bibTeX entry (bib)] [HDR Dataset (905MB zip file)] [Browse HDR Dataset]	Proc. of IS&T/SPIE's Human Vision and Electronic Imaging, 2011. Copyright 2011 Society of Photo-Optical Instrumentation Engineers. One print or electronic copy may be made for personal use only. Systematic reproduction and distribution, duplication of any material in this paper for a fee or for commercial purposes, or modification of the content of the paper are prohibited.
Aydin, T.O., Čadík, M., Myszkowski, K., Seidel, H.P.	Video Quality Assessment for Computer Graphics Applications [Video Quality Metric Online] [Paper (pdf)] [Video (mp4)] [Validation experiment (pdf)] [Formulae (pdf)] [bibTeX entry (bib)]	ACM Transactions on Graphics (SIGGRAPH Asia 2010), vol. 29, no. 5, ISSN 0730-0301, ACM, 2010.
Čadík, M.	Perceptually Based Image Quality Assessment and Image Transformations [bibTeX entry (bib)]	Ph.D. Thesis, Czech Technical University in Prague, 2008. Outstanding dissertation award, President of CTU in Prague, December 2009.
Čadík, M., Slavík, P.	Evaluation of Two Principal Approaches to Objective Image Quality Assessment [bibTeX entry (bib)]	8th International Conference on Information Visualization. Los Alamitos: IEEE Computer Society Press, 2004.
Čadík, M.	Perceptual Image Quality Assessment Metrics	CODATA Information Visualization Workshop 2004. Prague, 2004.
Čadík, M.	Human Perception and Computer Graphics	Postgraduate Study Report. Czech Technical University in Prague, 2004.
Čadík, M., Slavík, P.	Comparing Image-Processing Operators by Means of the Visible Differences Predictor	WSCG 2004. Pilsen: University of West Bohemia, 2004.

RESOURCES

Our experimental data and input images
Our HDR video quality dataset
Our LOCCG dataset (localized distortions in computer graphics) for evaluation of image quality metrics
Our LOCCG visual saliency dataset
Our CLFM (contrast-luminance-frequency-masking) dataset for low-level evaluation of IQMs

Image and Video Quality Assessment at LIVE
Video Quality Experts Group (VQEG)