Metrics

There are five main approaches to assess the quality of cloud masking algorithms. It is important to note that all different methodologies are also bound to a specific definition of input data. The five main approaches are:  

1. Visual inspection of images covering:
   • Typical cases (Relatively simple surface types and spectral situations)
   • Critical cases (known issues for S2 spectral bands: cloud vs snow, semi-transparent clouds, small patchy cloud fields, coastlines, bright beaches, salt lakes, urban areas, etc.) 
2. Statistical assessment on global, representative scale 
   • Expert pixel collections (PixBox)
   • Manual classifications  (IRIS)
3. Self-consistency in reflectance time series 
   • Undetected clouds add noise of surface reflectance time series. Comparing the noise on this surface reflectance time series allows for comparison of the performances of different cloud masking methods 
4. Level 3 composites  
   • Temporal aggregation of surface reflectance or L2 parameters criteria  
   • “coloring” of level 3 products
   • Artifacts (i.e. whiter pixels showing cloud residuals) 
   • Number of valid observations
5. Object-oriented errors related to:
   • Over segmentation  
   • Under segmentation  
   • Edge-location  
   • Fragmentation and shape