Quality information for LP DAAC (Land Processes Distributed Active Archive Center) data products is important to consider when determining the usability and usefulness of a data product for a particular science application. However, this information has been notoriously difficult for users to access. Quality information is often stored as an integer value that requires the user to decode into binary strings. In order to interpret the binary string users must map the unique combinations of bits found in separate subsets of the binary string (i.e. bit-fields) to quality tables that characterize the particular quality attribute associated with each bit-field. This process is difficult for most users and is often passed over.
The LP DAAC has added a quality web service to its AppEEARS service API to alleviate the difficult process of decoding and interpreting quality layers for tiled MODIS and WELD products. The quality web service will decode an integer value for a data product/layer and return the quality attribute information associated with that particular integer value. Additionally, users can determine if a quality layer exists for a particular data product/layer as well as get details about individual quality layers.
Required Packages
NOTE
%autosave 0
Let's start by importing the required packages.
import requests
import pandas as pd
Next, assign the AppEEARS services API URL to a variable
SERVICES_URL = 'https://lpdaacsvc.cr.usgs.gov/services/appeears-api/'
NOTE: Placing https://lpdaacsvc.cr.usgs.gov/services/appeears-api/ in your browser will take you to the AppEEARS services API page. The page contains all of the AppEEARS services accompanied by their documentation.
Say we wish to know what layers have a quality layer associated with them. We can find that out! To start, let's print out a count of the number of layers with associated quality layers.
qualityLayers = requests.get('{}/quality?format=json'.format(SERVICES_URL)).json()
#print(qualityLayers)
'Quality information was found for {0} data layers!'.format(len(qualityLayers))
The request returns a list of dictionaries that can be indexed to filter the information returned
qualityLayers[218:222]
The list of products contained in qualityLayers is long, and trying to find the product/layer of interest via indexing can be inefficient. Instead, let's use a python expression, known as a list comprehension, to get all of the layers in MYD13A2.006 that have quality layers associated with them.
productQuality_A = [q for q in qualityLayers if q['ProductAndVersion'] == 'MYD13A2.006']
productQuality_A
If you don't want to use indexing or list comprehensions (the previous two steps), just use the product parameter in the quality service URL to get all of the layers in MYD13A2.006 that have associated quality layers.
productQuality_B = requests.get('{}/quality/MYD13A2.006?format=json'.format(SERVICES_URL)).json()
productQuality_B
productQuality_B[0]['QualityLayers'][0]
NOTE: In both cases above, we can see that EVI (_1_km_16_days_EVI) and NDVI (_1_km_16_days_NDVI) have the same quality layer (_1_km_16_days_VI_Quality). This is the case for many of the MODIS products. However, there are occasions when the layers within products have their own associated quality layers (e.g., MOD/MYD11 products).
Information within quality layers is often stored as bit-packed integers. This means that in order to get the actual quality information, one must convert the integer to binary and parse the binary string into bit-fields. Bit-fields are distinct combinations of bits representing quality categories that give an indication as to the usability and usefulness of the data product. Let's get the quality layer description for MYD13A2.006 by adding the quality layer parameter to the quality service URL.
qualityLayerInfo = requests.get('{}/quality/MYD13A2.006/_1_km_16_days_VI_Quality?format=json'.format(SERVICES_URL)).json()
#qualityLayerInfo
pd.DataFrame(qualityLayerInfo)
NOTE: The table above contains the description for each bit-field in MYD13A2.006's quality layer (i.e., _1_km_16_days_VI_Quality). For more information of quality layer descriptions, please consult with the product pages on the LP DAAC website.
Say that we have extracted a pixel value from the EVI layer (i.e., _1_km_16_days_EVI) of product MYD13A2.006. We can use the quality service to decode the associated quality value for the pixel by inserting the pixel value from the quality layer (_1_km_16_days_VI_Quality) in the quality service URL. We see that our pixel value from the quality layer equals 4160 and that our EVI value equals 0.3091.
qualityIntDecoder = requests.get('{}/quality/MYD13A2.006/_1_km_16_days_VI_Quality/4160?format=json'.format(SERVICES_URL)).json()
qualityIntDecoder
Display qualityIntDecoder as Pandas dataframe
pd.DataFrame(qualityIntDecoder)
The above code snippet returns the decoded quality information for value 4160, however, the order in which the quality categories are displayed does not match the order in which the bit-fields are assembled in the binary string. To correct this, the same service call can be used, but the manner in which we bring the request into python changes.
import json
from collections import OrderedDict
qualityIntDecoder = json.loads(
requests.get('{}/quality/MYD13A2.006/_1_km_16_days_VI_Quality/4160?format=json'.format(SERVICES_URL)).text,
object_pairs_hook=OrderedDict)
Now let's print out the decoded quality value in its proper order.
#qualityIntDecoder
pd.DataFrame(qualityIntDecoder)
We can see in the MODLAND column that this is a good quality pixel and the Land/Water Flag column indicates that this pixel is likely near a lake shoreline (given that the pixel extracted was from tile h10v04).
Finally, we can take the decoded python object (a python dictionary) and filter it based on the bit-field description.
qualityIntDecoder['MODLAND']