The Application for Extracting and Exploring Analysis Ready Samples (AρρEEARS) offers a simple and efficient way to access and transform geospatial data from a variety of federal data archives in an easy-to-use web application interface. AρρEEARS enables users to subset geospatial data spatially, temporally, and by band/layer for point and area samples. AρρEEARS returns not only the requested data, but also the associated quality values, and offers interactive visualizations with summary statistics in the web interface. The AρρEEARS API offers users programmatic access to all features available in AρρEEARS, with the exception of visualizations. The API features are demonstrated in this notebook.
Connect to the AρρEEARS API, query the list of available products, submit an area sample request, download the request, become familiar with the AρρEEARS Quality API, and import the results into Python for visualization. AρρEEARS area sample requests allow users to subset their desired data by spatial area via vector polygons (shapefiles or GeoJSONs). Users can also reproject and reformat the output data. AρρEEARS returns the valid data from the parameters defined within the sample request.
To access AρρEEARS, visit: https://lpdaacsvc.cr.usgs.gov/appeears/
For comprehensive documentation of the full functionality of the AρρEEARS API, please see the AρρEEARS API Documentation
Throughout the tutorial, specific sections of the API documentation can be accessed by clicking on the bracketed [] links in the section headings.
# Import packages
import requests as r
import getpass, pprint, time, os, cgi, json
import geopandas as gpd
# Set input directory, change working directory
inDir = 'D:/appeears-api-getting-started/' # IMPORTANT: Update to reflect directory on your OS
os.chdir(inDir) # Change to working directory
api = 'https://lpdaacsvc.cr.usgs.gov/appeears/api/' # Set the AρρEEARS API to a variable
getpass
package to enter your NASA Earthdata login Username and Password. When prompted after executing the code block below, enter your username followed by your password.¶user = getpass.getpass(prompt = 'Enter NASA Earthdata Login Username: ') # Input NASA Earthdata Login Username
password = getpass.getpass(prompt = 'Enter NASA Earthdata Login Password: ') # Input NASA Earthdata Login Password
requests
package to post your username and password. A successful login will provide you with a token to be used later in this tutorial to submit a request. For more information or if you are experiencing difficulties, please see the API Documentation.¶token_response = r.post('{}login'.format(api), auth=(user, password)).json() # Insert API URL, call login service, provide credentials & return json
del user, password # Remove user and password information
token_response # Print response
product_response = r.get('{}product'.format(api)).json() # request all products in the product service
print('AρρEEARS currently supports {} products.'.format(len(product_response))) # Print no. products available in AppEEARS
products = {p['ProductAndVersion']: p for p in product_response} # Create a dictionary indexed by product name & version
products['MCD15A3H.006'] # Print information for MCD15A3H.006 LAI/FPAR Product
prodNames = {p['ProductAndVersion'] for p in product_response} # Make list of all products (including version)
for p in prodNames: # Make for loop to search list of products 'Description' for a keyword
if 'Leaf Area Index' in products[p]['Description']:
pprint.pprint(products[p]) # Print info for each product containing LAI in its description
MCD15A3H.006
.¶prods = ['MCD15A3H.006'] # Start a list for products to be requested, beginning with MCD15A3H.006
prods.append('MOD11A2.006') # Append the MOD11A2.006 8 day LST product to the list of products desired
prods.append('SRTMGL1_NC.003') # Append the SRTMGL1_NC.003 product to the list of products desired
prods # Print list
ProductAndVersion
property. For a list of the layer names only, print the keys from the dictionary below.¶lst_response = r.get('{}product/{}'.format(api, prods[1])).json() # Request layers for the 2nd product (index 1) in the list: MOD11A2.006
list(lst_response.keys())
'LST_Day_1km'
to see the information for that layer in the response.¶lst_response['LST_Day_1km'] # Print layer response
layers = [(prods[1],'LST_Day_1km'),(prods[1],'LST_Night_1km')] # Create tupled list linking desired product with desired layers
MCD15A3H.006
product.¶lai_response = r.get('{}product/{}'.format(api, prods[0])).json() # Request layers for the 1st product (index 0) in the list: MCD15A3H.006
list(lai_response.keys()) # Print the LAI layer names
lai_response['Lai_500m']['Description'] # Make sure the correct layer is requested
layers.append((prods[0],'Lai_500m')) # Append to tupled list linking desired product with desired layers
SRTMGL1_NC.003
product.¶dem_response = r.get('{}product/{}'.format(api, prods[2])).json() # Request layers for the 3rd product (index 2) in the list: SRTMGL1_NC.003
list(dem_response.keys()) # Print the SRTM DEM layer names
SRTMGL1_DEM
to the tupled list of desired products/layers.¶layers.append((prods[2], 'SRTMGL1_DEM')) # Append to tupled list linking desired product with desired layers
prodLayer = []
for l in layers:
prodLayer.append({
"layer": l[1],
"product": l[0]
})
prodLayer
token = token_response['token'] # Save login token to a variable
head = {'Authorization': 'Bearer {}'.format(token)} # Create a header to store token information, needed to submit a request
geopandas
package. The shapefile is publically available for download from the NPS website.¶nps = gpd.read_file('{}nps_boundary.shp'.format(inDir + os.sep)) # Read in shapefile as dataframe using geopandas
print(nps.head()) # Print first few lines of dataframe
geopandas
dataframe for the national park that you are interested in using for your region of interest, here Grand Canyon National Park.¶nps_gc = nps[nps['UNIT_NAME']=='Grand Canyon National Park'].to_json() # Extract Grand Canyon NP and set to variable
nps_gc = json.loads(nps_gc) # Convert to json format
projections = r.get('{}spatial/proj'.format(api)).json() # Call to spatial API, return projs as json
projections # Print projections and information
Name
as the keys.¶projs = {} # Create an empty dictionary
for p in projections: projs[p['Name']] = p # Fill dictionary with `Name` as keys
list(projs.keys()) # Print dictionary keys
projs['geographic']
task_name = input('Enter a Task Name: ') # User-defined name of the task: 'NPS Vegetation Area' used in example
task_type = ['point','area'] # Type of task, area or point
proj = projs['geographic']['Name'] # Set output projection
outFormat = ['geotiff', 'netcdf4'] # Set output file format type
startDate = '07-01-2017' # Start of the date range for which to extract data: MM-DD-YYYY
endDate = '07-31-2017' # End of the date range for which to extract data: MM-DD-YYYY
recurring = False # Specify True for a recurring date range
#yearRange = [2000,2016] # if recurring = True, set yearRange, change start/end date to MM-DD
nps_gc
is inserted from the shapefile transformed to a json via the geopandas
and json
packages above in section 3a.¶task = {
'task_type': task_type[1],
'task_name': task_name,
'params': {
'dates': [
{
'startDate': startDate,
'endDate': endDate
}],
'layers': prodLayer,
'output': {
'format': {
'type': outFormat[0]},
'projection': proj},
'geo': nps_gc,
}
}
task
json created above.¶task_response = r.post('{}task'.format(api), json=task, headers=head).json() # Post json to the API task service, return response as json
task_response # Print task response
pretty
to True to format the response as an organized json, making it easier to read. Additional information on AρρEEARS API pagination and formatting can be found in the API documentation.¶params = {'limit': 2, 'pretty': True} # Limit API response to 2 most recent entries, return as pretty json
tasks_response = r.get('{}task'.format(api), params=params, headers=head).json() # Query task service, setting params and header
tasks_response # Print tasks response
task_response
that was generated when submitting your request, and use the AρρEEARS API status service to check the status of your request.¶task_id = task_response['task_id'] # Set task id from request submission
status_response = r.get('{}status/{}'.format(api, task_id), headers=head).json() # Call status service with specific task ID & user credentials
status_response
# Ping API until request is complete, then continue to Section 4
starttime = time.time()
while r.get('{}task/{}'.format(api, task_id), headers=head).json()['status'] != 'done':
print(r.get('{}task/{}'.format(api, task_id), headers=head).json()['status'])
time.sleep(20.0 - ((time.time() - starttime) % 20.0))
print(r.get('{}task/{}'.format(api, task_id), headers=head).json()['status'])
destDir = os.path.join(inDir, task_name) # Set up output directory using input directory and task name
if not os.path.exists(destDir):os.makedirs(destDir) # Create the output directory
bundle = r.get('{}bundle/{}'.format(api,task_id)).json() # Call API and return bundle contents for the task_id as json
bundle # Print bundle contents
files = {} # Create empty dictionary
for f in bundle['files']: files[f['file_id']] = f['file_name'] # Fill dictionary with file_id as keys and file_name as values
files
files
dictionary and a for
loop to automate downloading all of the output files into the output directory.¶for f in files:
dl = r.get('{}bundle/{}/{}'.format(api, task_id, f), stream=True) # Get a stream to the bundle file
filename = os.path.basename(cgi.parse_header(dl.headers['Content-Disposition'])[1]['filename']) # Parse the name from Content-Disposition header
filepath = os.path.join(destDir, filename) # Create output file path
with open(filepath, 'wb') as f: # Write file to dest dir
for data in dl.iter_content(chunk_size=8192): f.write(data)
print('Downloaded files can be found at: {}'.format(destDir))
offset
which allows you to set the number of results to skip before starting to return entries. Next, make a call to list all of the data product layers and the associated quality product and layer information.¶params = {'limit': 6, 'pretty': True, 'offset': 20} # Limit response to 6 entries, start w/ 20th entry, return pretty json
quality_response = r.get('{}quality'.format(api), params=params).json() # Call quality API using pagination and return json
quality_response # Print response
product = 'MCD15A3H.006' # Product used in the example
ql_response = r.get('{}quality/{}'.format(api,product)).json() # Call API to retrieve quality layers for selected product
ql_response # Print response
qlayer = ql_response[1]['QualityLayers'][0] # Set quality layer from ql_response for 'Lai_500m'
qv_response = r.get('{}quality/{}/{}'.format(api, product, qlayer)).json() # Call API for list of bit-word quality values
qv_response # Print response
val = 1 # Set a specific value
q_response = r.get('{}quality/{}/{}/{}'.format(api, product, qlayer, val)).json() # Call quality API for specific value
q_response # Print response
# Import packages
import matplotlib.pyplot as plt
import numpy as np
from osgeo import gdal
list(files.values()) # List files downloaded
dem = gdal.Open(destDir + '/SRTMGL1_NC.003_SRTMGL1_DEM_doy2000042_aid0001.tif' ) # Read file in
demBand = dem.GetRasterBand(1) # Read the band (layer)
demData = demBand.ReadAsArray().astype('float') # Import band as an array with type float
nan
.¶demFill = demBand.GetNoDataValue() # Returns fill value
demData[demData == demFill] = np.nan # Set fill value to nan
# Set matplotlib plots inline
%matplotlib inline
plt.imshow(demData); # Visualize a basic plot of the DEM data
import warnings
warnings.filterwarnings("ignore")
fig = plt.figure(figsize = (10,7.5)) # Set the figure size (x,y)
plt.axis('off') # Remove the axes' values
ax = fig.add_subplot(111)
# Plot the array, using a colormap and setting a custom linear stretch based on the min/max Elevation values
plt.imshow(demData, vmin = np.nanmin(demData), vmax = np.nanmax(demData), cmap = 'terrain');
plt.style.use("dark_background") # Default to a black background
fig2 = plt.figure(figsize=(10,7.5)) # Set the figure size
plt.axis('off') # Remove the axes' values
ax1 = fig2.add_subplot(111) # Make a subplot
fig2.subplots_adjust(top=3.8) # Adjust spacing
ax1.set_title('SRTM DEM: Grand Canyon NP',fontsize=15,fontweight='bold',color='white') # Add title
# Plot the masked data, using a colormap and setting a custom linear stretch based on the min/max DEM values
im = plt.imshow(demData, vmin = np.nanmin(demData), vmax = np.nanmax(demData), cmap = 'terrain');
cb = plt.colorbar(im, orientation='horizontal', fraction=0.047, pad=0.004, shrink=0.6) # Add a colormap legend
cb.set_label(label='Elevation (m)', color = 'white') # Set Label and color
cb.outline.set_edgecolor('white') # Set edge color
Contact: LPDAAC@usgs.gov
Voice: +1-605-594-6116
Organization: Land Processes Distributed Active Archive Center (LP DAAC)
Website: https://lpdaac.usgs.gov/
Date last modified: 04-28-2020
$^{1}$Innovate! Inc., contractor to the U.S. Geological Survey, Earth Resources Observation and Science (EROS) Center, Sioux Falls, South Dakota, 57198-001, USA. Work performed under USGS contract G15PD00467 for LP DAAC$^{2}$.
$^{2}$LP DAAC Work performed under NASA contract NNG14HH33I.