Getting Started with GEDI L2B Data in Python

Details:

Published: May 13, 2020

Getting Started with GEDI L2B Data in Python

This tutorial demonstrates how to work with the Canopy Cover and Vertical Profile Metrics (GEDI02_B.001) data product.

The Global Ecosystem Dynamics Investigation (GEDI) mission aims to characterize ecosystem structure and dynamics to enable radically improved quantification and understanding of the Earth's carbon cycle and biodiversity. The GEDI instrument produces high resolution laser ranging observations of the 3-dimensional structure of the Earth. GEDI is attached to the International Space Station and collects data globally between 51.6° N and 51.6° S latitudes at the highest resolution and densest sampling of any light detection and ranging (lidar) instrument in orbit to date. The Land Processes Distributed Active Archive Center (LP DAAC) distributes the GEDI Level 1 and Level 2 products. The L1B and L2 GEDI products are archived and distributed in the HDF-EOS5 file format.


Use Case Example:

This tutorial was developed using an example use case for a project being completed by the National Park Service. The goal of the project is to use GEDI L2B data to observe tree canopy height, cover, and profile over Redwood National Park in northern California.

This tutorial will show how to use Python to open GEDI L2B files, visualize the full orbit of GEDI points (shots), subset to a region of interest, visualize GEDI canopy height and vertical profile metrics, and export subsets of GEDI science dataset (SDS) layers as GeoJSON files that can be loaded into GIS and/or Remote Sensing software programs.


Data Used in the Example:

  • GEDI L2B Canopy Cover and Vertical Profile Metrics Data Global Footprint Level - GEDI02_B.001
    • The purpose of the L2B dataset is to extract biophysical metrics from each GEDI waveform. These metrics are based on the directional gap probability profile derived from the L1B waveform and include canopy cover, Plant Area Index (PAI), Plant Area Volume Density (PAVD) and Foliage Height Diversity (FHD).
    • Science Dataset (SDS) layers:
      • /geolocation/digital_elevation_model
      • /geolocation/elev_lowestmode
      • /geolocation/elev_highestreturn
      • /geolocation/lat_lowestmode
      • /geolocation/lon_lowestmode
      • /rh100
      • /l2b_quality_flag
      • /degrade_flag
      • /sensitivity
      • /pai
      • /pavd_z
      • /geolocation/shot_number
      • /dz

Topics Covered:

  1. Get Started
    1.1 Import Packages
    1.2 Set Up the Working Environment and Retrieve Files
  2. Import and Interpret Data
    2.1 Open a GEDI HDF5 File and Read File Metadata
    2.2 Read SDS Metadata and Subset by Beam
  3. Visualize a GEDI Orbit
    3.1 Subset by Layer and Create a Geodataframe
    3.2 Visualize a Geodataframe
  4. Work with GEDI L2B Data
    4.1 Import and Extract PAVD
    4.2 Visualize PAVD
  5. Work with GEDI L2B Beam Transects
    5.1 Quality Filtering
    5.2 Plot Beam Transects
    5.3 Subset Beam Transects
  6. Plot Profile Transects
    6.1 Plot PAVD Transects
  7. Spatial Visualization
    7.1 Import, Subset, and Quality Filter all Beams
    7.2 Spatial Subsetting
    7.3 Visualize All Beams: Canopy Height, Elevation, and PAI
  8. Export Subsets as GeoJSON Files

Before Starting this Tutorial:

Setup and Dependencies

It is recommended to use Conda, an environment manager to set up a compatible Python environment. Download Conda for your OS here: https://www.anaconda.com/download/. Once you have Conda installed, Follow the instructions below to successfully setup a Python environment on Linux, MacOS, or Windows.

This Python Jupyter Notebook tutorial has been tested using Python version 3.7. Conda was used to create the python environment.

  • Using your preferred command line interface (command prompt, terminal, cmder, etc.) type the following to successfully create a compatible python environment:

    conda create -n geditutorial -c conda-forge --yes python=3.7 h5py shapely geopandas pandas geoviews holoviews

conda activate geditutorial

jupyter notebook

If you do not have Jupyter Notebook installed, you may need to run:

conda install jupyter notebook

Having trouble getting a compatible Python environment set up? Contact LP DAAC User Services.

If you prefer to not install Conda, the same setup and dependencies can be achieved by using another package manager such as pip.


Example Data:

This tutorial uses the GEDI L2B observation from June 19, 2019 (orbit 02932). Use the links below to download the files directly from the LP DAAC Data Pool:
- https://e4ftl01.cr.usgs.gov/GEDI/GEDI02_B.001/2019.06.19/GEDI02_B_2019170155833_O02932_T02267_02_001_01.h5 (1.33 GB)

A NASA Earthdata Login account is required to download the data used in this tutorial. You can create an account at the link provided.

You will need to have the file above downloaded into the same directory as this Jupyter Notebook in order to successfully run the code below.

Source Code used to Generate this Tutorial:

The repository containing all of the required files is located at: https://git.earthdata.nasa.gov/projects/LPDUR/repos/gedi-tutorials/browse
- Jupyter Notebook
- Redwood National Park GeoJSON
- Contains the administrative boundary for Redwood National Park, available from: Administrative Boundaries of National Park System Units 12/31/2017 - National Geospatial Data Asset (NGDA) NPS National Parks Dataset

NOTE: This tutorial was developed for GEDI L2B HDF-EOS5 files and should only be used for that product.

1. Get Started

1.1 Import Packages

Import the required packages and set the input/working directory to run this Jupyter Notebook locally.

import os
import h5py
import numpy as np
import pandas as pd
import geopandas as gp
from shapely.geometry import Point
import geoviews as gv
from geoviews import opts, tile_sources as gvts
import holoviews as hv
gv.extension('bokeh', 'matplotlib')

1.2 Set Up the Working Environment and Retrieve Files

The input directory is defined as the current working directory. Note that you will need to have the Jupyter Notebook and example data (.h5 and .geojson) stored in this directory in order to execute the tutorial successfully.

inDir = os.getcwd() + os.sep  # Set input directory to the current working directory
NOTE: If you have downloaded the tutorial materials to a different directory than the Jupyter Notebook, `inDir` above needs to be changed. You will also need to add a line: `os.chdir(inDir)` and execute it below.

In this section, a GEDI .h5 file has been downloaded to the inDir defined above. You will need to download the file directly from the LP DAAC Data Pool in order to execute this tutorial.

Direct Link to file:

  • https://e4ftl01.cr.usgs.gov/GEDI/GEDI02_B.001/2019.06.19/GEDI02_B_2019170155833_O02932_T02267_02_001_01.h5

Make sure to download the files into the inDir directory defined above.

gediFiles = [g for g in os.listdir() if g.startswith('GEDI02_B') and g.endswith('.h5')]  # List all GEDI L2B .h5 files in inDir
gediFiles
['GEDI02_B_2019170155833_O02932_T02267_02_001_01.h5']

2. Import and Interpret Data

2.1 Open a GEDI HDF5 File and Read File Metadata

Read the file using h5py.

L2B = 'GEDI02_B_2019170155833_O02932_T02267_02_001_01.h5'
L2B
'GEDI02_B_2019170155833_O02932_T02267_02_001_01.h5'

The standard format for GEDI filenames is as follows:

GEDI02_B: Product Short Name
2019170155833: Julian Date and Time of Acquisition (YYYYDDDHHMMSS)
O02932: Orbit Number
T02267: Track Number
02: Positioning and Pointing Determination System (PPDS) type (00 is predict, 01 rapid, 02 and higher is final)
001: GOC SDS (software) release number
01: Granule Production Version

Read in a GEDI HDF5 file using the h5py package.

gediL2B = h5py.File(L2B, 'r')  # Read file using h5py

Navigate the HDF5 file below.

list(gediL2B.keys())
['BEAM0000',
 'BEAM0001',
 'BEAM0010',
 'BEAM0011',
 'BEAM0101',
 'BEAM0110',
 'BEAM1000',
 'BEAM1011',
 'METADATA']

The GEDI HDF5 file contains groups in which data and metadata are stored.

First, the METADATA group contains the file-level metadata.

list(gediL2B['METADATA'])
['DatasetIdentification']

This contains useful information such as the creation date, PGEVersion, and VersionID. Below, print the file-level metadata attributes.

for g in gediL2B['METADATA']['DatasetIdentification'].attrs: print(g)
PGEVersion
VersionID
abstract
characterSet
creationDate
credit
fileName
language
originatorOrganizationName
purpose
shortName
spatialRepresentationType
status
topicCategory
uuid
print(gediL2B['METADATA']['DatasetIdentification'].attrs['purpose'])
The purpose of the L2B dataset is to extract biophysical metrics from each GEDI waveform. These metrics are based on the directional gap probability profile derived from the L1B waveform and include canopy cover, Plant Area Index (PAI), Plant Area Volume Density (PAVD) and Foliage Height Diversity (FHD).

2.2 Read SDS Metadata and Subset by Beam

The GEDI instrument consists of 3 lasers producing a total of 8 beam ground transects. The eight remaining groups contain data for each of the eight GEDI beam transects. For additional information, be sure to check out: https://gedi.umd.edu/instrument/specifications/.

beamNames = [g for g in gediL2B.keys() if g.startswith('BEAM')]
beamNames
['BEAM0000',
 'BEAM0001',
 'BEAM0010',
 'BEAM0011',
 'BEAM0101',
 'BEAM0110',
 'BEAM1000',
 'BEAM1011']

One useful piece of metadata to retrieve from each beam transect is whether it is a full power beam or a coverage beam.

for g in gediL2B['BEAM0000'].attrs: print(g)
description
wp-l2-l2b_githash
wp-l2-l2b_version
for b in beamNames:
    print(f"{b} is a {gediL2B[b].attrs['description']}")
BEAM0000 is a Coverage beam
BEAM0001 is a Coverage beam
BEAM0010 is a Coverage beam
BEAM0011 is a Coverage beam
BEAM0101 is a Full power beam
BEAM0110 is a Full power beam
BEAM1000 is a Full power beam
BEAM1011 is a Full power beam

Below, pick one of the full power beams that will be used to retrieve GEDI L2B shots in Section 3.

beamNames = ['BEAM0110']

Identify all the objects in the GEDI HDF5 file below.

Note: This step may take a while to complete.

gediL2B_objs = []
gediL2B.visit(gediL2B_objs.append)                                           # Retrieve list of datasets
gediSDS = [o for o in gediL2B_objs if isinstance(gediL2B[o], h5py.Dataset)]  # Search for relevant SDS inside data file
[i for i in gediSDS if beamNames[0] in i][0:10]                              # Print the first 10 datasets for selected beam
['BEAM0110/algorithmrun_flag',
 'BEAM0110/ancillary/dz',
 'BEAM0110/ancillary/l2a_alg_count',
 'BEAM0110/ancillary/maxheight_cuttoff',
 'BEAM0110/ancillary/rg_eg_constraint_center_buffer',
 'BEAM0110/ancillary/rg_eg_mpfit_max_func_evals',
 'BEAM0110/ancillary/rg_eg_mpfit_maxiters',
 'BEAM0110/ancillary/rg_eg_mpfit_tolerance',
 'BEAM0110/ancillary/signal_search_buff',
 'BEAM0110/ancillary/tx_noise_stddev_multiplier']

3. Visualize a GEDI Orbit

In the section below, import GEDI L2B SDS layers into a GeoPandas GeoDataFrame for the beam specified above.

Use the lat_lowestmode and lon_lowestmode to create a shapely point for each GEDI shot location.

3.1 Subset by Layer and Create a Geodataframe

Read in the SDS and take a representative sample (every 100th shot) and append to lists, then use the lists to generate a pandas dataframe.

lonSample, latSample, shotSample, qualitySample, beamSample = [], [], [], [], []  # Set up lists to store data

# Open the SDS
lats = gediL2B[f'{beamNames[0]}/geolocation/lat_lowestmode'][()]
lons = gediL2B[f'{beamNames[0]}/geolocation/lon_lowestmode'][()]
shots = gediL2B[f'{beamNames[0]}/geolocation/shot_number'][()]
quality = gediL2B[f'{beamNames[0]}/l2b_quality_flag'][()]

# Take every 100th shot and append to list
for i in range(len(shots)):
    if i % 100 == 0:
        shotSample.append(str(shots[i]))
        lonSample.append(lons[i])
        latSample.append(lats[i])
        qualitySample.append(quality[i])
        beamSample.append(beamNames[0])

# Write all of the sample shots to a dataframe
latslons = pd.DataFrame({'Beam': beamSample, 'Shot Number': shotSample, 'Longitude': lonSample, 'Latitude': latSample,
                         'Quality Flag': qualitySample})
latslons
Beam Shot Number Longitude Latitude Quality Flag
0 BEAM0110 29320618800000001 111.996300 -51.803868 0
1 BEAM0110 29320604600000101 112.039132 -51.803905 0
2 BEAM0110 29320614600000201 112.080271 -51.803836 0
3 BEAM0110 29320600400000301 112.121445 -51.803737 0
4 BEAM0110 29320610400000401 112.162622 -51.803621 0
... ... ... ... ... ...
9792 BEAM0110 29320617400979201 88.208452 -51.803578 0
9793 BEAM0110 29320603200979301 88.249610 -51.803614 0
9794 BEAM0110 29320613200979401 88.290753 -51.803581 0
9795 BEAM0110 29320623200979501 88.331913 -51.803548 0
9796 BEAM0110 29320609000979601 88.373089 -51.803506 0

9797 rows × 5 columns

Above is a dataframe containing columns describing the beam, shot number, lat/lon location, and quality information about each shot.

# Clean up variables that will no longer be needed
del beamSample, quality, qualitySample, gediL2B_objs, latSample, lats, lonSample, lons, shotSample, shots

Below, create an additional column called 'geometry' that contains a shapely point generated from each lat/lon location from the shot.

# Take the lat/lon dataframe and convert each lat/lon to a shapely point
latslons['geometry'] = latslons.apply(lambda row: Point(row.Longitude, row.Latitude), axis=1)

Next, convert to a Geopandas GeoDataFrame.

# Convert to a Geodataframe
latslons = gp.GeoDataFrame(latslons)
latslons = latslons.drop(columns=['Latitude','Longitude'])
latslons['geometry']
0       POINT (111.99630 -51.80387)
1       POINT (112.03913 -51.80391)
2       POINT (112.08027 -51.80384)
3       POINT (112.12145 -51.80374)
4       POINT (112.16262 -51.80362)
                   ...             
9792     POINT (88.20845 -51.80358)
9793     POINT (88.24961 -51.80361)
9794     POINT (88.29075 -51.80358)
9795     POINT (88.33191 -51.80355)
9796     POINT (88.37309 -51.80351)
Name: geometry, Length: 9797, dtype: geometry

Pull out and plot an example shapely point below.

latslons['geometry'][0]

png

3.2 Visualize a GeoDataFrame

In this section, use the GeoDataFrame and the geoviews python package to spatially visualize the location of the GEDI shots on a basemap and import a geojson file of the spatial region of interest for the use case example: Redwood National Park.

# Define a function for visualizing GEDI points
def pointVisual(features, vdims):
    return (gvts.EsriImagery * gv.Points(features, vdims=vdims).options(tools=['hover'], height=500, width=900, size=5,
                                                                        color='yellow', fontsize={'xticks': 10, 'yticks': 10,
                                                                                                  'xlabel':16, 'ylabel': 16}))

Import a geojson of Redwood National Park as an additional GeoDataFrame. Note that you will need to have downloaded the geojson from the bitbucket repo containing this tutorial and have it saved in the same directory as this Jupyter Notebook.

redwoodNP = gp.GeoDataFrame.from_file('RedwoodNP.geojson')  # Import geojson as GeoDataFrame
redwoodNP
GIS_LOC_ID UNIT_CODE GROUP_CODE UNIT_NAME UNIT_TYPE META_MIDF LANDS_CODE DATE_EDIT GIS_NOTES geometry
0 None REDW None Redwood National Park None None None Shifted 0.06 miles MULTIPOLYGON (((-124.01829 41.44539, -124.0184...
redwoodNP['geometry'][0]  # Plot GeoDataFrame

png

Defining the vdims below will allow you to hover over specific shots and view information about them.

# Create a list of geodataframe columns to be included as attributes in the output map
vdims = []
for f in latslons:
    if f not in ['geometry']:
        vdims.append(f)
vdims
['Beam', 'Shot Number', 'Quality Flag']

Below, combine a plot of the Redwood National Park Boundary (combine two geoviews plots using *) with the point visual mapping function defined above in order to plot (1) the representative GEDI shots, (2) the region of interest, and (3) a basemap layer.

# Call the function for plotting the GEDI points
gv.Polygons(redwoodNP).opts(line_color='red', color=None) * pointVisual(latslons, vdims = vdims)

png

Above is a good illustration of the full GEDI orbit (GEDI files are stored as one ISS orbit). One of the benefits of using geoviews is the interactive nature of the output plots. Use the tools to the right of the map above to zoom in and find the shots intersecting Redwood National Park.

(HINT: find where the orbit intersects the west coast of the United States)

Below is a screenshot of the region of interest:

alt text

Side Note: Wondering what the 0's and 1's for l2b_quality_flag mean?

print(f"Quality Flag: {gediL2B[b]['l2b_quality_flag'].attrs['description']}")
Quality Flag: Flag simpilfying selection of most useful data for Level 2B

Above, 0 is poor quality and a quality_flag value of 1 indicates the laser shot meets criteria based on energy, sensitivity, amplitude, and real-time surface tracking quality. We will show an example of how to quality filter GEDI data in section 5.1.

After finding one of the shots within Redwood NP, find the index for that shot number so that we can find the correct shot to visualize in Section 4.

Each GEDI shot has a unique shot identifier (shot number) that is available within each data group of the product. The shot number is important to retain in any data subsetting as it will allow the user to link any shot record back to the original orbit data, and to link any shot and its data between the L1 and L2 products. The standard format for GEDI Shots is as follows:

Shot: 29320619900465601

2932: Orbit Number
06: Beam Number
199: Minor frame number (0-241)
00465601: Shot number within orbit

del latslons  # No longer need the geodataframe used to visualize the full GEDI orbit

4. Work with GEDI L2B Data

The L2B product contains biophysical information derived from the geolocated GEDI return waveforms including total and vertical profiles of canopy cover and Plant Area Index (PAI), the vertical Plant Area Volume Density (PAVD) profile, and Foliage Height Diversity (FHD).

Detailed product information can be found on the GEDI L2B Product Page.

4.1 Import and Extract Specific Shots

Notice that there are over a thousand datasets available in the GEDI L2B product. In the code blocks below, you will subset to just a few of the datasets available.

In this section, learn how to extract and subset specific shots and plot Plant Area Volume Density (PAVD) using holoviews.

len(gediSDS)
1488
beamNames
['BEAM0110']
beamSDS = [g for g in gediSDS if beamNames[0] in g]  # Subset to a single beam
len(beamSDS)
186

We will set the shot index used as an example from the GEDI L1B Tutorial and GEDI L2A Tutorial to show how to subset a single shot of GEDI L2B data.

shot = 29320619500465599
index = np.where(gediL2B[f'{beamNames[0]}/shot_number'][()]==shot)[0][0]  # Set the index for the shot identified above
index
465598

4.2 Visualize PAVD

In section 4.2, import the PAVD metrics (pavd_z) and begin exploring how to plot them.

pavd = gediL2B[[g for g in beamSDS if g.endswith('/pavd_z')][0]]  # PAVD

Print the description for the PAVD dataset.

print(f"Plant Area Volume Density is {pavd.attrs['description']}")
Plant Area Volume Density is Vertical Plant Area Volume Density profile with a vertical step size of dZ

Below, open the dz layer in order to define the correct vertical step size.

# Grab vertical step size
dz = gediL2B[f'{beamNames[0]}/ancillary/dz'][0]
dz
5.0

So the vertical step size is 5.0 meters.

print(f"The shape of PAVD is {pavd.shape}.")
The shape of PAVD is (979699, 30).

And it looks like PAVD includes 30 "steps" in each shot, describing the PAVD at height = step # * dz.

Now, bring in other useful L2B datasets such as elev_lowestmode, lat_lowestmode and lon_lowestmode.

# Bring in the desired SDS
elev = gediL2B[f'{beamNames[0]}/geolocation/elev_lowestmode'][()]  # Latitude
lats = gediL2B[f'{beamNames[0]}/geolocation/lat_lowestmode'][()]  # Latitude
lons = gediL2B[f'{beamNames[0]}/geolocation/lon_lowestmode'][()]  # Longitude

Grab the location, elevation, and PAVD metrics for the shot defined above:

shotElev = elev[index]
shotLat = lats[index]
shotLon = lons[index]
shotPAVD = pavd[index]

Put everything together to identify the shot that we want to extract:

print(f"The shot is located at: {str(shotLat)}, {str(shotLon)} (shot ID: {shot}, index {index}) and is from {beamNames[0]}.")
The shot is located at: 41.28472739326018, -124.03109998658007 (shot ID: 29320619500465599, index 465598) and is from BEAM0110.

Next, reformat PAVD into a list of tuples containing each PAVD value and height.

pavdAll = []
pavdElev = []

for i, e in enumerate(range(len(shotPAVD))):
    if shotPAVD[i] > 0:
        pavdElev.append((shot, shotElev + dz * i, shotPAVD[i]))  # Append tuple of shot number, elevation, and PAVD
pavdAll.append(pavdElev)                                         # Append to final list

Below, plot each shot by using holoviews Path() function, with the PAVD plotted in the third dimension in shades of green.

path1 = hv.Path(pavdAll, vdims='PAVD').options(color='PAVD', clim=(0,0.13), cmap='Greens', line_width=20, colorbar=True,
                                               width=700, height=550, clabel='PAVD', xlabel='Shot Number', tools=['hover'],
                                               ylabel='Elevation (m)', fontsize={'title':16, 'xlabel':16, 'ylabel': 16,
                                                                                 'xticks':12, 'yticks':12,
                                                                                 'clabel':12, 'cticks':10})
path1

png

Congratulations! You have plotted your first PAVD profile.


5. Work with GEDI L2B Beam Transects

Next, import a number of desired SDS layers for BEAM0110 (for the entire orbit) and create a pandas Dataframe to store the arrays.

# Open all of the desired SDS
dem = gediL2B[[g for g in beamSDS if g.endswith('/digital_elevation_model')][0]][()]
zElevation = gediL2B[[g for g in beamSDS if g.endswith('/elev_lowestmode')][0]][()]
zHigh = gediL2B[[g for g in beamSDS if g.endswith('/elev_highestreturn')][0]][()]
zLat = gediL2B[[g for g in beamSDS if g.endswith('/lat_lowestmode')][0]][()]
zLon = gediL2B[[g for g in beamSDS if g.endswith('/lon_lowestmode')][0]][()]
canopyHeight = gediL2B[[g for g in beamSDS if g.endswith('/rh100')][0]][()]
quality = gediL2B[[g for g in beamSDS if g.endswith('/l2b_quality_flag')][0]][()]
degrade = gediL2B[[g for g in beamSDS if g.endswith('/degrade_flag')][0]][()]
sensitivity = gediL2B[[g for g in beamSDS if g.endswith('/sensitivity')][0]][()]
pavd = gediL2B[f'{beamNames[0]}/pavd_z'][()]
shotNums = gediL2B[f'{beamNames[0]}/shot_number'][()]

# Create a shot index
shotIndex = np.arange(shotNums.size)

In the GEDI L2B product, Canopy Height is stored in units (cm), so below convert to meters.

canopyHeight = canopyHeight / 100  # Convert RH100 from cm to m

As mentioned in the sections above, Plant Area Volume Density (pavd) is defined as the Vertical Plant Area Volume Density profile with a vertical step size of dZ. Below, reformat the shape of the PAVD layer in order to add it to the dataframe below.

print(f"The shape of Canopy Height is {canopyHeight.shape} vs. the shape of PAVD, which is {pavd.shape}.")
The shape of Canopy Height is (979699,) vs. the shape of PAVD, which is (979699, 30).

Above, notice that unlike a SDS layer like Canopy Height, which has a single value for each shot, PAVD has 30 values (representing different vertical heights) for each shot.

Below, reformat the data into a list of values for each shot.

# Set up an empty list to append to
pavdA = []
for i in range(len(pavd)):

    # If any of the values are fill value, set to nan
    pavdF = [np.nan]
    for p in range(len(pavd[i])):
        if pavd[i][p]!= -9999:
            pavdF.append(pavd[i][p])  # If the value is not fill value, append to list
    pavdA.append(pavdF)               # Append back to master list

Note: The cell above may take up to a minute to process.

Below, notice the reformatted PAVD layer, which should now fit into the dataframe created below.

len(pavdA)
979699
# Take the DEM, GEDI-produced Elevation, and Canopy height and add to a Pandas dataframe
transectDF = pd.DataFrame({'Shot Index': shotIndex, 'Shot Number': shotNums, 'Latitude': zLat, 'Longitude': zLon,
                           'Tandem-X DEM': dem, 'Elevation (m)': zElevation, 'Canopy Elevation (m)': zHigh,
                           'Canopy Height (rh100)': canopyHeight, 'Quality Flag': quality, 'Degrade Flag': degrade,
                           'Plant Area Volume Density': pavdA, 'Sensitivity': sensitivity})
transectDF
Shot Index Shot Number Latitude Longitude Tandem-X DEM Elevation (m) Canopy Elevation (m) Canopy Height (rh100) Quality Flag Degrade Flag Plant Area Volume Density Sensitivity
0 0 29320618800000001 -51.803868 111.996300 -999999.0 21242.515625 21242.515625 0.0 0 0 [nan] -3.436965
1 1 29320618900000002 -51.803867 111.996712 -999999.0 21242.505859 21242.505859 0.0 0 0 [nan] 30.496670
2 2 29320619000000003 -51.803867 111.997123 -999999.0 21242.496094 21242.496094 0.0 0 0 [nan] 8.071431
3 3 29320619100000004 -51.803867 111.997535 -999999.0 21242.484375 21242.484375 0.0 0 0 [nan] -212.896439
4 4 29320619200000005 -51.803866 111.997946 -999999.0 21242.474609 21242.474609 0.0 0 0 [nan] -6.853874
... ... ... ... ... ... ... ... ... ... ... ... ...
979694 979694 29320618400979695 -51.803445 88.411747 -999999.0 18017.906250 18017.906250 0.0 0 0 [nan] 22.138037
979695 979695 29320618500979696 -51.803445 88.412159 -999999.0 18017.296875 18017.296875 0.0 0 0 [nan] 4.475757
979696 979696 29320618600979697 -51.803444 88.412570 -999999.0 18017.884766 18017.884766 0.0 0 0 [nan] 10.112548
979697 979697 29320618700979698 -51.803444 88.412981 -999999.0 18017.275391 18017.275391 0.0 0 0 [nan] 424.691803
979698 979698 29320618800979699 -51.803443 88.413393 -999999.0 18017.263672 18017.263672 0.0 0 0 [nan] 15.887813

979699 rows × 12 columns

Notice the unusual values listed above--those shots are flagged as poor quality and will be removed in Section 5.1.

Now that you have the desired SDS into a pandas dataframe, begin plotting the entire beam transect:

# Plot Canopy Height
canopyVis = hv.Scatter((transectDF['Shot Index'], transectDF['Canopy Height (rh100)']))
canopyVis.opts(color='darkgreen', height=500, width=900, title=f'GEDI L2B Full Transect {beamNames[0]}',
               fontsize={'title':16, 'xlabel':16, 'ylabel': 16}, size=0.1, xlabel='Shot Index', ylabel='Canopy Height (m)')

png

Congratulations! You have plotted your first GEDI full orbit beam transect. Notice above that things look a little messy--before we dive deeper into plotting full transects, let's quality filter the shots in the section below.

del canopyVis, canopyHeight, degrade, dem, pavd, pavdA, quality, sensitivity, shotIndex, shotNums, zElevation, zHigh, zLat, zLon

5.1 Quality Filtering

Now that you have the desired layers imported as a dataframe for the entire beam transect, let's perform quality filtering.

Below, remove any shots where the l2b_quality_flag is set to 0 by defining those shots as nan.

The syntax of the line below can be read as: in the dataframe, find the rows "where" the quality flag is not equal (ne) to 0. If a row (shot) does not meet the condition, set all values equal to nan for that row.

transectDF = transectDF.where(transectDF['Quality Flag'].ne(0))  # Set any poor quality returns to NaN
transectDF
Shot Index Shot Number Latitude Longitude Tandem-X DEM Elevation (m) Canopy Elevation (m) Canopy Height (rh100) Quality Flag Degrade Flag Plant Area Volume Density Sensitivity
0 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
1 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
2 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
3 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
4 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
... ... ... ... ... ... ... ... ... ... ... ... ...
979694 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
979695 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
979696 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
979697 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
979698 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN

979699 rows × 12 columns

Below, quality filter even further by using the degrade_flag (Greater than zero if the shot occurs during a degrade period, zero otherwise) and the Sensitivity layer, using a threshold of 0.95.

transectDF = transectDF.where(transectDF['Degrade Flag'].ne(1))
transectDF = transectDF.where(transectDF['Sensitivity'] > 0.95)

Below, drop all of the shots that did not pass the quality filtering standards outlined above from the transectDF.

transectDF = transectDF.dropna()  # Drop all of the rows (shots) that did not pass the quality filtering above
print(f"Quality filtering complete, {len(transectDF)} high quality shots remaining.")
Quality filtering complete, 66317 high quality shots remaining.

5.2 Plot Beam Transects

Next, plot the full remaining transect of high quality values using holoviews Scatter(). Combine the Tandem-X derived elevation, the GEDI-derived elevation, and the Canopy Top Elevation in a combined holoviews plot.

# Plot Digital Elevation Model
demVis = hv.Scatter((transectDF['Shot Index'], transectDF['Tandem-X DEM']), label='Tandem-X DEM')
demVis = demVis.opts(color='black', height=500, width=900, fontsize={'xlabel':16, 'ylabel': 16}, size=1.5)
# Plot GEDI-Retrieved Elevation
zVis = hv.Scatter((transectDF['Shot Index'], transectDF['Elevation (m)']), label='GEDI-derived Elevation')
zVis = zVis.opts(color='saddlebrown', height=500, width=900, fontsize={'xlabel':16, 'ylabel': 16}, size=1.5)
# Plot Canopy Top Elevation
rhVis = hv.Scatter((transectDF['Shot Index'], transectDF['Canopy Elevation (m)']), label='Canopy Top Elevation')
rhVis = rhVis.opts(color='darkgreen', height=500, width=900, fontsize={'xlabel':16, 'ylabel': 16}, size=1.5,
                   tools=['hover'], xlabel='Shot Index', ylabel='Elevation (m)')
# Combine all three scatterplots
(demVis * zVis * rhVis).opts(show_legend=True, legend_position='top_left',fontsize={'title':15, 'xlabel':16, 'ylabel': 16},
                             title=f'{beamNames[0]} Full Transect: {L2B.split(".")[0]}')

png

The plot still looks a bit messy this far zoomed out--feel free to pan, zoom, and explore different areas of the plot. The waveforms plotted in section 4 were 46597-46600. If you zoom into the high-quality shots between 4.000e+5 and 5.000e+5, you will find the portion of the transect intersecting Redwood National Park, seen below:

alt text

5.3 Subset Beam Transects

Now, subset down to a smaller transect centered on the shot analyzed in the sections above.

print(index)
465598
# Grab 50 points before and after the shot visualized above
start = index - 50
end = index + 50
print(f"The transect begins at ({transectDF['Latitude'][start]}, {transectDF['Longitude'][start]}) and ends at ({transectDF['Latitude'][end]}, {transectDF['Longitude'][end]}).")
The transect begins at (41.26951477815523, -124.05868759659765) and ends at (41.299873598763384, -124.00358737366548).

Below, subset the transect using .loc.

transectDF = transectDF.loc[start:end]  # Subset the Dataframe to only the selected region of interest over Redwood NP

6. Plot Profile Transects

In this section, plot the transect subset using elevation, canopy height, and plant area volume density (PAVD) metrics.

In order to get an idea of the length of the beam transect that you are plotting, you can plot the x-axis as distance, which is calculated below.

# Calculate along-track distance
distance = np.arange(0.0, len(transectDF.index) * 60, 60)  # GEDI Shots are spaced 60 m apart
transectDF['Distance'] = distance                          # Add Distance as a new column in the dataframe

6.1 Plot PAVD Transects

Similar to what was done with PAVD in the sections above, reformat PAVD into a list of tuples containing each PAVD value and height by shot.

pavdAll = []
for j, s in enumerate(transectDF.index):
    pavdShot = transectDF['Plant Area Volume Density'][s]
    elevShot = transectDF['Elevation (m)'][s]
    pavdElev = []

    # Remove fill values
    if np.isnan(pavdShot).all():
        continue
    else:
        del pavdShot[0]
    for i, e in enumerate(range(len(pavdShot))):
        if pavdShot[i] > 0:
            pavdElev.append((distance[j], elevShot + dz * i, pavdShot[i]))  # Append tuple of distance, elevation, and PAVD
    pavdAll.append(pavdElev)                                                # Append to final list
canopyElevation = [p[-1][1] for p in pavdAll]  # Grab the canopy elevation by selecting the last value in each PAVD

Below, plot each shot by using holoviews Path() function, with the PAVD plotted in the third dimension in shades of green.

path1 = hv.Path(pavdAll, vdims='PAVD').options(color='PAVD', clim=(0,0.3), cmap='Greens', line_width=8, colorbar=True,
                                               width=950, height=500, clabel='PAVD', xlabel='Distance Along Transect (m)',
                                               ylabel='Elevation (m)', fontsize={'title':16, 'xlabel':16, 'ylabel': 16,
                                                                                 'xticks':12, 'yticks':12,
                                                                                 'clabel':12, 'cticks':10})
path1

png

Add in the ground elevation and canopy top elevation for better context as to where in the canopy the highest PAVD exists.

path2 = hv.Curve((distance, transectDF['Elevation (m)']), label='Ground Elevation').options(color='black', line_width=2)
path3 = hv.Curve((distance, canopyElevation), label='Canopy Top Elevation').options(color='grey', line_width=1.5)
# Plot all three together
path = path1 * path2 * path3
path.opts(height=500,width=980, ylim=(min(transectDF['Elevation (m)']) - 5, max(canopyElevation) + 5),
          xlabel='Distance Along Transect (m)', ylabel='Elevation (m)', legend_position='bottom_right',
          fontsize={'title':15, 'xlabel':15, 'ylabel': 15, 'xticks': 14, 'yticks': 14, 'legend': 14},
          title=f'GEDI L2B {beamNames[0]} PAVD over Redwood National Park on June 19, 2019')

png

Above, you can get an idea about the terrain over the region of interest, particularly the classic "V" representing the river valley that is bisected by the transect. In terms of vegetation structure, this plot does a good job of showing not only which portions of the canopy are taller, but also where they are denser (darker shades of green).

del distance, canopyElevation, pavdAll, pavdElev, pavdShot, transectDF

At this point you have visualized the elevation, canopy, and vertical structure of specific footprints over Redwood national park, and for a transect cutting through the national park. In section 7 you will look at mapping all of the high-quality shots from all eight GEDI beams for a given region of interest in order to gain knowledge on the spatial distribution of and characteristics of the canopy over Redwood National Park.


7. Spatial Visualization

Section 7 combines many of the techniques learned above including how to import GEDI datasets, perform quality filtering, spatial subsetting, and visualization.

7.1 Import, Subset, and Quality Filter All Beams

Below, re-open the GEDI L2B observation--but this time, loop through and import data for all 8 of the GEDI beams.

beamNames = [g for g in gediL2B.keys() if g.startswith('BEAM')]
beamNames
['BEAM0000',
 'BEAM0001',
 'BEAM0010',
 'BEAM0011',
 'BEAM0101',
 'BEAM0110',
 'BEAM1000',
 'BEAM1011']

Loop through each of the desired datasets (SDS) for each beam, append to lists, and transform into a pandas DataFrame.

# Set up lists to store data
shotNum, dem, zElevation, zHigh, zLat, zLon, canopyHeight, quality, degrade, sensitivity, pai, beamI = ([] for i in range(12))
# Loop through each beam and open the SDS needed
for b in beamNames:
    [shotNum.append(h) for h in gediL2B[[g for g in gediSDS if g.endswith('/shot_number') and b in g][0]][()]]
    [dem.append(h) for h in gediL2B[[g for g in gediSDS if g.endswith('/digital_elevation_model') and b in g][0]][()]]
    [zElevation.append(h) for h in gediL2B[[g for g in gediSDS if g.endswith('/elev_lowestmode') and b in g][0]][()]]  
    [zHigh.append(h) for h in gediL2B[[g for g in gediSDS if g.endswith('/elev_highestreturn') and b in g][0]][()]]  
    [zLat.append(h) for h in gediL2B[[g for g in gediSDS if g.endswith('/lat_lowestmode') and b in g][0]][()]]  
    [zLon.append(h) for h in gediL2B[[g for g in gediSDS if g.endswith('/lon_lowestmode') and b in g][0]][()]]  
    [canopyHeight.append(h) for h in gediL2B[[g for g in gediSDS if g.endswith('/rh100') and b in g][0]][()]]  
    [quality.append(h) for h in gediL2B[[g for g in gediSDS if g.endswith('/l2b_quality_flag') and b in g][0]][()]]  
    [degrade.append(h) for h in gediL2B[[g for g in gediSDS if g.endswith('/degrade_flag') and b in g][0]][()]]  
    [sensitivity.append(h) for h in gediL2B[[g for g in gediSDS if g.endswith('/sensitivity') and b in g][0]][()]]  
    [beamI.append(h) for h in [b] * len(gediL2B[[g for g in gediSDS if g.endswith('/shot_number') and b in g][0]][()])]  
    [pai.append(h) for h in gediL2B[f'{b}/pai'][()]]    
# Convert lists to Pandas dataframe
allDF = pd.DataFrame({'Shot Number': shotNum, 'Beam': beamI, 'Latitude': zLat, 'Longitude': zLon, 'Tandem-X DEM': dem,
                      'Elevation (m)': zElevation, 'Canopy Elevation (m)': zHigh, 'Canopy Height (rh100)': canopyHeight,
                      'Quality Flag': quality, 'Plant Area Index': pai,'Degrade Flag': degrade, 'Sensitivity': sensitivity})
del beamI, canopyHeight, degrade, dem, gediSDS, pai, quality, sensitivity, zElevation, zHigh, zLat, zLon, shotNum

7.2 Spatial Subsetting

Below, subset the pandas dataframe using a simple bounding box region of interest. If you are interested in spatially clipping GEDI shots to a geojson region of interest, be sure to check out the GEDI-Subsetter python script available at: https://git.earthdata.nasa.gov/projects/LPDUR/repos/gedi-subsetter/browse.

len(allDF)
3547051

Over 3.5 million shots are contained in this single GEDI orbit! Below subset down to only the shots falling within this small bounding box encompassing Redwood National Park. RedwoodNP our geopandas geodataframe can be called for the "envelope" or smallest bounding box encompassing the entire region of interest. Here, use that as the bounding box for subsetting the GEDI shots.

redwoodNP.envelope[0].bounds
(-124.16015705494489,
 41.080601363502545,
 -123.84950230520286,
 41.83981133687605)
minLon, minLat, maxLon, maxLat = redwoodNP.envelope[0].bounds  # Define the min/max lat/lon from the bounds of Redwood NP

Filter by the bounding box, which is done similarly to filtering by quality in section 6.1 above.

allDF = allDF.where(allDF['Latitude'] > minLat)
allDF = allDF.where(allDF['Latitude'] < maxLat)
allDF = allDF.where(allDF['Longitude'] > minLon)
allDF = allDF.where(allDF['Longitude'] < maxLon)
allDF = allDF.dropna()  # Drop shots outside of the ROI
len(allDF)
4477

Notice you have drastically reduced the number of shots you are working with (which will greatly enhance your experience in plotting them below). But first, remove any poor quality shots that exist within the ROI.

# Set any poor quality returns to NaN
allDF = allDF.where(allDF['Quality Flag'].ne(0))
allDF = allDF.where(allDF['Degrade Flag'].ne(1))
allDF = allDF.where(allDF['Sensitivity'] > 0.95)
allDF = allDF.dropna()
len(allDF)
2077

Down to roughly 2000 shots, next create a Shapely Point out of each shot and insert it as the geometry column in the [soon to be geo]dataframe.

# Take the lat/lon dataframe and convert each lat/lon to a shapely point
allDF['geometry'] = allDF.apply(lambda row: Point(row.Longitude, row.Latitude), axis=1)
# Convert to geodataframe
allDF = gp.GeoDataFrame(allDF)
allDF = allDF.drop(columns=['Latitude','Longitude'])

7.3 Visualize All Beams: Canopy Height, Elevation, and PAI

Now, using the pointVisual function defined in section 3.2, plot the geopandas GeoDataFrame using geoviews.

allDF['Shot Number'] = allDF['Shot Number'].astype(str)  # Convert shot number to string

vdims = []
for f in allDF:
    if f not in ['geometry']:
        vdims.append(f)

visual = pointVisual(allDF, vdims = vdims)
visual * gv.Polygons(redwoodNP).opts(line_color='red', color=None)

png

Feel free to pan and zoom in to the GEDI shots in yellow.

Now let's not only plot the points in the geodataframe but also add a colormap for Canopy Height (m), Elevation (m), and Plant Area Index (PAI).

allDF['Canopy Height (rh100)'] = allDF['Canopy Height (rh100)'] / 100  # Convert canopy height from cm to m
# Plot the basemap and geoviews Points, defining the color as the Canopy Height for each shot
(gvts.EsriImagery * gv.Points(allDF, vdims=vdims).options(color='Canopy Height (rh100)',cmap='plasma', size=3, tools=['hover'],
                                                          clim=(0,102), colorbar=True, clabel='Meters',
                                                          title='GEDI Canopy Height over Redwood National Park: June 19, 2019',
                                                          fontsize={'xticks': 10, 'yticks': 10, 'xlabel':16, 'clabel':12,
                                                                    'cticks':10,'title':16,'ylabel':16})).options(height=500,
                                                                                                                  width=900)

png

Above and in the screenshot below, notice the higher canopy heights (shades of yellow) over the Redwood stands of the national park vs. other types of forests (pink-blue) vs. the low-lying (and consequently flat) profiles over lakes and rivers (purple).

alt text

Next, take a look at the GEDI-derived elevation over the shots. Notice below that the colormap is changed to 'terrain'.

(gvts.EsriImagery * gv.Points(allDF, vdims=vdims).options(color='Elevation (m)',cmap='terrain', size=3, tools=['hover'],
                                                          clim=(min(allDF['Elevation (m)']), max(allDF['Elevation (m)'])),
                                                          colorbar=True, clabel='Meters',
                                                          title='GEDI Elevation over Redwood National Park: June 19, 2019',
                                                          fontsize={'xticks': 10, 'yticks': 10, 'xlabel':16, 'clabel':12,
                                                                    'cticks':10,'title':16,'ylabel':16})).options(height=500,
                                                                                                                  width=900)

png

Last but certainly not least, Plant Area Index:

(gvts.EsriImagery * gv.Points(allDF, vdims=vdims).options(color='Plant Area Index',cmap='Greens', size=3, tools=['hover'],
                                                          clim=(0,1), colorbar=True, clabel='m2/m2',
                                                          title='GEDI PAI over Redwood National Park: June 19, 2019',
                                                          fontsize={'xticks': 10, 'yticks': 10, 'xlabel':16, 'clabel':12,
                                                                    'cticks':10,'title':16,'ylabel':16})).options(height=500,
                                                                                                                  width=900)

png

Success! You have now learned how to start working with GEDI L2B files in Python as well as some interesting strategies for visualizing those data in order to better understand your specific region of interest. Using this Jupyter Notebook as a workflow, you should now be able to switch to GEDI files over your specific region of interest and re-run the notebook. Good Luck!


8. Export Subsets as GeoJSON Files

In this section, export the GeoDataFrame as a .geojson file that can be easily opened in your favorite remote sensing and/or GIS software and will include an attribute table with all of the shots/values for each of the SDS layers in the dataframe.

gediL2B.filename  # L2B Filename
'GEDI02_B_2019170155833_O02932_T02267_02_001_01.h5'
outName = gediL2B.filename.replace('.h5', '.json')  # Create an output file name using the input file name
outName
'GEDI02_B_2019170155833_O02932_T02267_02_001_01.json'
allDF.to_file(outName, driver='GeoJSON')  # Export to GeoJSON
del allDF

Contact Information

Material written by Cole Krehbiel1

    Contact: LPDAAC@usgs.gov
    Voice: +1-605-594-6116
    Organization: Land Processes Distributed Active Archive Center (LP DAAC)
    Website: https://lpdaac.usgs.gov/
    Date last modified: 05-08-2020
1KBR Inc., contractor to the U.S. Geological Survey, Earth Resources Observation and Science (EROS) Center, Sioux Falls, South Dakota, 57198-001, USA. Work performed under USGS contract G15PD00467 for LP DAAC2. 2LP DAAC Work performed under NASA contract NNG14HH33I.

Relevant Products

Product Long Name
GEDI02_B.001 GEDI L2B Canopy Cover and Vertical Profile Metrics Data Global Footprint Level

Tools

Name Filters Description