Report on work done during the US Visit and SVT Meeting by DLBJ & Jay
April 13-29, 2002.

Itinerary

I went to the Andover Modtran/Hitran Workshop in Andover, Ma for the first week (April 15 – 19) and to the ASPRS (April 22), the EO-1 CalVal workshop (April 23) and the EO-1 SVT Team Meeting (April 24-26).

I had the ENVI/IDL hasp dongle with me and got quite a lot of processing done. Some of the processing needs to be made more efficient I think (I had the laptop chugging away during the day at the Ontar workshop) but maybe the last week of IDL learning in Melbourne can make a step in that direction.

Jay and I made a lot of progress on the work although some important work could not be completed. This document records what has been done on EO-1, largely Hyperion, as preparation for the TGARS paper.

Band Selection

We need to standardise the choice of bands so that we know what people are doing. The original 242 clearly need to be reduced in processing and there seems to be a lot of value in having a “stable” set to undertake exploration and processing that is less susceptible to noise and residual effects from atmospheric water vapour and other transmittance features.

The 196 bands

The level 1B1 processing results in 198 bands that are not set to zero. This differs from the reported 200 bands and it seems the zero bands changed with the different Levels 1An to 1B. The zero-enforced bands for Level 1B1 are:

[1-7], [58-76], [225-242]

There are 4 bands left in the overlap between the spectrometers. These are [56-57] and [77-78]. It is usual to eliminate two of these to get 196 unique bands.

Jay normally uses bands [8-56] and [78-224].

We have been using [8-57] and [79-224]. We need to resolve the difference and settle on a rational choice. Visual inspection seems to indicate band 57 is less noisy than band 78 but maybe more images need to be polled and some objective criterion used.

Note that the zero bands have changed at each level change. Luckily Level 1B1 could still be obtained from Level 1A data.

The 175 bands

While processing the reflectance data provided by AFRL Flaash it became clear that some bands had all been set to zero. I believe that is because Flaash rejects negative reflectances. The zero bands did “great” things to MNF.

Inspection of the Flaash processed data and the original data showed the bands to be at the very deepest areas of the major SWIR water absorption areas.

Leaving these out will not make a difference for land surface studies (nor atmospheric correction) unless the image is of Arizaro where the water vapour is so low even the deep water vapour bands are not down to zero transmittance. Accepting this as a good criterion for band selection gives a minimally censored “maximally useable” set of 175 bands:

VNIR: [8-56]
SWIR: [78-120], [129-166], [179-223]

This selection has used Jay’s choice of overlap bands – which still needs to be resolved. The bands [121-128], [167-178] and [224] are all deep water absorption areas as can be confirmed with any image and by Modtran and even physics as well.

These bands seem to provide a good set to remove and still retain all the “wings” of the atmospheric features as well as the water vapour areas normally used for atmospheric correction.

[Another good reason to use them came up after the Australian Workshop. We found that integrating the ASD data to Hyperion bands resulted in an error due to the wild ASD “noise” in these bands. At the time we used the 155 bands described below but I confirmed in the US that the 175 are effective in avoiding the problem with the advantage of keeping maximum data fro the atmospheric correction.]

The 155 bands

At the workshop we presented a set that had been devised based on a study of “stability”. A close relation of these was used in the IGARSS’02 paper. With some feedback from the workshop and re-thinking we are now using 155 bands as the “Stable” set. These are:

VNIR: [10-57]
SWIR: [81-97], [101-119], [134-164], [182-221]

These bands avoid the edges of atmospheric absorption areas and the Landsat and ALI bandpasses fall into the same regions of the spectrum. For many cases using only these bands will provide the most effective processing.

For example, very clean and stable images can be obtained by:

Bad pixel filling;
De-streaking;
MNF on the 175 stable bands;
Inverse MNF keeping the first 20 MNF bands;
Atmospheric correction;
Possible MNF re-smooth using 155 stable bands
Focus on the 155 band reflectances.

This is not a standard “recipe” yet but was done to illustrate combinations of noise reduction and stable band utilisation. The 175 are needed as inputs to atmospheric correction.

Some of this is discussed further below but the point is that pre-processing to remove noise and correct bad pixels can result in excellent data for indices and detailed band structure studies. Most indices of interest also lie in “stable” areas so stable processing can be very useful indeed.

Bad Pixels and Streaks

Bad Pixel List

Hyperion has pixels that are “dud” or “bad” in that the data are usually absent or of low value. It also has “streaks” where the actual data are modified by an offset and gain persistently down a line. Some streaks seem to occur in lots of images but others come and go both between images and along track. The streaking is especially a problem in the SWIR where it is much more variable. The streaks in the SWIR need to be looked at carefully as to whether they are actually “bad pixels”.

The Flaash and Hatch atmospheric correction had trouble with bad pixels and streaks. The Flaash water vapour channel was very badly streaked and they have now processed clean data as well as a comparison. It really improves the water vapour estimation although the effects on the reflectance data need to be looked at on a spatial basis.

A colleague of Dave Goodenough’s carefully checked two images and made a long list of bad pixels. However, many of these were actually “streaks”. He did, however, confirm Jenny’s observations of bad pixels not the BadPix3 list which theoretically is the most up-to-date used by TRW. He also classified the effects and it would be useful to check the results against Australian data.

Consideration of the dark image and discussions between us led to the summary presented in Jay’s talk. This needs careful confirmation on our data (but note that Jenny found a misprint in the table which needs to be fixed). We certainly need to check for the persistence of the bad pixels and ensure they are filled in. Also, the pixel 256 of the SWIR after the Level 1B1 shift is zero and should be treated as a bad pixel. Others need to be carefully treated due to the change in pixel coordinates that Level 1B1 makes in the SWIR.

One interesting observation is that some bad pixels are displaced 11 pixels from others. This is the shift in the “echo” so something involving the echo correction is involved. Lawrence Ong is checking whether his code gets the same results. It is possible that this is created by saturated pixels and the fact that they were saturated is “smeared” by processing. We should also check the log for the saturated pixels and see if there are associated bad pixels.

If all of these pixels are “fixed” it does improve the resulting images – so all this is worth doing! It needs some work and maybe the CEM pixel “plugger” – while a bit crude – is a good way to do it. If it had more robust statistics in it there may be even more value in it.

Streaks

De-streak was put to work on a number of images. Some results were reported in IGARSS and in the workshop notes. Coleambally seems to be able to be effectively de-streaked by just the statistical approach (mean and variance) but others cannot be so easily tamed. Stratification by major land cover type is essential.

Data were also processed in the US before and after atmospherically correcting. It seems it makes little or no difference to the reflectance image but it does to the water vapour image. Water vapour images become much better with de-streaking so there is most likely a good effect in the reflectance images near the water vapour bands.

Hatch and Flaash were compared by de-streaking both radiance input and reflectance images and using MNF comparisons. It was interesting that the “smile” effect persisted in MNF-1 for ALL images – even the calibrated Hatch. This is significant against the discussions that arose during the week on the smile effect and the possibility that it is more than previously estimated. It also put into question the effectiveness of the Hatch “wavelength calibration”. Alex Goetz came to the conclusion that if the reported smile effect is offset as some people suggested then his Hatch algorithm would not have been able to calibrate. So he is off trying some things…

For Coleambally, after de-streaking for either radiance or reflectance there were always 20 good MNF bands to use.

As reported above:

Bad pixel fix
Destreak
MNF and invert on 20 bands

Results in very clean data indeed. These will have to be looked at for the next workshop and have been processed by Flaash folks.

De-streaking is also important to do before binning to Landsat and ALI – otherwise the streaks get into the binned data. Someone reported this did not happen at the workshop but it did on our images!

It is clear that data cleaning and noise removal is the first step to a useful and operational result. I believe we made a big step to a final strategy to do this for Coleambally.

There is work yet to do with de-streaking. I took the final Coleambally of 1400 lines that had the bad pixels all fixed and de-streaked it and took the difference between the result and the un-de-streaked image. Then I used MNF to analyse the difference image. Clearly the de-streaking was effective but the difference image MNF showed some of the “streaking” it removed was land cover effects propagated down the columns.

There was a lot of discussion of the de-streaking at the SVT meeting. Most people are just up to statistical balancing. In the future robust and local methods will arrive and maybe someone will find a really innovative solution.

Binning

Binning is an issue. It means summing the bands of a finely resolved data set in a weighted sum to represent the data from a broadband instrument. When the bandwidth being estimated is not too much greater than that of the approximation sensor and when the sampling density of the approximating sensor is not high there are serious issues with the simple binning currently being used.

Before binning it is important that the data are consistently scaled across the bands. This is not the case with Hyperion when delivered. I changed all the data to 100 times radiance. I used the method in the Workshop notes but have now changed to a new program that takes less time and effort.

One issue is to put data sets on a consistent basis. For example, we have at times tried to compare ASD data with atmospherically corrected Hyperion data. Really you need to integrate the ASD data to Hyperion bands first.

Another is to compare radiances at TOA between Hyperion, Landsat and ALI on the same (broadband) basis. This must be done in a consistent and standardised – and document – way if it is to contribute to current discussions on cross-calibration.

Another issue is to compare Hymap data. In this case, Hyperion needs to be “binned” to Hymap. This has been done in the past weeks but not yet put into an ENVI spectral library.

I produced the matrices to do all this and made them into Spectral Libraries as described in the workshop notes. This allows them to be easily applied to images and to spectra. We need to get everyone using the same set now.

An example of the work done is in Jay’s talk. Tom sent me the Landsat image and some GCPs showing equivalent Hyperion and Landsat (GDA) points. I used a spreadsheet to equate the average ASD site locations with a landsat pixel and obtained 3 by 3 data values around the points.

In the case plotted, Landsat and Hyperion agree very well except in the band affected by water vapour. More sites need to be done now and some TOA models used to compare with the ASD data. It would have been good to have the ALI data as well but unfortunately they were not able to be sent.

I hope to be provided with the best final transformations between Hyperion, Landsat, ALI and the map to compare the data at the sites and on specific land covers. It is an important part of the binning tests as indicated below.

I have put together a table of the current set of spectral libraries below. Some of these need updating and others should be used a standard by everyone. I have also provided an added note on binning (which is fully covered in the Workshop background document) in matrix form. It is closer to what we do with spectral libraries than the more theoretical workshop background document. I hope to standardise the base files and methods of generating the others as well. It is a very nice toolkit and we produced very nice synthetic Landsat and ALI image. It was a pity we could not complete the initial comparison.

[As an aside, Jay provided me with the description of the binning Pamela used. It was in their IGARSS paper. I have some concerns about its “normalisation” – especially on narrow band data - and will try and see if we can get the actual coefficients. I have a draft paper now on how to measure how “good” a binning formula is. I suspect they usually do “OK” with the broad Landsat bands but not on narrower bands. Pamela notes that ALI band 4 is not well approximated. This will be even more the case with Hymap. The reason is the binning approach. This is also discussed further in the attached document.]

The file list follows. The filters can be applied to spectral libraries or images allowing very easy production of synthetic Landsat and ALI data from Hyperion or Hyperion data from ASD data.

File name Description Notes
1. Convert ASD Data
ASD2H_Lib ASD to Hyperion Has problems with very noisy data in deep water vapour bands
ASD2H_175_Lib ASD to Hyperion 175 bands Good results
ASD2H_155_Lib ASD to Hyperion 155 bands Very nice spectra result from this choice
ASD2A_Lib ASD to ALI Broadband data result
ASD2L_Lib ASD to Landsat ETM Broadband data result
2. Convert Hyperion Data
H2A_Lib Hyperion binning to ALI based on 196 bands Has some unstable bands at the margins
H2A_175_Lib Hyperion binning to ALI based on 175 bands Very little difference but aimed to be applied to 175 band images
H2A_155_Lib Hyperion binning to ALI based on 155 bands Applies to 155 band images.
H2L_Lib Same for Landsat
H2L_175_Lib Same for Landsat Note choice of bands in overlap
H2L_155_Lib Same for Landsat
3. Spectral Data
ASD_All_Lib All eight ASD spectra Mean of all samples at sites
ASD2H_All_175_Lib Hyperion bands restricted to the 175
ASD2H_All_155_Lib Hyperion bands restricted to the 155
Flaash_Spectra_Lib Flaash radiances at sites Average of 3 by 3 nh
Flaash_Spectra_Refl_Lib Flaash reflectances at sites
Col_Summ_ASD_Lib Coleambally summary ASD
Col_Summ_ASD2H_Lib Converted to Landsat
Col_Summ_Flaash_Lib Flaash radiance data at points


Flaash radiance data were converted to both Hyperion and ALI but only the Landsat data comparison could be made. The result was excellent.

Note that to convert Hyperion to “Aviris” we simply need to interpolate. The sliding cubic should do best. I have a set of Aviris band passes now (following the Ontar workshop) and will add these to the set so that ASD2Av etc is available.

Three attachments describe other work. One is the matrix form of the binning used. It also seems to provide concern for the previous binning. We need to get coefficients to test the results and see if there are better formulae. A second document was written to discuss SNR as it is an area of no standardisation or consistency. A third is the SPIE abstract that went to Steve. It may provide the basis for the TGARS paper outline.

Smile

Spectral smile was a topic I pursued in Andover and later at the SVT. I tried a method we have of de-smiling. It did something but was not good enough at the major edge of the VNIR where the smile is greatest.

However, at the SVT meeting both Rob Green and Barbara Carlson showed results suggesting that the form of the smile is similar to the published result but there was an offset. Barbara’s mapping of the O2-A line also showed the extra effect we could not remove with de-smiling.

It is possible that if the smile can be re-estimated vicariously then atmospheric corrections would improve and de-smiling would improve.

I summarised Rob and Barbara’s data against the TRW smile and showed results of tracking the O2-A line using local minima. They all showed a similar shape but different offsets. I showed how the use of different types of continuum correction gave very different answers.

I summarised this in a PPT file called SVT_Comments.ppt which is also part of this packaged “report”.

It is clear the problem will have to be pursued very carefully. Barbara and Rob and Alex Goetz as well as the Flaash folks at AFRL are all setting out to work on a standard set of images Lawrence is getting together. A pair of Frome images is also being selected.

The Frome subset which is all salt is shown in the PPT file. I have cobbled together an IDL routine that was used as a guinea pig for widgets, structures, pointers and other things at the IDL course. It simply collects the average spectra down columns (like de-streak) but just writes out a file on spectral means in CSV format.

For people who know the notation, if we assume we know the salt signature (which we do for the January 2001 image reasonably) and assume the neighbourhood of the salt is salt (not too bad) then:

[missing equation - ask David.Jupp@csiro.au]

With good input data this will enhance the O2 line and we should be able to carefully correlate its position across the lake. Tracking this line is not simple since it is not a clear “spike” as shown in the PPT file. But we know its form and the tracking can be done. Or maybe we can leave it to Barbara and the others – I will see how I recover in coming days!

The importance of using images at different dates is that we need to know if the smile is changing with time or is consistently off set. Maybe the differences between Frome and the others were due to the different times. This would be a scary outcome.

It may be that with some work the de-smiling will be a practical way to standardise the image results. At this time, if spectral calibrating makes the wavelength bands data dependent and (worse) pixel location dependent in a way that changes from image to image then many of the processing methods we use will be very messy or impossible without simply “ignoring” the effect and hoping for the best. Alex Goetz believes de-smiling is possible but should be done AFTER atmospheric correction. This is also a way to go. It is a good way I think.


DLBJ, May, 2002.

Copyright CSIRO 2002©

Use of this web site and information available from it is subject to our
Legal Notice and Disclaimer