MODIS Meeting, EOC Canberra, 29 August 2002

Record of discussion on processing software for post-Level 1B products

Ian Grant, CAR

Acronyms

Summary

L1B background There are two software packages to take MODIS Direct Broadcast (DB) data to Level 1B (L1B), NASA/Miami and IMAPP from Wisconsin, with minor differences in L1B format. L1B products from the DAACs have the format produced by the NASA package. Local L1B production was discussed the following day.

Validation There is a scientific need for validation in Australia that the Australian community should take up, perhaps by involvement with the international validation teams. Validation is necessary before modifying post_L1B software, and includes determining how to adjust algorithm parameters to suit local conditions. BoM and ACRES managers are reluctant to distribute unvalidated products.

Software access Whatever DAAC software that is accessible should be made available in Australia so that locally produced products, possibly modified from standard products, have a heritage traceable to the DAAC products.

Software management A repository and management system for source code should be established to document changes. This job would be distributed across ACRES, BoM and CSIRO. Algorithm improvements should be fed back to the US software providers.

Formats Both DOLA and ACRES currently use MS2GT to produce binary products from L1B. BoM will need L1B data in HDF format from several reception sites. Australian L1B data interchange should standardise on an HDF format that looks like the DAAC L1B data, except possibly for the minor IMAPP/NASA differences.

Distribution BoM will be running real-time MODIS systems in a few years, and it is best if that is coordinated with the rest of the community. BoM will probably distribute small image products. The BoM experience is that if their real-time system is designed properly, the archived real-time data and metadata automatically satisfies the needs of other users.

In summary, the approach to post-L1B production is:

Action items

A MODIS Coordination Group will be formed to determine the accessibility of DAAC post-L1B code and consider validation. Responsibilities for coordinating product groups were assigned as follows:

Discussion in detail

David Jupp (EOC) chaired the discussion and began by outlining some considerations in assessing the need for local (Australian) production of MODIS products higher than Level 1B (L1B):

There are two software packages to take MODIS Direct Broadcast (DB) data to L1B: NASA/Miami and IMAPP from Wisconsin. L1B products from the DAACs have the format produced by the NASA package. The format of L1B data from IMAPP differs in a very small and not significant way. Local L1B production will be discussed tomorrow.

Arnold Dekker (CLW) suggested starting by examining L2 products to establish whether they are valid for Australia, and asked if there was support for a MODIS validation exercise in Australian coastal waters. Ian Barton agreed with Arnold that the current MODIS atmospheric correction is not accurate enough for ocean colour, but noted that land requirements were less stringent and land users were generally satisfied with the atmospheric correction.

Ian Barton noted NASA has been sending L1B data (essentially all passes for the whole continent) for archive at CMR in Hobart, but earlier passes were processed by early versions of software.

Peter Turner agrees with Arnold that product validation or evaluation is necessary before changing software, and echoes John Le Marshall's sentiment that the right approach is to feed algorithm improvements back to the US software providers.

David Jupp commented there is an obvious scientific need for validation in Australia that the Australian community should take up.

David Griersmith (BoM) addressed the coordination aims listed by Ian Barton. On data acquisition, he noted that some Australian needs will be met by the DAACs, some by the DB reception on the Australian X-band network. David also addressed analysis, validation, distribution and archiving. BoM is currently defining its needs, but will probably want to distribute small (100 Kbyte) images displaying fog, hot-spots, etc. The BoM experience is that if their real-time system is designed properly, the archived real-time data with its associated metadata automatically satisfies the needs of other, nonreal-time users. No matter what else the Australian MODIS community does, BoM will be running real-time systems; clearly it is best if that is coordinated with the rest of the community.

Richard Smith (DOLA) stressed that any local NRT products, possibly modified from standard products, must have a heritage traceable to the DAAC products, to avoid confusion. Thus the DAAC software must be made available. David Griersmith commented that, as a manager, he would be reluctant to use un-validated products. Ian Barton suggested we need to establish a coordinated system to implement local algorithm changes. For instance, improvements to the atmospheric correction for ocean colour should be coordinated between CLW and CMR. Furthermore, NASA should be notified of improvements. Ian Barton quoted John Le Marshall as stating that a mechanism to pass feedback to the Wisconsin group exists, and David Griersmith confirmed the desire of the package originators for feedback.

David Jupp noted that research code might not be suitable for operational production and that needed to be carefully considered by ACRES and BoM. David Griersmith said that BoM is inclined towards IMAPP for L1B production rather than the NASA package, but it seemed poorly suited to real-time operation. The meeting recognised that production of L1B data would be discussed in a session on the following day and ACRES has since noted that IMAPP is currently much faster than the NASA package. This will be something to pursue in a later workshop.

Richard Smith noted that algorithms, such as Alex Held's implementation of MOD14 hot-spot detection in the Sentinel system, may have parameters such as thresholds that may be set locally and should be standardised or recorded. Peter Turner (CAR) suggested that a repository and management system for source code should be established to document changes, and Ian Barton and Richard Smith agreed. Ian Barton suggested the job could be done by ACRES, BoM or CSIRO, or be distributed. Richard Smith commented that with some forty products, the task would necessarily be distributed.

Shanti Reddy (ACRES) would like to see algorithms for 250-m land products implemented locally. Ian Grant (CAR) doubted that all land products existed at 250-m resolution. David Jupp stated it was important to have access to the software that exists.

Craig Smith (ACRES) stated that quality assurance is needed to maintain ISO 9002 qualification, and asked if we should do that on L1B data or higher-level products. Ian Barton answered that the only practical approach to L1B quality was to accept it as NASA's best effort. Craig Smith suggested that there would be more confidence in L1B data that was produced locally than that produced in the US for which the processing history might be unclear on issues such as A-side versus B-side electronics and software versions. Fred Prata suggested that Australia could at least compare locally produced and DAAC-produced L1B data.

Richard Smith stated that Australia needs to be involved with the international validation teams, to extend validation to Australia and Australian conditions.

Fire detection was suggested as a good case history in local higher-level production. Alex Held noted that fire detection products could be well validated since they had many involved users.

David Jupp noted that the current discussion focussed on Levels 2 and higher (L2++), for which validation is the dominant issue. Validation here includes determining how to tweak the algorithm parameters to suit local conditions. The discussion kept falling back to production of L1B, which David suggested was because participants knew what they wanted to do beyond L1B.

Ron Craig described the software DOLA uses beyond L1B: the MS2GT package is used to read L1B data in HDF format, then reproject it, with output in a binary format. ACRES also use MS2GT to progress from L1B.

David Jupp suggested that the issue to be resolved is the coordination of file formats. David Griersmith and Antony Rea (BoM) said that BoM will need L1B data in HDF format from several reception sites. David Griersmith notes that Liam Gumley (Wisconsin) has compared the NASA and IMAPP packages in detail, and concluded that if a 24-hour wait for ancillary geolocation data is acceptable then IMAPP is preferable, but for NRT applications NASA should be used and yields generally acceptable geolocation.

David Jupp summarised the approach to post-L1B production:

Peter Turner commented that raw binary is not acceptable for product interchange. David Griersmith requested that HDF be the Australian standard for L1B data. Ian Barton suggested that for exchange of L1B products between Australian data providers, Australia should standardise on an L1B distribution format that looks like the DAAC product (ignoring the tiny NASA/IMAPP differences). Craig Smith noted that, like DOLA, ACRES generates binary format products beyond L1B that are unsuitable for further processing.

Satellite to L1B to be discussed on the following day.

David Jupp drew a diagram summarising post-L1B issues:

Paul Tildesley has obtained the code for ocean products, and it appears to require considerable work to install.

Richard Smith noted that Liam Gumley will spend five weeks in Perth from 1 November. Ian Barton suggested a follow on MODIS meeting should be held in Perth to coincide with Liam's visit.

Ian Shepherd (ACRES) commented that the biggest task is the coordination of production of Level 2, and that validation is a choice to be made by the coordinating group. David Jupp suggested establishing a MODIS Coordination Group, which would need committed people, and could include people already involved with TERSS and WASTAC. Richard Smith said the Group should be charged with some actions. It was agreed there should be a wider email discussion group.

Peter Turner asked if MISR should be included within the Group's scope [Peter was concerned about managing local modifications to MISR post-L1B processing]. Richard Smith replied that the group's focus should begin with MODIS DB operations, but there was general agreement that it could address all Terra and Aqua instruments.

David Jupp called for volunteers to coordinate product groups:

Ian Shepherd commented that WASTAC is more focused on MODIS than is TERSS.

Copyright CSIRO 2002 ©

Use of this web site and information available from it is subject to our
Legal Notice and Disclaimer