|
News and Technical Information Archive |
15 February 2000
CAPS version 2.0.3 has been released. Users are warned to treat the NOAA 14 calibration information with increasing caution. The last real measured values for the coefficients are over a year old and the current results are derived by extrapolation. We are attempting to gain access to updated measurements and will make these available as soon as possible. The Windows PC version of CAPS is expected to be released within the next two weeks. This version will run under Windows 95/98 and Windows NT.
3 December 1999
CAPS version 2.0.3 will replace CAPS 2.0.2 shortly. CAPS 2.0.3 includes some minor enhancements to version 2.0.2 and will be the first release of a Windows version of CAPS. Version 2.0.3 also corrects the NOAA 14 reflectance calculation error (see bugs). We have implemented the CLAVR cloud detection algorithm in a script. This is available on request now and we hope to include it in the contribution area for 2.0.3. CAPS uses ISO standard 4 digit date fields and modified Julian days for time difference calculations. We do not anticipate any Y2K problems. However, users should be aware that updates for the AVHRR solar reflectance channel calibration have not been made since January 1999. The calibration extrapolation will work beyond the "use by" date, however the accuracy of the results becomes increasingly suspect. We are trying to obtain calibration updates for NOAA 14 and 15.
22 November 1999
CAPS version 2.0.1 bug fix to be released later today! Do not use version 2.0!
Errors Fixed
1. An optimisation associated with indexing coordinate variables introduced a rounding error. This error is particularly nasty and has wide effects, it is only present in CAPS 2.0, (1.x OK) and has been corrected in 2.0.1.
2. Calls to Tcl proceedures from within NAP expressions failed to load the requested proceedure. This error is only preset in CAPS 2.0 and has been corrected in 2.0.1.
3. A long standing error relating to the nao sequence number has been fixed. The
symptom of this error was that when the nao sequence number reached 1 million, nao's with
sequence numbers at 1 million or higher were not destroyed. In intensive processing
operations memory would be exhausted and processing failed. This error has been corrected.
There is still an absolute limit on the nao sequence number of 2 to the power 31. We
estimate that this number could be reached in around 50 days of continuous intensive
processing.. We recommend restarting CAPS reasonably frequently to avoid such problems.
(PS The nao id is "nao<sequence number>-<slot number>")
4. Array indexing failed with ragged arrays. (fixed).
We recommend not using version 2.0.
11 November 1999
CAPS version 2.0 is now available in the download area. There are versions for Sun, SGI and Linux available. A new contributions area has been created with specialised scripts and extra libraries. At present we have included a script version of the CLAVR cloud detection algorithm and an associated world digital terrain map access function and data. CAPS 2.0 uses dynamic loading and is supplied with Tcl/Tk 8.2.2. The 8.2.2 Tk code has been modified to support printing of Tk photo image types. The new release must have Tcl/Tk 8.2+ to work. Caps is now started using wish or tclsh. Please see the README associated with the download to install the new release.
The version includes corrections to the caps get_HRPT command some new spatial operations for nap. A number of small errors have been corrected. The big changes relate to the structure of the code to be consistent with Tcl/Tk 8.2 plus the Linux version. The Windows PC version has slipped in priority, although in principle should be easy to produce..
Contributions - we are now encouraging users to make script contributions to the distribution. If you have developed any useful scripts please email us and we can include what you have done in the new contrib area.
17 September 1999
ERROR - NOAA 12 - Brightness Temperatures 4 and 5. An error in AVHRR_12.tir space radiance values has been found. This file should be replaced with the corrected version from the download area. NOAA 12 brightness temperatures are in error by several degrees. The NOAA 12 radiance values for channels 4 and 5 will also be incorrect. This error is specific to NOAA 12 4 and 5 brightness temperatures and radiances.
24 August 1999
Development of a Linux and PC version of CAPS is continuing. The w versions of CAPS will be dynamically loaded with Tcl/Tk. There is quite a bit of reorganisation of the code required to perform this step. An error has been found in the avhrr2hdf processing script. The mercator projection option should not be used. The symptom is a core dump from caps. The script incorrectly codes the mercator projection and gives the nap ap function an infinity. Nap ap does not handle the infinity and causes the core dump. The script is being corrected. Please contact us if you need the mercator projection. Some updates have been made to the documentation - there is still quite a bit that needs work! New functions coming for NAP are morphological binary erosion and dilation and a moving window range function. A generalised correlation matching function will also be included in NAP. The CLAVR algorithm for AVHRR cloud detection is also likely to be available. Some simple histogram and graph plotting scripts have been developed and there is a script version fo the roujean BRDF model in the pipeline.
18 June 1999 Version 1.2 Release!
We are still working on improved documentation. The new release has already had substantial testing of its basic operations and should be a large improvement on the previous version. There is a new menu script to display full pass AVHRR data and an improved HDF browser interface. HDF I/O now fully supports scaling processed data from 32 bit float to 16 bit integer in the HDF file. Meta-data are now saved automatically in the output HDF file if the standard processing scripts are used. A large number of code corrections and efficiency gains have been made. The calibration system for the AVHRR solar channels has been significantly revised. Many parameter names started with "_", the leading "_" has been dropped in most cases. For example _avhrr_1 is now just avhrr_1. The quantity reflectance has been renamed reflectanceFactor to more correctly identify that it has not been normalised by the cosine of the solar zenith angle. These changes have been accommodated in changes to the CAPS library scripts.
The new release includes some major changes to the warp process, and a whole series of corrections and refinements to the bulk of the CAPS code. Please email or phone us if you have any problems with the new release or want to offer suggestions for further improvements/changes.
22 January 1999Version 1.2 Soon to be released
This release is significant in that there are big changes to warp. Warp has been recoded to improve speed and dramatically reduce memory usage. The new version of warp produces output in a form which is not the same as the old version. Warp now produces output in the form of a ragged array. This is a new nao type that is able to store data where some or all rows (lines) are incomplete. The HDF writing routines have been modified to deal with the new nao type and write the data to file. The display proceedures are also able to deal with the new data type. A "pad" function is available to make ragged arrays into normal arrays. There will also be a mechanisim available to collect output from the new Warp command to make complete chunks if further processing is required. These changes should have little effect on the user interface except that it will be nolonger necessary to call warp twice when trying to remap an image outside the process loop. The old warp command will be renamed warp98 to indicate that it is considered fairly useless and should not be used!
The HDF output system has been significantly modified to accommodate the ragged array concept and in the process a number of problems have been fixed and some enhancements have been made. Changes to the documentation will be made over the next few weeks to reflect the operation of version 1.2 of the code.
A number of minor errors have been corrected. The NAO naming convention has been changed from nao#<id number> to nao.<id number> to avoid confusion with the "#" operator used by nap.
22 November Version 1.1.4 Released
A full list of changes will be made available shortly. Scripts have been provided to simplify the production of standard AVHRR products and scripts like view_nao, lvtk and hdf are now auto-loaded. A range of minor problems have been fixed.
1. The warp program still requires two calls when used outside the process command to work properly. The warp program is being recoded to reduce unnecessary memory usage, improve performance and work properly outside the process command loop.
2. The get_cal command does not like carriage return characters in AVHRR_<n>.sol files. The initial versions of the calibration files were generated on a Unix system, and subsequently these files have been generated on a PC. The PC version contains <CR> which the CAPS software does not like. The work-around is to remove the <CR> characters. The problem has finally been corrected in the working version of CAPS. This will appear in the next release of code. The visible calibration coefficients are also out-of-date, this situation is being rectified.
3. There are still problems with the "Clift" navigation. It appears that the CAPS code is correct and there is a problem with the state vector information from Hobart. As a result the navigation will give results approximately 2 lines in error and the error may vary according to the pass direction. We are still in the process of diagnosing the error in the state vector generation code in Hobart which is not part of CAPS. We recommend using the Brouwer Lyddane navigation until the problem has been sorted out.
16 October 1998 Version 1.1.3 Released
Version 1.1.3 is a bug fix release. Changes and Bug fixes
1. The satellite start time retrieval has been corrected to generate the correct start time for the first line of data. However, although this has made a substantial improvement to the Clift navigation system there still appear to be some residual problems with the accuracy of the Clift navigation.
2.get_HRPT will now return the satelliteStartTime withou the need to supply coordinate variables.
3. A number of cases where intermediate nao results were not being deleted have been fixed.
4. get_HRPT core dump problem fixed.
Residual bugs
1. Warp algorithm still takes too much memory and when used outside the process loop requires a second call with null valued CV's to force the output of the results.
2. There are still some cases where nao's are not being deleted.
3. Although improved there are still problems with the navigation system.
New Bugs
1. view_nao does not release the Tk image that is created for screen display. This results in a loss of available memory. The proble has been corrected and a new version of view_nao is available on the download site.
2. The corrected "UTC" time generated from the satelliteStartTime and the Hobart clock correction file does not give accurate UTC time. The "UTC" time is only within 300 msecs.
9 October 1998 Version 1.1.2 Still More bugs
Clift Model State Vector attitude and time corrections
NOAA 15 orbit information is now continuously updated
There are some small inaccuracies in some of the navigation data files presently available. These files will be upgraded from time to time to reflect improved results. If you wish to be notified when this happens please contact Chris Rathbone at Chris.Rathbone@marine.csiro.au to be placed on a list for email notification of such changes.
Clift navigation code bug.
NOAA 12 and NOAA 15 orbit close to the terminator. The Clift navigation code generates an error with the message: eclips: cta = 3 > 1.0 This message was meant as a warning that the solar radiation pressure calculation had failed. This only causes a very small error in the calculated satellite position and would normally be ignored. Unfortunately this condition causes geolocate_clift to quit. The problem will be fixed in version 1.1.3. The work around is to use Brouwer-Lyddane navigation for those cases where this message is seen.
7 October 1998
Caps Version 1.1.2 More Bugs
Navigation Error
During systematic testing of the Clift and Brouwer-Lyddane navigation an error in the satelliteStartTime has been found. Errors of 5 or more lines (typically 6) in latitude are likely as a result. The satelliteStartTime is derived from the time code words in the HRPT data. The HRPT data often has bit errors at the start of a pass and to avoid these errors the code in get_HRPT attempts to find 5 lines (frames) with a consistent time increment of 0.16 to 0.17 seconds. Having found such a sequence the time of start line is determined by by subtracting 0.167 times the number of line from the start where a reliable time has been identified. The present code does not subtract the time offset and returns the time associated with the fifth line in the stable sequence. Navigation results can be improved by subtracting 1.15741e-05 days (1 second, 6 lines because the time for the first line is often corrupt) from the returned satelliteStartTime.
get_HRPT satelliteStartTime
nap _satelliteStartTime = _satelliteStartTime - 1.15741e05
(assuming set capsValuePrefix "")
This should improve the navigation result, however, if the data is noisy in the first few lines the start time returned could be for a line more than 6 lines into the data and hence the correction will only partially compensate for the error in the algorithm. The error will be fixed in release 1.1.3 of CAPS which is due very soon!
Warp Memory Usage
Warp uses too much memory when "chunking" and does not produce a result when used outside the process loop. The warp algorithm requires some adjustment. Warp uses nearly twice the amount of memory it really needs to do the job because it is not producing a result when it should. This is an involved problem and may not be fixed for release 1.1.3. The work around is to limit the number of channels being remapped in any loop and repeat the entire process as required. Alternatively, the warp can be done in two stages taking and east and west portion of the result and separately writing them to the output HDF file. This approach will also reduce memory requirements by a factor of approximately two.
23 September 1998
Caps Version 1.1.2 Know Bug list
We hope to release a bug fix Version 1.1.3 in the next week.
11 September
New Clift navigation files have been made available for NOAA 14 and a limited number of orbits for NOAA 15. The Marine Labs are having some difficulties providing navigation information for NOAA 15. This problem should be rectified reasonably soon. Users should be cautious with the navigation results particularly when the space craft attitude information is included. It is possible that there an error in sign of one or more of the attitude parameters. This is being investigated.
2 September 1998
Version 1.1.2 of CAPS has been released for Solaris and SGI. There has been a rationalisation of directory structure and the need to download Tcl/Tk and HDF has been removed. The README in the download area explains how to install CAPS. A few minor bugs have been fixed including some that were related to Tcl/Tk. Patch level 2 for Tcl/Tk 8.0 has made some GUI related problems on Solaris go away. CAPS now also includes the ability to read data from ATSR_2 GBT files. There is no documentation yet, however examining the CAPS DataDictionary will show the input and output parameters used by the atsr command.
26 June 1998
The proposed May release has been held back and a full phase 1.1 release containing remapping and support for NOAA 15 will be available shortly. Watch this space!
25 May 1998
Mapping from the satellite projection to geographic coordinates is being developed. Part of the algorithm is included in this release of CAPS. However, the remapping functionaltiy will not be available in this release.
25 March 1998
CAPS version 1.0.4 has been released. This version is the current bug fix for version 1.0. Some minor enhancements have been made to the object oriented commands particularly in terms of the presentation of array data values.
17 February 1998
Some difficulties have been encountered with the PVL library that parses the ASDA header. The original PVL library, written by the Australian Bureau of Meteorology, had difficulty parsing a statement peculiar to the Western Australian reception facility. A new library has been released by the Bureau but has introduced some new errors into the PVL parser. Most of the errors have been fixed. However, we are now encountering what appears to be a memory corruption problem that appears to be related to the new PVL library, but could also relate to recent changes to CAPS itself. Resolving these problems means that we have lost atleast a week in our schedule. Navigation now appears more likely to be complete in mid March at the earliest.
28 January 1998
Work is progressing on the more accurate Hobart navigation system. A new release of CAPS including the new navigation function is expected to be available in early to mid March 1998.
23 December 1997
The alpha release of CAPS was installed at the Bureau of Meteorology in Melbourne on a Hewlett Packard machine running HPUX. A few minor problems were encountered during the installation, however, these were overcome and all appears to be working as planned. A sun Solaris 2.5 release of CAPS can be obtained by emailing Harvey.Davies@dar.csiro.au.
11 December 1997
An alpha release of CAPS is scheduled for 18 December 1997. This release will be able to read Hobart and ASDA format HRPT data. Full calibration facilities are available and navigation based on the T-BUS orbital elements and Brouer Lyddane predictor is included. The Hobart orbit predictor and geographic remapping will be included in the final release for CAPS phase 1. This release, is now scheduled for the end of January 1998.
Additions are being made in the documentation area. Unfortunately Microsoft Word 97 and Frontpage 97 are doing a poor job converting word documents to html. The latest update to Word 97 comes close to formatting the html correctly, however, Frontpage 97 then messes up the section numbering. We hope to move to Frontpage 98 shortly in the hope that this product might solve some of the problems getting documentation onto the Web. We apologise for the crazy section numbering and indentation problems.
21 November 1997
The release of a working version of CAPS is still being delayed by completion of the generic HRPT reading process. However, navigation capabilities are being introduced into the current version of CAPS. It is likely that the first software release will include limited navigation capabilities.
15 October 1997
A demonstration of CAPS was given at CSIRO Marine Research on Tuesday 7 October. The AVHRR data in several Hobart format HRPT files was processed to give radiances, reflectances and brightness temperatures. These quantities were written into an HDF file. It took approximately 3 minutes to process 3000 lines by 2048 pixels by 5 channels. In a final demonstration a simple formula for NDVI was introduced into the processing system. The current version of CAPS does not support image navigation and can only read Hobart format HRPT file. We are currently working to make the HRPT input more generic to include ASDA format and some common Australian HRPT formats. Once this work is complete (2 to 3 weeks) a test version of the software will be released.
23 June 1997
We are recommending Practical Programming in Tcl & Tk (second Edition) by Brent B. Welch as the most appropriate text for those who are interested in Tcl/Tk (ISBN 0-13-616830-2). The book covers version 8.0 of Tcl and Tk and has just been released. It also comes with Unix and PC versions of Tcl and Tk on CD-ROM. The book retails for $86 but can be obtained for around $69 from some suppliers.
.
20 June 1997
Current expectations are that the numeric array processor which supports CAPS will be functionally complete at the end of June. The next major task is the coding of the CAPS command and parameter interface. Work is also being carried out on HRPT I/O, navigation and calibration.
Technical Information
This section will be updated as current design decisions are made. Please feel free to email the CAPS team members if you have comments.
Index
Script User Interface (SUI) 20 June 1997
Data Dictionary Fields 20 June 1997
Chunking and Buffering 20 June 1997
T001 20 June 1997 Script User Interface (SUI)
The original specification for the user interface was developed bottom up. The starting point was the assumption that every parameter associated with a CAPS command would have a name. The parameter name would then be assigned a value.
caps solcal {
in_avhrr_1=_n14_12345_avhrr_1
in_responsivity_avhrr_1=_n14_12345_responsivity_avhrr_1
in_spaceCount_avhrr_1=_n14_12345_spaceCount_avhrr_1
out_radiance_meanSpectral_avhrr_1=_n14_12345_radiance_meanSpectral_avhrr_1
}
In this example there is clearly redundancy in the names used and a lot of information required. To reduce complexity we decided to define a dependency table which would allow the user to specify what they wanted and not need to assign all the inputs required. This when coupled with a Data Dictionary allowed us to reduce the above command to the form:
caps solcal {
in_.*=_n14_12345_
out_radiance_meanSpectral_avhrr_1=_n14_12345_
}
the syntax can be described as
caps <command name>{
in_.*=<the name space where the inputs are coming from>
<a Tcl Regular Expression (RE) for the outputs required>=<name space where the outputs are going>
}
A user could use the above template with information about parameter names to generate CAPS commands. However, the CAPS team has recently revisited the SUI design and asked the question "what is the simplest way we can possibly write a CAPS command?" The answer is:
caps solcal out_radiance_meanSpectral_avhrr_1
the syntax being
caps <command name> <RE for output parameter names> <RE for output parameter names> ...
and we specify the name path for input and output in Tcl variables in the command environment. This form has been adopted as part of the SUI design. We still allow the original assignment syntax which can be mixed with the simplified syntax. This gives the user a simple entry to the SUI but still provides full flexibility in dealing with more complex situations.
T002 20 June 1997 Data Dictionary Fields
Data Dictionary fields are separated by ";". The comment character is a "#" and obeys the same rules as the Tcl "#" comment. The table below shows the current data dictionary fields.
| In/Out | name | name | data | attributes | units | default | range | range | ||||||||||
| prefix | suffix | type | value | minimum | maximum |
The description of the parameter is inserted using the "#" and can either follow the last field or be added underneath the dictionary entry.
T003 20 June 1997 Chunking and Buffering
The CAPS system processes large data files by taking pieces or chunks of the file and processing that chunk in memory through several CAPS commands to processing completion. Commands which operate on data in a pointwise manner cause no difficulty with this processing method. However, processes that us spatial information require a region of overlap between successive chunks. Managing this overlap requirement can be done in two ways:
1. Find the command requiring the largest overlap in the process stream and centrally manage the overlap for the whole process.
2. Place the responsibility for managing data overlap requirements with commands that need data overlap.
Option 2 has been chosen for CAPS. The advantage of selecting option 2 is that no overlap region is required for most commands and this saves processing the same data twice. Option 2 is achieved by supporting static buffers for commands that process spatial information. In addition all CAPS commands must be able to accept zero data chunks to manage the situation where a command has to buffer more input data before it can write to an output chunk in the processing loop.
 |
© Copyright, CSIRO Australia 2001 |