NWPSAF 1D-Var v2.1 ---------------------------------------------------- The full documentation for this code can be found in docs/pdf/nwpsaf-mo-ud-032_NWPSAF_1DVar_Manual.pdf or docs/html/nwpsaf-mo-ud-032_NWPSAF_1DVar_Manual.html. Contents of this file: - Prerequisites - Updates for this release - Installing the 1D-Var - Other considerations for compiling - Considerations specific to the use of the Portland Fortran Compiler - Using an emissivity atlas - A note on the 1D-Var Sample Output - Installing the Observation Simulation Code - Installing the code to convert from BT to PC and/or vice versa - Retrieving surface emissivity Prerequisites ------------- This code requires RTTOV v12 or v13 or v14 to be downloaded and compiled. It is incorporated into the 1D-Var as a library. RTTOV is an NWPSAF deliverable and may be obtained from https://nwp-saf.eumetsat.int/site/ You will also require RTTOV coefficient files(s) for any instrument(s) you wish to simulate. These can also be obtained at the above URL. Furthermore, if you wish to use microwave scattering with RTTOV for microwave cloudy retrievals, you will need to obtain the hydrotable file(s) for the relevant RTTOV version (v14 or v13) appropriate to the instrument(s) you are interested in. As with RTTOV itself, you may require NetCDF/HDF5 libraries to use certain coefficient files or atlases. The 1D-Var does not require anything in addition to what is required for RTTOV itself. It is essential to run 1D-Var with RTTOV v14 using profiles specified on the native vertical grid of the NWP model from which the profiles are derived. Additionally, an appropriate B matrix must be used, derived from the same NWP model. RTTOV v14 cannot generally be executed with input profiles specified on fixed pressure levels, as in the provided 54L examples. These examples are simply there to demonstrate technically how to run 1D-Var with all supported RTTOV versions and test the installation. Updates for this release ------------------------ This release adds support for RTTOV v14. Support for RTTOV v13 and v12 is maintained. Support for RTTOV v11 has been removed. In RTTOV v14 the microwave scattering is integrated into the main RTTOV model. Version 2.0 added support for RTTOV v13 and RTTOV-SCATT (as part of RTTOV v13). The microwave cloud liquid water retrieval functionality has been updated for compatibility with RTTOV-SCATT. Note that, as currently implemented, where RTTOV-SCATT is switched on, it will be used for all instruments used in the retrieval. Therefore, this option is not currently compatible with compound instruments where one or more instruments do not require RTTOV-SCATT (e.g. ATOVS, which includes HIRS). Changes have been made to the handling of gas units which fix inconsistencies present in previous releases: - Units of kg/kg(wet) are now used everywhere internally for q and ozone. - Sample ozone profiles have been updated accordingly. - More accurate unit conversions are now done for water vapour. Control namelist variables "LwpSD" and "WindspeedSD" have been added to enable the user to over-ride the default background error values for liquid water path and wind speed. The docs/nwpsaf-mo-ud-031-NWPSAF_1DVar_ReleaseNote.pdf describes the changes to the 1D-Var package at this and previous releases. Installing the 1D-Var --------------------- 1) Compile the code - Go to the makefiles directory - Edit the top section of the makefile to point to your installation of RTTOV, and change to your compiler of choice. Specify the location of your HDF5 installation if required. - type make - if you need to run a clean build, you can type "make clean" first - compiled object files and the executable NWPSAF_1DVar should appear in the build directory - with many compilers you will get an unimportant warning message about "bad preprocessor" directive: #include "throw.h" 2) Set up a directory from which to run the 1D-Var code: - cd into WorkDir (or make a copy of this directory if you prefer - the name of this directory is not important). - Edit main_paths.sh script with the location of HDF5 and NetCDF libraries, and the path to the test directory of your 1D-Var installation. - Edit the run scripts to point to the correct rtcoef files for each "1DVar" script. - Make any other changes you want to make to the test script (such as to $OUTPUT_DIR). - Note that with ifort, you will need to keep the command 'ulimit -s unlimited' in the main_paths.sh to avoid a memory fault. 3) Run test scripts to check the code works - Each Run_1DVar* script can be run independently or the wrapper script Run_1DVar_tests can be used to run all the tests. - Most scripts are set up to run 54L profiles. If you are using 54L RTTOV coefficients (as used to generate the sample output), then profile interpolation will be disabled. In this case, you can change line 33 to LEVELS=43 if you want to check the use of profile interpolation as well. - The PC tests are currently only for RTTOV v13 and v12, as the appropriate files for RTTOV v14 were not available at the time of this 1DVar release. - See NWPSAF_vn1.1_Release_Info.txt for more information 4) Optionally: compare your results with the sample results provided - You can run a differencing tool to compare your results with those in directories Sample_Output_RTTOV12, Sample_Output_RTTOV13 and Sample_Output_RTTOV14. - The RTTOV coefficient files used to produce the sample output are specified in readme.txt files in each Sample_Output_RTTOVxx directory. - See the Release Note for more information 5) Optionally: a test script and sample input and output files are provided for PC score runs as well (with sample output in the directories above) and example files for running in Radiance units rather than brightness temperatures. Other considerations for compiling ---------------------------------- If you change the code yourself, in particular the interfaces between subroutines, you may find you need to run 'make clean' before recompiling. This is not new behaviour - the reason is that the makefile is basic, and includes only the bare minimum of dependencies to allow the code to compile. Using an emissivity atlas: -------------------------- - An NetCDF(RTTOV v14)/HDF5(RTTOV v13) library must be linked in the makefile to enable the atlas to be read in. You will need to modify the top of the makefile to specify the path to your NetCDF/HDF5 installation in NetCDF/HDF5_DIR. The linking to relevant NetCDF/HDF5-related libraries should then be taken care of automatically. - When reading emissivities from an atlas an extra line needs to be added to each observation's entry in the observation file. This line goes between the line beginning "Obs ID:" and the one beginning "Latitude:". The information required on this line is the date of the observation in the form Year: ???? Month: ?? Day: ?? The code is capable of reading observation files with or without this extra line but the month is definitely required when using an emissivity atlas. This line has been added to some of the sample observation files. A note on 1D-Var Sample Output ------------------------------ - The output is marginally sensitive to compiler, but very sensitive to the choice of RTTOV coefficient files. For this reason, the test script does not do a diff automatically. - Sample output is provided for gfortran 11.5.0, ifort 19.0.0, nagfor 7.0 and pgf90 18.7.0, for RTTOV v12, RTTOV v13 and RTTOV v14. - Note that the background profiles have changed since previous versions so the output will not be the same. These changes to the background profiles have been made so all supported RTTOV versions can use the same sample files. Some test observations were simulated with an older version of RTTOV, and are becoming increasinly incompatible with the latest science. This means that with RTTOV12, with the interpolation option interp_mode=5, the Sample Output for CrIS shows non-convergence of the 1D-Var. This behaviour in the sample output is normal, but the user should be cautious when using simulated observations with the 1D-Var that the simulations are done with settings that are consistent with the RT simulation in the 1D-Var itself (or that the differences are accounted for by increasing the observation error term). This is not new behaviour - it is a result of improvements in radiative transfer science. The observation simulation code ------------------------------------------ Note: simspec is currently only supported with RTTOV12 and RTTOV13. The routines in src/sim_spec were used to generate the sample observation files used in the test datasets. They are provided as an example to the user and may be adapted for your own use. However, they do not form part of the core NWPSAF 1D-Var package and are therefore unsupported. It is recommended that users who wish to generate synthetic radiances should use the NWPSAF Radiance Simulator (available from https://nwp-saf.eumetsat.int/site/software/radiance-simulator/). The following is a brief guide to adapting the sim_spec code to your own needs: 1) Set up the code to meet your needs - Edit the code src/sim_spec/sim_spec.f90 to include paths to coefficient files, profile files, error covariance files and output files. - Modify the options in the files to match your requirements (e.g. simulate radiances, add errors etc) - you may also want to modify the RTTOV options (though take care that your simluation settings match those that you use in the 1D-Var - see the note above on Sample Output) - Note that these routines were written to simulate IASI observations, so you may need to change some array sizes etc depending on what you are simulating 2) Compile - Go to build directory - type "make sim_spec" - if you need to run a clean build, you can type "make clean_simspec" first 3) Run test scripts - in WorkDir, ensure the main_paths.sh has your paths. - run Run_SimSpec.ksh 4) Example IASI output is provided in Sample_Obs/SimSpec_RTTOV11 and Sample_Obs/SimSpec_RTTOV12 (the PC output is used in the PC test script mentioned above) -------------------------------------------------------------------- This code is independent of RTTOV, though still requires RTTOV pccoef files to be read in. The format of these files has not changed between RTTOV-11 and RTTOV-13 so only one routine is provided. 1) Set up the code to meet your needs - Edit the code src/sim_spec/spec_to_pc.f90 or pc_to_spec.f90 to include paths to coefficient files, input observation files and output files. - Modify the options in the file pc_to_spec to match your requirements: you can choose whether to convert to radiance or brightness temperatures. You may also want to modify the PC-RTTOV options. - spec_to_pc reads the observation file header to determine the units of the input file and converts BTs to radiances first if necessary. - Note that these routines were written to convert IASI observations, so you may need to change some array sizes etc depending on what you are converting 2) Compile - Go to build directory - type "make spec_to_pc" or "make pc_to_spec" - if you need to run a clean build, you can type "make clean_simspec" first 3) Run test scripts - in WorkDir, ensure the main_paths.sh has your paths. - run Run_Convert.ksh - this converts from BT/Rad to PC and back again (depending on input files and settings). 4) Example IASI output is provided in Sample_Obs/SimSpec_RTTOV11 and Sample_Obs/SimSpec_RTTOV12 Retrieving surface emissivity ----------------------------- Version 1.2 of the NWPSAF 1D-Var package adds the new capability to retrieve surface emissivity for the first time. The surface emissivity is represented by principal components in the retrieval state vector. This allows a compact representation of the surface emissivity and keeps the number of additional state vector elements low. The capability is applied to IASI, AIRS and CrIS for limited subsets of channels (up to 314 channels for IASI, 324 for AIRS and 399 for CrIS). The *_EmisEigenVec file for each instrument (see below) contains the channel numbers of the selected subset. For the surface emissivity retrieval, two additional auxiliary files are required which can be found in the Sample_SurfEmiss/ directory One of these files, "EmisEigenVec" contains the principal components and the other an atlas specifying the weights (or scores) of these "EmisPCAtlas". The latter are a function of latitude, and longitude, which are specified by the user in the "ObsFile.dat". All the user has to do to trigger a surface emissivity retrieval is to add a line in "Retrievals.NL". An example script to demonstrate the surface emissivity retrieval capability has been constructed: WorkDir/Run_1DVar_SurfEmiss.ksh. The script tests the code for AIRS, IASI and CrIS. For AIRS and CrIS, the input observations are the same as those used in the normal Run_1DVar_test.ksh script (apart from a change in the specified latitude/longitude, which is used to force the use of a desert-type background emissivity for testing purposes). For IASI, a simulated IASI observation using a known land surface emissivity is contained in Sample_ObsFiles/ObsFile_IASI_SURFEMISS.dat; this is the file that was used in the initial scientific evaluation of the emissivity retrieval scheme before the realease of v1.2 of the code. The quantities in the retrieval state vector to be retrieved (which include the surface emissivity principal components) are controlled by Sample_Namelists/Retrieval_IASI_SURFEMISS_54L.NL etc. The files specific to the test case in the IASI_COEFFS_DIR directory are ChannelChoice_surf_emiss.dat and Rmatrix_surf_emiss which are specifically designed for the 314 IASI channels involved. See also docs/nwpsaf-mo-ud-031_release_note.pdf for the test case. --------------------------- Met Office 18th March 2025