NWPSAF 1DVar vn1.1.1 Release Information -------------------------------------- Note that a bug was found in the NWPSAF 1D-Var package (version 1.1) which has been fixed in this release (version 1.1.1). It affected microwave retrievals that included the liquid water path. a bug was found in the NWPSAF 1D-Var package (version 1.1) which affected microwave retrievals that included the liquid water path. In the 1DVar source code file NWPSAFMod_LiquidWater.f90 the order of the arguments to the function Layers_to_LWP was incorrect. Also, the sample test data for SSMIS_43L ( for RTTOV11 / RTTOV12 and ifort / gfortran ) were affected by the bug. --------------------------- Stephan Havemann Met Office 22nd June 2018 In the following a description is given about what has changed between v1.0 and the major release v1.1. If you were used to the old makefile, for example, please read this file in full to find out how things have changed. Contents: --------- - 1DVar/build/makefile - Compiler warnings - New features of 1D-Var v 1.1 - More info on the test script - More info on Sample Output - Compilers tested 1DVar/build/makefile: --------------------- The directory structure has been tidied up for this release: the fortran files are in the 'src' directory, but the interface files are now in 'include'. The makefile is now in 'build', and the compiled executable, along with object, mod, and any other byproducts of compilation (e.g. *_genmod.f90 files), will also appear in the 'build' directory. The makefile has been significantly improved from the previous release. You no longer have to comment out and uncomment sections of the makefile, or use "dummy" routines to control which version of the RT model you are using (please also read the section on Compiler Warnings below). All you should need to do is edit the top section of the file to set the relevant options for how you wish to use the 1D-Var at the top, and the makefile should automatically link in the relevant libraries. The comments in the makefile should make it clear what you need for each option, but you will certainly need to change a few: COMPILER CHOICE: The makefile is set up for the Intel Fortran Compiler (ifort) by default. The Top section of the makefile can be used to select a different compiler by setting the FC variable. Compilers that are provided for are ifort, nagfor, pgf90, gfortran and xlf95_r. xlf95_r has not been tested for this release but is provided as a convenient starting point. The compiling options in the make file are for Linux machines. Replace these with the relevant options for your own compiler if necessary, noting that you may need to adjust CPPFLAGS and LFLAGS as well as FFLAGS. For ifort and gfortran, two sets of FFLAGS are provided: normal compiling and debug compiling. The normal compiling flags are used by default, but if you encounter any problems, you could try switching to the debug FFLAGS instead (you may want to compile RTTOV with debug options as well before recompiling 1D-Var). Feel free to edit the FFLAGS to add whatever compiling options you want for your compiler. Note that the FFLAGS are set below the line that says "No Editing below this line should be necessary"! RT MODEL LOCATION: Edit the variables in the next section to point to your installation of RTTOV. OPTIONS FOR COMPILATION: This is the area of the makefile where you specify the location of HDF/NetCDF libraries, and whether you want to run with the emissivity atlas or not. Compiler Warnings: ------------------ As part of the new massively simplified make process, preprocessor directives have been added to the code. This allows you to only compile and execute certain parts of the code if the relevant flag has been set in the makefile. Preprocessor directives have been added to control the choice of RT model, and to inform the compiler whether to execute code relating to the use of the emissivity atlas. If you have compiled with EmissAtlas=0, you will get a warning about an unused variable in NWPSAF_RTTOVxx.Interface.f90 TYPE(rttov_emis_atlas_data) :: emis_atlas This warning is not an error, but a consequence of the new lower maintenance software structure and the massively simplified makefile. New Features of 1DVar v1.1: --------------------------- The main new features of this release are: ENHANCED RT: - Code upgraded to support RTTOV-11.3 (note, will no longer run with RTTOV11.1 as there are a few differences in the interface to RTTOV between the two versions). - Support for RTTOV-12, enabling the most up-to-date RT capabilities - Ability to use PC-RTTOV to simulate PC scores from hyperspectral sounders, for use with PC score observations. This mode is for experimentation and has not been extensively tested. - Support for calculation and use in the 1D-Var of radiance observations as well as brightness temperatures and PC scores. ENHANCED OUTPUT DIAGNOSTICS: - A new file, Profile.QC, is output, which gives a single number per profile. 0 = successful minimisation 1 = 1D-Var did not reach convergence. 2 = observation not processed (bad BT, failure in RTTOV etc) - A new namelist option, EnhancedDiagnostics (default=.false.) has been added, to be used in conjunction with higher verbosity levels. Production of the AMatrix and AmMatrix files is now controlled by this option. In addition, a number of other files are produced when EnhancedDiagnostics=.true., containing Averaging Kernel matrices output for the retrieved profile, and Jacobians output for the background profile and the retrieved profile. INPUT PROFILE FLEXIBILITY: - The code can now read a profile file on levels expressed either in Pa or hPa, and they can run either top-of-atmosphere to surface, or surface to top-of-atmosphere. CODE TIDIED: - The code has been tidied considerably, and following user consultation, all out-of-date RT models have been removed. - The code under NWPSAF_Fastmodel_Interface.f90 has also be greatly modified to make it easier to read and maintain. The results should be completely unaffected by these changes (the restructure was tested before upgrading the RT model, and produced identical results for the standard tests), so please report any problems that you see. (Note, however that the new code does not run with RTTOV11.1 and that the results are dependent on the coefficient files, which are not backwards compatible for RTTOV12. Also, various settings such as gas_units and interpolation_mode have had a change in the default behaviour which affects the RT calculation. See below.) - Some routines have been renamed, and some utility routines for liquid water path retrieval and svp have been converted to functions that now reside in a dedicated module. - Some switches that were INTEGER variables have been converted to LOGICAL for better clarity. - The code will now fail if an input file is not there, rather than just reporting a warning and hanging. DIRECTORY STRUCTURE TIDIED: - A new 'include' directory has been introduced. The *.interface files now reside here. - A new 'build' directory has been introduced. The makefile now resides here, and compilation should be done from within that directory. All compiled object files etc will be placed in this directory. - The 'src' directory now contains only code. - The 'Sample_Scripts' directory has been replaced by the directory 'WorkDir'. NEW CALLING SCRIPT: - 'WorkDir' contains a script, Run_1DVar_test.ksh that will run through the installation test configurations. All required files will be linked into WorkDir. This avoids cluttering the top directory with files. - Output from the installation tests will be put into a directory called 'testoutput', and you can compare this output with that in the 'Sample_Output_RTTOV11' or 'Sample_Output_RTTOV12' directory to test your installation. Information about the coefficient files used to produce this output is included below. - There is also a very similar script for one ob type, Run_1DVar.ksh that is easy for users to modify for their own application. WorkDir can be copied multiple times; just set different values for outputdir and myoutputdir in each copy, and the user can then run multiple instances of the 1D-Var at once. -Two directory locations, which in previous releases were set in the ControlData.NL file, are now set via environment variables. These are: * COEFFS_DIR * OUTPUT_DIR These variables should be set and exported before calling the 1D-Var. CHANGES TO OUTPUT: - The RTTOV11 interface code now sets profiles(1) % gas_units = 2 by default If you wish to replicate results with the previous version of the NWPSAF_1DVar, set gas_units=0 in the Control.NL file. Please see RTTOV documentation for more information. - Most of the Sample_output files were run with a mode Legacy_Settings in the Control.NL file. There are two settings (apart from gas_units) that massively alter the output of RTTOV: use_q2m and interpolation_mode=5. For compatibility with the older test files, Legacy_settings was set up to set these to the older settings. By default, the code will run with the newer recommended settings and may thus produce different results from the previous release even with the same RTTOV coefficient files.. ADDITIONAL UTILITIES: - See Appendix E of the documentation for more information. - Code provided to simulate observation files, in either radiance, brightness temperature, or PC Score units, for IASI. This code can be adapted for other instruments. Simulated noise is added to the observations based on provision of a diagonal R-matrix file in the standard format required for 1D-Var. - Code provided to convert observation input files from brightness temperature or radiance to principal component scores, and back again. - The new utilities are also compiled using the same makefile as the 1D-Var. - Calling scripts have been added to WorkDir to run these utilities. More info on the test script ---------------------------- The test script should be run in the WorkDir directory, or a copy of it, rather than in the top installation directory as previously. You will need to link the necessary RTTOV coefficient files into this directory. There are no longer any coefficient files on 43 levels but the code will still run a retrieval on 43 levels, with coefficients on any number of levels, as the RTTOV internal interpolation will be called automatically. Look through the script and read the readme.txt file for more info on what changes you need to make to the script before you run it. Run_1DVar_test.ksh will copy sample input files in to the top 1DVar directory and copies one of the two B matrices in to the relevant coeffs directory. The sample script is set up to run using 54 levels by default for everything except SSMIS (there is only a 43 level test profile containing cloud liquid water provided). If you want to run the test retrievals on 43 levels, change the line where the number of levels is specified (line 36). More information on Sample Output --------------------------------- Sample_Output_RTTOV* directory. Sample output is provided for both RTTOV11 and RTTOV12. The sample output was generated using code compiled with - ifort-12.0.4 - gfortran-4.7.7 on a DELL T3500 running RHEL-6.8. The following RTTOV-11 coefficients were used to generate the sample output for retrievals on 54 and 43 levels (except SSMIS, run on 43L only) using RTTOV11.3. HDF coefficient files for all instruments where available. - IASI: rtcoef_metop_2_iasi.H5: RTTOV-9 Predictors / LBLRTM / 101L - AIRS: rtcoef_eos_2_airs.dat: RTTOV-7 Predictors / LBLRTM / 54L - ATOVS: rtcoef_noaa_15_hirs.dat: RTTOV-7 Predictors / LBLRTM / 54L rtcoef_noaa_15_amsua.dat: RTTOV-7 Predictors / LIEBE-MPM / 54L rtcoef_noaa_15_amsub.dat: RTTOV-7 Predictors / LIEBE-MPM / 54L - ATMS: rtcoef_jpss_0_atms.dat: RTTOV-7 Predictors / LIEBE-MPM / 54L - CrIS: rtcoef_jpss_0_cris.dat: RTTOV-7 Predictors / LBLRTM / 54L - SSMIS: rtcoef_dmsp_16_ssmis.dat: RTTOV-7 Predictors / LIEBE-MPM / 54L This output is in Sample_Output_RTTOV11. The following RTTOV-12 coefficients were used to generate the sample output for retrievals on 54 and 43 levels (except SSMIS, run on 43L only) using RTTOV12. HDF coefficient files for all instruments where available. - IASI: rtcoef_metop_2_iasi_pcrttov_compat.H5: RTTOV-9 Predictors / LBLRTM / 101L - AIRS: rtcoef_eos_2_airs.dat: RTTOV-7 Predictors / LBLRTM / 54L - ATOVS: rtcoef_noaa_15_hirs.dat: RTTOV-7 Predictors / LBLRTM / 54L rtcoef_noaa_15_amsua.dat: RTTOV-7 Predictors / LIEBE-MPM / 54L rtcoef_noaa_15_amsub.dat: RTTOV-7 Predictors / LIEBE-MPM / 54L - ATMS: rtcoef_jpss_0_atms.dat: RTTOV-7 Predictors / LIEBE-MPM / 54L - CrIS: rtcoef_jpss_0_cris.dat: RTTOV-7 Predictors / LBLRTM / 54L - SSMIS: rtcoef_dmsp_16_ssmis.dat: RTTOV-7 Predictors / LIEBE-MPM / 54L This output is in Sample_Output_RTTOV12. For RTTOV12, additionally, the following were used to generate PC Score and Radiance output for IASI: rtcoef_metop_2_iasi_pcrttov_compat.H5: RTTOV-9 predictors / LBLRTM / 101L pccoef_metop_2_iasi_landsea_nlte.H5 For RTTOV11, additionally, the following were used to generate PC Score output for IASI: rtcoef_metop_2_iasi.H5: RTTOV-9 predictors / LBLRTM / 101L pccoef_metop_2_iasi.H5 The output from RTTOV11 and RTTOV12 is near identical: there are differences to a few thousandths of a K in one or two channels for each instrument (more channels for SSMIS); and some differences in the retrieved profiles, notably to the surface water vapour (s2m % q) retrieval. Compilers tested ---------------- The code has been compiled, and the test scripts run for both RTTOV11 and RTTOV12 with the following compilers: ifort 12.0.4 gfortran 4.4.7 gfortran 6.2.0 nagfor 5.2 pgf90 11.7 pgf90 14.6 pgf90 15.1 It has also been compiled with ifort 17.0. --------------------------- Fiona Smith Met Office 16th February 2017