Test climo and ts for land variables

Test climo and ts tasks for land variables. Resolves #531.

The monthly land ts looks like it mostly worked, but 20+ variables (from ILAMB variables) couldn't be found. Not sure if that's a problem with what I have specified as input though.
$ cat ts_land_monthly_1850-1851-0002.o431480

cmip
Climatology operations invoked with command:
/lcrc/soft/climate/e3sm-unified/spack/e3sm_unified_1_9_1_chrysalis_gnu_openmpi/opt/spack/linux-centos8-zen2/gcc-9.2.0/nco-5.1.9-ptt72cwya67odmjs7dj25lyrqkdrxu5h/bin/ncclimo -c v2.LR.historical_0201 -v ALT,AR,BTRAN,CH4PROD,DENIT,EFLX_LH_TOT,ELAI,ER,ESAI,FAREA_BURNED,FCEV,FCH4,FCH4TOCO2,FCOV,FCTR,FGEV,FGR,FGR12,FH2OSFC,FINUNDATED,FIRA,FIRE,FLDS,FPG,FPI,FPSN,FROST_TABLE,FSA,FSAT,FSDS,FSH,FSM,FSNO,FSR,F_DENIT,F_NIT,GPP,GROSS_NMIN,H2OSFC,H2OSNO,HR,HTOP,LAND_USE_FLUX,LEAFC,FROOTC,NDEP_TO_SMINN,NBP,NEE,NEP,NET_NMIN,NFIX_TO_SMINN,NPP,Q2M,QCHARGE,QDRAI,QOVER,QRUNOFF,QRGWL,QSNOMELT,QSOIL,QVEGE,QVEGT,RAIN,RH2M,SMIN_NO3,SMIN_NH4,SNOW,SNOWDP,SNOWICE,SNOWLIQ,SNOW_DEPTH,SNOW_SINKS,SNOW_SOURCES,SOMHR,TG,TSA,TREFMXAV,TREFMNAV,TSAI,TLAI,TV,QBOT,TBOT,AGNPP,FROOTC_ALLOC,LEAFC_ALLOC,WOODC_ALLOC,WOOD_HARVESTC,CH4_SURF_AERE_SAT,CH4_SURF_AERE_UNSAT,CH4_SURF_DIFF_SAT,CH4_SURF_DIFF_UNSAT,CH4_SURF_EBUL_SAT,CONC_CH4_SAT,CONC_CH4_UNSAT,FCH4_DFSAT,MR,TOTCOLCH4,ZWT_CH4_UNSAT,FSDSND,FSDSNI,FSDSVD,FSDSVI,TWS,VOLR,WA,ZWT_PERCH,ZWT,WIND,COL_FIRE_CLOSS,F_DENIT_vr,F_NIT_vr,H2OSOI,O_SCALAR,SOILICE,SOILLIQ,SOILPSI,TLAKE,TSOI,T_SCALAR,W_SCALAR,SOIL1N,SOIL2N,SOIL3N,SOIL1C,SOIL2C,SOIL3C,TOTVEGC,TOTVEGN,TOTECOSYSC,TOTLITC,TOTLITC_1m,TOTLITN_1m,TOTSOMC,TOTSOMC_1m,TOTSOMN_1m,CWDC,PBOT --mem_mb=0 --split --var_xtr=landfrac --yr_srt=1850 --yr_end=1851 --ypf=2 --map=/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc -o trash -O output --prc_typ=elm
Started climatology splitting at Wed Nov 22 12:10:24 CST 2023
Running climatology script ncclimo from directory /lcrc/soft/climate/e3sm-unified/spack/e3sm_unified_1_9_1_chrysalis_gnu_openmpi/opt/spack/linux-centos8-zen2/gcc-9.2.0/nco-5.1.9-ptt72cwya67odmjs7dj25lyrqkdrxu5h/bin
NCO binaries version 5.1.9 from directory /lcrc/soft/climate/e3sm-unified/spack/e3sm_unified_1_9_1_chrysalis_gnu_openmpi/opt/spack/linux-centos8-zen2/gcc-9.2.0/nco-5.1.9-ptt72cwya67odmjs7dj25lyrqkdrxu5h/bin
Parallelism mode = background
Timeseries will be created for each of 138 variables
Background parallelism processing variables in var_nbr/job_nbr = 138/138 = 1 sequential batches each concurrently processing job_nbr = 138 jobs (1 per variable), then remaining 0 jobs/variables simultaneously
Will split data for each variable into one timeseries of length 2 years and 0 months
Splitting climatology from list of 24 raw input files piped to stdin
Each input file assumed to contain statistics for one month
Native-grid split files to directory trash
Regridded split files to directory output
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'CH4_SURF_DIFF_SAT' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'WOOD_HARVESTC' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'ZWT_CH4_UNSAT' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'CONC_CH4_SAT' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'SOIL3C' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'HR' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'TOTSOMC_1m' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'TOTCOLCH4' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'COL_FIRE_CLOSS' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'F_DENIT' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'TOTECOSYSC' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'ER' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'CH4_SURF_DIFF_UNSAT' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'T_SCALAR' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'ALT' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'FCH4TOCO2' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'SOIL1C' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'TOTSOMC' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'DENIT' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'CH4_SURF_AERE_SAT' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'NPP' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'HTOP' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'MR' is not in and/or does not match contents of input file
ncclimo: ERROR Failed to split. cmd_sbs[0] failed. Debug this:
 OMP_PROC_BIND=false ncrcat -O -v ALT,landfrac --no_tmp_fl --hdr_pad=10000   -p /lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/scripts/tmp.431480.Ksjz  v2.LR.historical_0201.elm.h0.1850-01.nc v2.LR.historical_0201.elm.h0.1850-02.nc v2.LR.historical_0201.elm.h0.1850-03.nc v2.LR.historical_0201.elm.h0.1850-04.nc v2.LR.historical_0201.elm.h0.1850-05.nc v2.LR.historical_0201.elm.h0.1850-06.nc v2.LR.historical_0201.elm.h0.1850-07.nc v2.LR.historical_0201.elm.h0.1850-08.nc v2.LR.historical_0201.elm.h0.1850-09.nc v2.LR.historical_0201.elm.h0.1850-10.nc v2.LR.historical_0201.elm.h0.1850-11.nc v2.LR.historical_0201.elm.h0.1850-12.nc v2.LR.historical_0201.elm.h0.1851-01.nc v2.LR.historical_0201.elm.h0.1851-02.nc v2.LR.historical_0201.elm.h0.1851-03.nc v2.LR.historical_0201.elm.h0.1851-04.nc v2.LR.historical_0201.elm.h0.1851-05.nc v2.LR.historical_0201.elm.h0.1851-06.nc v2.LR.historical_0201.elm.h0.1851-07.nc v2.LR.historical_0201.elm.h0.1851-08.nc v2.LR.historical_0201.elm.h0.1851-09.nc v2.LR.historical_0201.elm.h0.1851-10.nc v2.LR.historical_0201.elm.h0.1851-11.nc v2.LR.historical_0201.elm.h0.1851-12.nc trash/ALT_185001_185112.nc

As for the globally averaged ts, 50+ variables couldn't be found:
$ cat ts_land_monthly_glb_1850-1854-0005.o431482

cmip
Climatology operations invoked with command:
/lcrc/soft/climate/e3sm-unified/spack/e3sm_unified_1_9_1_chrysalis_gnu_openmpi/opt/spack/linux-centos8-zen2/gcc-9.2.0/nco-5.1.9-ptt72cwya67odmjs7dj25lyrqkdrxu5h/bin/ncclimo -c v2.LR.historical_0201 -v ALT,AR,BTRAN,CH4PROD,DENIT,EFLX_LH_TOT,ELAI,ER,ESAI,FAREA_BURNED,FCEV,FCH4,FCH4TOCO2,FCOV,FCTR,FGEV,FGR,FGR12,FH2OSFC,FINUNDATED,FIRA,FIRE,FLDS,FPG,FPI,FPSN,FROST_TABLE,FSA,FSAT,FSDS,FSH,FSM,FSNO,FSR,F_DENIT,F_NIT,GPP,GROSS_NMIN,H2OSFC,H2OSNO,HR,HTOP,LAND_USE_FLUX,LEAFC,FROOTC,NDEP_TO_SMINN,NBP,NEE,NEP,NET_NMIN,NFIX_TO_SMINN,NPP,Q2M,QCHARGE,QDRAI,QOVER,QRUNOFF,QRGWL,QSNOMELT,QSOIL,QVEGE,QVEGT,RAIN,RH2M,SMIN_NO3,SMIN_NH4,SNOW,SNOWDP,SNOWICE,SNOWLIQ,SNOW_DEPTH,SNOW_SINKS,SNOW_SOURCES,SOMHR,TG,TSA,TREFMXAV,TREFMNAV,TSAI,TLAI,TV,QBOT,TBOT,AGNPP,FROOTC_ALLOC,LEAFC_ALLOC,WOODC_ALLOC,WOOD_HARVESTC,CH4_SURF_AERE_SAT,CH4_SURF_AERE_UNSAT,CH4_SURF_DIFF_SAT,CH4_SURF_DIFF_UNSAT,CH4_SURF_EBUL_SAT,CONC_CH4_SAT,CONC_CH4_UNSAT,FCH4_DFSAT,MR,TOTCOLCH4,ZWT_CH4_UNSAT,FSDSND,FSDSNI,FSDSVD,FSDSVI,TWS,VOLR,WA,ZWT_PERCH,ZWT,WIND,COL_FIRE_CLOSS,F_DENIT_vr,F_NIT_vr,H2OSOI,O_SCALAR,SOILICE,SOILLIQ,SOILPSI,TLAKE,TSOI,T_SCALAR,W_SCALAR,SOIL1N,SOIL2N,SOIL3N,SOIL1C,SOIL2C,SOIL3C,TOTVEGC,TOTVEGN,TOTECOSYSC,TOTLITC,TOTLITC_1m,TOTLITN_1m,TOTSOMC,TOTSOMC_1m,TOTSOMN_1m,CWDC,PBOT --mem_mb=0 --split --var_xtr=landfrac --yr_srt=1850 --yr_end=1854 --ypf=5 -o output --rgn_avg --area=area --prc_typ=elm
Started climatology splitting at Wed Nov 22 12:10:24 CST 2023
Running climatology script ncclimo from directory /lcrc/soft/climate/e3sm-unified/spack/e3sm_unified_1_9_1_chrysalis_gnu_openmpi/opt/spack/linux-centos8-zen2/gcc-9.2.0/nco-5.1.9-ptt72cwya67odmjs7dj25lyrqkdrxu5h/bin
NCO binaries version 5.1.9 from directory /lcrc/soft/climate/e3sm-unified/spack/e3sm_unified_1_9_1_chrysalis_gnu_openmpi/opt/spack/linux-centos8-zen2/gcc-9.2.0/nco-5.1.9-ptt72cwya67odmjs7dj25lyrqkdrxu5h/bin
Parallelism mode = background
Timeseries will be created for each of 138 variables
Background parallelism processing variables in var_nbr/job_nbr = 138/138 = 1 sequential batches each concurrently processing job_nbr = 138 jobs (1 per variable), then remaining 0 jobs/variables simultaneously
Will split data for each variable into one timeseries of length 5 years and 0 months
Splitting climatology from list of 60 raw input files piped to stdin
Each input file assumed to contain statistics for one month
Native-grid split files to directory output
Split files will not be regridded
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'FCH4' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'GPP' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'F_DENIT_vr' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'FROOTC' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'F_DENIT' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'TOTCOLCH4' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'FINUNDATED' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'F_NIT_vr' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'LEAFC_ALLOC' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'FAREA_BURNED' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'TOTSOMC_1m' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'MR' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'FCH4TOCO2' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'TOTLITC' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'FPG' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'NBP' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'WOODC_ALLOC' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'ER' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'NEP' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'SOIL1C' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'T_SCALAR' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'WOOD_HARVESTC' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'SOILPSI' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'FPI' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'SOIL2C' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'FCH4_DFSAT' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'F_NIT' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'W_SCALAR' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'AGNPP' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'CONC_CH4_UNSAT' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'O_SCALAR' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'SOMHR' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'TOTSOMN_1m' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'SOIL2N' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'GROSS_NMIN' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'SOIL1N' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'CWDC' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'CH4_SURF_DIFF_SAT' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'ZWT_CH4_UNSAT' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'CH4_SURF_AERE_SAT' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'TOTECOSYSC' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'NFIX_TO_SMINN' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'HTOP' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'CH4_SURF_DIFF_UNSAT' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'SOIL3C' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'DENIT' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'TOTLITN_1m' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'LEAFC' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'COL_FIRE_CLOSS' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'ALT' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'SMIN_NH4' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'LAND_USE_FLUX' is not in and/or does not match contents of input file
ncclimo: ERROR Failed to split. cmd_sbs[0] failed. Debug this:
 OMP_PROC_BIND=false ncrcat -O -v ALT,landfrac,area,lat --no_tmp_fl --hdr_pad=10000   -p /lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/scripts/tmp.431482.B2O8  v2.LR.historical_0201.elm.h0.1850-01.nc v2.LR.historical_0201.elm.h0.1850-02.nc v2.LR.historical_0201.elm.h0.1850-03.nc v2.LR.historical_0201.elm.h0.1850-04.nc v2.LR.historical_0201.elm.h0.1850-05.nc v2.LR.historical_0201.elm.h0.1850-06.nc v2.LR.historical_0201.elm.h0.1850-07.nc v2.LR.historical_0201.elm.h0.1850-08.nc v2.LR.historical_0201.elm.h0.1850-09.nc v2.LR.historical_0201.elm.h0.1850-10.nc v2.LR.historical_0201.elm.h0.1850-11.nc v2.LR.historical_0201.elm.h0.1850-12.nc v2.LR.historical_0201.elm.h0.1851-01.nc v2.LR.historical_0201.elm.h0.1851-02.nc v2.LR.historical_0201.elm.h0.1851-03.nc v2.LR.historical_0201.elm.h0.1851-04.nc v2.LR.historical_0201.elm.h0.1851-05.nc v2.LR.historical_0201.elm.h0.1851-06.nc v2.LR.historical_0201.elm.h0.1851-07.nc v2.LR.historical_0201.elm.h0.1851-08.nc v2.LR.historical_0201.elm.h0.1851-09.nc v2.LR.historical_0201.elm.h0.1851-10.nc v2.LR.historical_0201.elm.h0.1851-11.nc v2.LR.historical_0201.elm.h0.1851-12.nc v2.LR.historical_0201.elm.h0.1852-01.nc v2.LR.historical_0201.elm.h0.1852-02.nc v2.LR.historical_0201.elm.h0.1852-03.nc v2.LR.historical_0201.elm.h0.1852-04.nc v2.LR.historical_0201.elm.h0.1852-05.nc v2.LR.historical_0201.elm.h0.1852-06.nc v2.LR.historical_0201.elm.h0.1852-07.nc v2.LR.historical_0201.elm.h0.1852-08.nc v2.LR.historical_0201.elm.h0.1852-09.nc v2.LR.historical_0201.elm.h0.1852-10.nc v2.LR.historical_0201.elm.h0.1852-11.nc v2.LR.historical_0201.elm.h0.1852-12.nc v2.LR.historical_0201.elm.h0.1853-01.nc v2.LR.historical_0201.elm.h0.1853-02.nc v2.LR.historical_0201.elm.h0.1853-03.nc v2.LR.historical_0201.elm.h0.1853-04.nc v2.LR.historical_0201.elm.h0.1853-05.nc v2.LR.historical_0201.elm.h0.1853-06.nc v2.LR.historical_0201.elm.h0.1853-07.nc v2.LR.historical_0201.elm.h0.1853-08.nc v2.LR.historical_0201.elm.h0.1853-09.nc v2.LR.historical_0201.elm.h0.1853-10.nc v2.LR.historical_0201.elm.h0.1853-11.nc v2.LR.historical_0201.elm.h0.1853-12.nc v2.LR.historical_0201.elm.h0.1854-01.nc v2.LR.historical_0201.elm.h0.1854-02.nc v2.LR.historical_0201.elm.h0.1854-03.nc v2.LR.historical_0201.elm.h0.1854-04.nc v2.LR.historical_0201.elm.h0.1854-05.nc v2.LR.historical_0201.elm.h0.1854-06.nc v2.LR.historical_0201.elm.h0.1854-07.nc v2.LR.historical_0201.elm.h0.1854-08.nc v2.LR.historical_0201.elm.h0.1854-09.nc v2.LR.historical_0201.elm.h0.1854-10.nc v2.LR.historical_0201.elm.h0.1854-11.nc v2.LR.historical_0201.elm.h0.1854-12.nc output/ALT_185001_185412.nc
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'TOTLITC_1m' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'CH4_SURF_EBUL_SAT' is not in and/or does not match contents of input file

As for climo, it seems like there's only one missing variable.
$ cat climo_land_monthly_180x360_aave_1850-1851.o431509
...

ncra: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'ALT' is not in and/or does not match contents of input file
srun: error: chr-0505: task 0: Exited with exit code 1
ncclimo: ERROR monthly climo cmd_clm[1] failed. Debug this:
srun --nodelist chr-0505 --nodes=1 ncra --clm_bnd=1850,1851,1,1,0 -O --no_tmp_fl -v ALT,AR,BTRAN,CH4PROD,DENIT,EFLX_LH_TOT,ELAI,ER,ESAI,FAREA_BURNED,FCEV,FCH4,FCH4TOCO2,FCOV,FCTR,FGEV,FGR,FGR12,FH2OSFC,FINUNDATED,FIRA,FIRE,FLDS,FPG,FPI,FPSN,FROST_TABLE,FSA,FSAT,FSDS,FSH,FSM,FSNO,FSR,F_DENIT,F_NIT,GPP,GROSS_NMIN,H2OSFC,H2OSNO,HR,HTOP,LAND_USE_FLUX,LEAFC,FROOTC,NDEP_TO_SMINN,NBP,NEE,NEP,NET_NMIN,NFIX_TO_SMINN,NPP,Q2M,QCHARGE,QDRAI,QOVER,QRUNOFF,QRGWL,QSNOMELT,QSOIL,QVEGE,QVEGT,RAIN,RH2M,SMIN_NO3,SMIN_NH4,SNOW,SNOWDP,SNOWICE,SNOWLIQ,SNOW_DEPTH,SNOW_SINKS,SNOW_SOURCES,SOMHR,TG,TSA,TREFMXAV,TREFMNAV,TSAI,TLAI,TV,QBOT,TBOT,AGNPP,FROOTC_ALLOC,LEAFC_ALLOC,WOODC_ALLOC,WOOD_HARVESTC,CH4_SURF_AERE_SAT,CH4_SURF_AERE_UNSAT,CH4_SURF_DIFF_SAT,CH4_SURF_DIFF_UNSAT,CH4_SURF_EBUL_SAT,CONC_CH4_SAT,CONC_CH4_UNSAT,FCH4_DFSAT,MR,TOTCOLCH4,ZWT_CH4_UNSAT,FSDSND,FSDSNI,FSDSVD,FSDSVI,TWS,VOLR,WA,ZWT_PERCH,ZWT,WIND,COL_FIRE_CLOSS,F_DENIT_vr,F_NIT_vr,H2OSOI,O_SCALAR,SOILICE,SOILLIQ,SOILPSI,TLAKE,TSOI,T_SCALAR,W_SCALAR,SOIL1N,SOIL2N,SOIL3N,SOIL1C,SOIL2C,SOIL3C,TOTVEGC,TOTVEGN,TOTECOSYSC,TOTLITC,TOTLITC_1m,TOTLITN_1m,TOTSOMC,TOTSOMC_1m,TOTSOMN_1m,CWDC,PBOT --hdr_pad=10000 --gaa climo_script=ncclimo --gaa climo_command="'/lcrc/soft/climate/e3sm-unified/spack/e3sm_unified_1_9_1_chrysalis_gnu_openmpi/opt/spack/linux-centos8-zen2/gcc-9.2.0/nco-5.1.9-ptt72cwya67odmjs7dj25lyrqkdrxu5h/bin/ncclimo --case=v2.LR.historical_0201 --jobs=4 --thr=1 --vars=ALT,AR,BTRAN,CH4PROD,DENIT,EFLX_LH_TOT,ELAI,ER,ESAI,FAREA_BURNED,FCEV,FCH4,FCH4TOCO2,FCOV,FCTR,FGEV,FGR,FGR12,FH2OSFC,FINUNDATED,FIRA,FIRE,FLDS,FPG,FPI,FPSN,FROST_TABLE,FSA,FSAT,FSDS,FSH,FSM,FSNO,FSR,F_DENIT,F_NIT,GPP,GROSS_NMIN,H2OSFC,H2OSNO,HR,HTOP,LAND_USE_FLUX,LEAFC,FROOTC,NDEP_TO_SMINN,NBP,NEE,NEP,NET_NMIN,NFIX_TO_SMINN,NPP,Q2M,QCHARGE,QDRAI,QOVER,QRUNOFF,QRGWL,QSNOMELT,QSOIL,QVEGE,QVEGT,RAIN,RH2M,SMIN_NO3,SMIN_NH4,SNOW,SNOWDP,SNOWICE,SNOWLIQ,SNOW_DEPTH,SNOW_SINKS,SNOW_SOURCES,SOMHR,TG,TSA,TREFMXAV,TREFMNAV,TSAI,TLAI,TV,QBOT,TBOT,AGNPP,FROOTC_ALLOC,LEAFC_ALLOC,WOODC_ALLOC,WOOD_HARVESTC,CH4_SURF_AERE_SAT,CH4_SURF_AERE_UNSAT,CH4_SURF_DIFF_SAT,CH4_SURF_DIFF_UNSAT,CH4_SURF_EBUL_SAT,CONC_CH4_SAT,CONC_CH4_UNSAT,FCH4_DFSAT,MR,TOTCOLCH4,ZWT_CH4_UNSAT,FSDSND,FSDSNI,FSDSVD,FSDSVI,TWS,VOLR,WA,ZWT_PERCH,ZWT,WIND,COL_FIRE_CLOSS,F_DENIT_vr,F_NIT_vr,H2OSOI,O_SCALAR,SOILICE,SOILLIQ,SOILPSI,TLAKE,TSOI,T_SCALAR,W_SCALAR,SOIL1N,SOIL2N,SOIL3N,SOIL1C,SOIL2C,SOIL3C,TOTVEGC,TOTVEGN,TOTECOSYSC,TOTLITC,TOTLITC_1m,TOTLITN_1m,TOTSOMC,TOTSOMC_1m,TOTSOMN_1m,CWDC,PBOT --parallel=mpi --yr_srt=1850 --yr_end=1851 --input=/lcrc/group/e3sm/ac.forsyth2//E3SMv2/v2.LR.historical_0201/archive/atm/hist --map=/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc --output=trash --regrid=output --prc_typ=eam'" --gaa climo_hostname=chr-0505 --gaa climo_version=5.1.9 --gaa yrs_averaged=1850-1851 -p /lcrc/group/e3sm/ac.forsyth2//E3SMv2/v2.LR.historical_0201/archive/atm/hist  v2.LR.historical_0201.eam.h0.1850-01.nc v2.LR.historical_0201.eam.h0.1851-01.nc trash/v2.LR.historical_0201_01_185001_185101_climo.nc

Using a smaller subset of vars, from https://acme-climate.atlassian.net/wiki/spaces/EIDMG/pages/3476979729/zppy+New+Feature+Requests:
$ cat climo_land_monthly_180x360_aave_1850-1851.o431524
...
ncclimo: ERROR Unable to find required input file /lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/scripts/tmp.431524.77Im/v2.LR.historical_0201.eam.h0.0001-01.nc
ncclimo: HINT All files implied to exist by the climatology bounds (start/end year/month) and by the specified (with -P or -m) or default model type, must be in /lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/scripts/tmp.431524.77Im before ncclimo will proceed

It looks like it's looking for the wrong years for some reason?
$ cat ts_land_monthly_1850-1851-0002.o431546
...
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'CPOOL' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'GPP' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'TOTECOSYSC' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'TOTVEGC' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'SMINN_TO_PLANT' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'NEE' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'SMINP_TO_PLANT' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'NFIRES' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'TOTSOMP' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'PLANT_NDEMAND_COL' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'TOTLITC' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'FPI' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'TOTSOMC' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'PLANT_PDEMAND_COL' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'FP_UPTAKE' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'TOTSOMN' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'FAREA_BURNED' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'NPP' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'NBP' is not in and/or does not match contents of input file
ncclimo: ERROR Failed to split. cmd_sbs[0] failed. Debug this:
 OMP_PROC_BIND=false ncrcat -O -v NBP,landfrac --no_tmp_fl --hdr_pad=10000   -p /lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/scripts/tmp.431546.q2CT  v2.LR.historical_0201.elm.h0.1850-01.nc v2.LR.historical_0201.elm.h0.1850-02.nc v2.LR.historical_0201.elm.h0.1850-03.nc v2.LR.historical_0201.elm.h0.1850-04.nc v2.LR.historical_0201.elm.h0.1850-05.nc v2.LR.historical_0201.elm.h0.1850-06.nc v2.LR.historical_0201.elm.h0.1850-07.nc v2.LR.historical_0201.elm.h0.1850-08.nc v2.LR.historical_0201.elm.h0.1850-09.nc v2.LR.historical_0201.elm.h0.1850-10.nc v2.LR.historical_0201.elm.h0.1850-11.nc v2.LR.historical_0201.elm.h0.1850-12.nc v2.LR.historical_0201.elm.h0.1851-01.nc v2.LR.historical_0201.elm.h0.1851-02.nc v2.LR.historical_0201.elm.h0.1851-03.nc v2.LR.historical_0201.elm.h0.1851-04.nc v2.LR.historical_0201.elm.h0.1851-05.nc v2.LR.historical_0201.elm.h0.1851-06.nc v2.LR.historical_0201.elm.h0.1851-07.nc v2.LR.historical_0201.elm.h0.1851-08.nc v2.LR.historical_0201.elm.h0.1851-09.nc v2.LR.historical_0201.elm.h0.1851-10.nc v2.LR.historical_0201.elm.h0.1851-11.nc v2.LR.historical_0201.elm.h0.1851-12.nc trash/NBP_185001_185112.nc

$ cat ts_land_monthly_glb_1850-1854-0005.o431548
...
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'NFIRES' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'FPI' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'NBP' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'SMINP_TO_PLANT' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'PLANT_NDEMAND_COL' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'SMINN_TO_PLANT' is not in and/or does not match contents of input file
ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression 'TOTSOMN' is not in and/or does not match contents of input file
ncclimo: ERROR Failed to split. cmd_sbs[0] failed. Debug this:
 OMP_PROC_BIND=false ncrcat -O -v NBP,landfrac,area,lat --no_tmp_fl --hdr_pad=10000   -p /lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/scripts/tmp.431548.EW5c  v2.LR.historical_0201.elm.h0.1850-01.nc v2.LR.historical_0201.elm.h0.1850-02.nc v2.LR.historical_0201.elm.h0.1850-03.nc v2.LR.historical_0201.elm.h0.1850-04.nc v2.LR.historical_0201.elm.h0.1850-05.nc v2.LR.historical_0201.elm.h0.1850-06.nc v2.LR.historical_0201.elm.h0.1850-07.nc v2.LR.historical_0201.elm.h0.1850-08.nc v2.LR.historical_0201.elm.h0.1850-09.nc v2.LR.historical_0201.elm.h0.1850-10.nc v2.LR.historical_0201.elm.h0.1850-11.nc v2.LR.historical_0201.elm.h0.1850-12.nc v2.LR.historical_0201.elm.h0.1851-01.nc v2.LR.historical_0201.elm.h0.1851-02.nc v2.LR.historical_0201.elm.h0.1851-03.nc v2.LR.historical_0201.elm.h0.1851-04.nc v2.LR.historical_0201.elm.h0.1851-05.nc v2.LR.historical_0201.elm.h0.1851-06.nc v2.LR.historical_0201.elm.h0.1851-07.nc v2.LR.historical_0201.elm.h0.1851-08.nc v2.LR.historical_0201.elm.h0.1851-09.nc v2.LR.historical_0201.elm.h0.1851-10.nc v2.LR.historical_0201.elm.h0.1851-11.nc v2.LR.historical_0201.elm.h0.1851-12.nc v2.LR.historical_0201.elm.h0.1852-01.nc v2.LR.historical_0201.elm.h0.1852-02.nc v2.LR.historical_0201.elm.h0.1852-03.nc v2.LR.historical_0201.elm.h0.1852-04.nc v2.LR.historical_0201.elm.h0.1852-05.nc v2.LR.historical_0201.elm.h0.1852-06.nc v2.LR.historical_0201.elm.h0.1852-07.nc v2.LR.historical_0201.elm.h0.1852-08.nc v2.LR.historical_0201.elm.h0.1852-09.nc v2.LR.historical_0201.elm.h0.1852-10.nc v2.LR.historical_0201.elm.h0.1852-11.nc v2.LR.historical_0201.elm.h0.1852-12.nc v2.LR.historical_0201.elm.h0.1853-01.nc v2.LR.historical_0201.elm.h0.1853-02.nc v2.LR.historical_0201.elm.h0.1853-03.nc v2.LR.historical_0201.elm.h0.1853-04.nc v2.LR.historical_0201.elm.h0.1853-05.nc v2.LR.historical_0201.elm.h0.1853-06.nc v2.LR.historical_0201.elm.h0.1853-07.nc v2.LR.historical_0201.elm.h0.1853-08.nc v2.LR.historical_0201.elm.h0.1853-09.nc v2.LR.historical_0201.elm.h0.1853-10.nc v2.LR.historical_0201.elm.h0.1853-11.nc v2.LR.historical_0201.elm.h0.1853-12.nc v2.LR.historical_0201.elm.h0.1854-01.nc v2.LR.historical_0201.elm.h0.1854-02.nc v2.LR.historical_0201.elm.h0.1854-03.nc v2.LR.historical_0201.elm.h0.1854-04.nc v2.LR.historical_0201.elm.h0.1854-05.nc v2.LR.historical_0201.elm.h0.1854-06.nc v2.LR.historical_0201.elm.h0.1854-07.nc v2.LR.historical_0201.elm.h0.1854-08.nc v2.LR.historical_0201.elm.h0.1854-09.nc v2.LR.historical_0201.elm.h0.1854-10.nc v2.LR.historical_0201.elm.h0.1854-11.nc v2.LR.historical_0201.elm.h0.1854-12.nc output/NBP_185001_185412.nc

@chengzhuzhang Above are my initial findings on computing climo and ts for land variables. I need to do more debugging.

The cfg is in the pull request. The scripts directory is /lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/scripts

Hi Ryan, thank you for starting on this, let's try a set of more standard variables: FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO
I currently don't have a good understanding of the Land output especially those with BGC options. We can get input from land groups later. At this point lets use above basic variables for testing purpose.
Based on Charlie's note (copied below), there is a way for nco to support fault-tolerant variable lists:
ncclimo can handle fault-tolerant variable lists via regular expressions: The option -v FSNT,AODVIS requires that both variables be present. You can indicate variables that are not required to be present by using a regular expression that matches the variable name exactly. For example, -v ^FSNT$,AODVIS allows FSNT to be absent and requires AODVIS to be present. In general, prefix each optional variable with ^ and postfix it with $. Nothing else changes. The climo will complete successfully with only the variables that are present that match the regular expressions. You can experiment with ncks to print the variable names that match the specified var_lst:
zender@sastrugi:~$ fl_in=/Users/zender/data/ne30/raw/20180129.DECKv1b_piControl.ne30_oEC.edison.cam.h0.0001-01.nc
zender@sastrugi:~$ var_lst=FSNT,AODVIS
zender@sastrugi:~$ ncks -C -v ${var_lst} --lst_xtr ${fl_in}
AODVIS,FSNT
zender@sastrugi:~$ var_lst=^FSNT$,AODVIS
zender@sastrugi:~$ ncks -C -v ${var_lst} --lst_xtr ${fl_in}
AODVIS,FSNT
zender@sastrugi:~$ var_lst=^QUARK$,^FSNT$,AODVIS
zender@sastrugi:~$ ncks -C -v ${var_lst} --lst_xtr ${fl_in}
ncks: WARNING: Regular expression "^QUARK$" does not match any variable
HINT: See regular expression syntax examples at http://nco.sf.net/nco.html#rx
AODVIS,FSNT
zender@sastrugi:~$

Hi Ryan, thanks for sharing the testing results. Definitely more variables are listed than what is provided. Not sure where the original list came from, but I think the list includes some BGC specific varaibles where the standard v2.historical run doesn't have. Though one thing I notice is that, the list of variables are not consistence cross all land related operations: climo, ts_monthly_glb, ts_monthly. I assume 3 of them should support the same set of available variables from the native output. What's more puzzling is that none of the variables are supported by climo--land_monthly_180x360_aave.
climo -- land_monthly_180x360_aave:

    Valid: None
    Had to delete: "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO"

ts -- land_monthly:

    Valid: "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO"
    Had to delete: None

ts -- land_monthly_glb:

    Valid: "FSH,RH2M,LAISUN,SOILLIQ,SOILWATER_10CM,TSA,H2OSNO"
    Had to delete: "QOVER,QVEGE, QRUNOFF,QVEGT,TSOI,QSOIL,SOILICE,LAISHA,QINTR

Not sure where the original list came from

Before your suggested shorter variable list, I was using variables listed at https://acme-climate.atlassian.net/wiki/spaces/EIDMG/pages/3476979729/zppy+New+Feature+Requests (linked at https://github.com/E3SM-Project/zppy/issues/385) and https://bitbucket.org/minxu/alm2ilamb_wkflow/src/8b8fb98911911d930cc7f598ea64b83fdb199d4c/elm_singlevar_ts.bash#lines-304 (linked at https://github.com/E3SM-Project/zppy/issues/507#issuecomment-1721879043).

I assume 3 of them should support the same set of available variables from the native output

Yes, that was quite puzzling that they should differ so much.

What's more puzzling is that none of the variables are supported by climo--land_monthly_180x360_aave

Agreed.

Okay, I think I found why none of the variables are supported by climo--land_monthly_180x360_aave.
In /lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/scripts/climo_land_monthly_180x360_aave_1850-1851.o434621 the ncclimo command used is:
/lcrc/soft/climate/e3sm-unified/spack/e3sm_unified_1_9_1_chrysalis_gnu_openmpi/opt/spack/linux-centos8-zen2/gcc-9.2.0/nco-5.1.9-ptt72cwya67odmjs7dj25lyrqkdrxu5h/bin/ncclimo --case=v2.LR.historical_0201 --jobs=4 --thr=1 --vars=FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --parallel=mpi --yr_srt=1850 --yr_end=1851 --input=/lcrc/group/e3sm/ac.forsyth2//E3SMv2/v2.LR.historical_0201/archive/atm/hist --map=/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc --output=trash --regrid=output --prc_typ=eam

Obviously, it is looking for land variabls in atmosphere output.

Re: ts -- land_monthly_glb, I don't see why variables needs to be deleted. Based on the log, It looks like all variables are generated with the short variable lists being tested? /lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/scripts/ts_land_monthly_glb_1855-1859-0005.o434631

Obviously, it is looking for land variabls in atmosphere output.

Oh wow, good catch. While the parameter inheritance can be useful, I'll admit it can make errors like this less obvious.

I don't see why variables needs to be deleted.

Hmm I was getting errors on the variables I deleted -- maybe they still ended up generating output though.

Ok setting up the land climo as follows:
  [[ land_monthly_180x360_aave ]]
  frequency = "monthly"
  input_files = "elm.h0"
  input_subdir = archive/lnd/hist
  vars = "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO"

I get:
ncks: ERROR nco_rgr_wgt() (aka "the regridder") reports unable to find sgs_frc_nm = landfrac in current input file, and unable to identify filename (ending with slash '/' or backslash '\', depending on the operating system) portion of that string to serve as local external file for sgs_frc input, exiting
ncks: ERROR nco_rgr_wgt() (aka "the regridder") reports unable to find sgs_frc_nm = landfrac in current input file, and unable to identify filename (ending with slash '/' or backslash '\', depending on the operating system) portion of that string to serve as local external file for sgs_frc input, exiting
ncks: ERROR nco_rgr_wgt() (aka "the regridder") reports unable to find sgs_frc_nm = landfrac in current input file, and unable to identify filename (ending with slash '/' or backslash '\', depending on the operating system) portion of that string to serve as local external file for sgs_frc input, exiting
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0493 --gaa remap_version=5.1.9    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc" "trash/v2.LR.historical_0201_05_185005_185105_climo.nc" "output/v2.LR.historical_0201_05_185005_185105_climo.nc"
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0493 --gaa remap_version=5.1.9    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc" "trash/v2.LR.historical_0201_01_185001_185101_climo.nc" "output/v2.LR.historical_0201_01_185001_185101_climo.nc"
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0493 --gaa remap_version=5.1.9    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc" "trash/v2.LR.historical_0201_09_185009_185109_climo.nc" "output/v2.LR.historical_0201_09_185009_185109_climo.nc"
ncks: ERROR nco_rgr_wgt() (aka "the regridder") reports unable to find sgs_frc_nm = landfrac in current input file, and unable to identify filename (ending with slash '/' or backslash '\', depending on the operating system) portion of that string to serve as local external file for sgs_frc input, exiting
ncks: ERROR nco_rgr_wgt() (aka "the regridder") reports unable to find sgs_frc_nm = landfrac in current input file, and unable to identify filename (ending with slash '/' or backslash '\', depending on the operating system) portion of that string to serve as local external file for sgs_frc input, exiting
ncks: ERROR nco_rgr_wgt() (aka "the regridder") reports unable to find sgs_frc_nm = landfrac in current input file, and unable to identify filename (ending with slash '/' or backslash '\', depending on the operating system) portion of that string to serve as local external file for sgs_frc input, exiting
ncks: ERROR nco_rgr_wgt() (aka "the regridder") reports unable to find sgs_frc_nm = landfrac in current input file, and unable to identify filename (ending with slash '/' or backslash '\', depending on the operating system) portion of that string to serve as local external file for sgs_frc input, exiting
ncks: ERROR nco_rgr_wgt() (aka "the regridder") reports unable to find sgs_frc_nm = landfrac in current input file, and unable to identify filename (ending with slash '/' or backslash '\', depending on the operating system) portion of that string to serve as local external file for sgs_frc input, exiting
ncks: ERROR nco_rgr_wgt() (aka "the regridder") reports unable to find sgs_frc_nm = landfrac in current input file, and unable to identify filename (ending with slash '/' or backslash '\', depending on the operating system) portion of that string to serve as local external file for sgs_frc input, exiting
ncks: ERROR nco_rgr_wgt() (aka "the regridder") reports unable to find sgs_frc_nm = landfrac in current input file, and unable to identify filename (ending with slash '/' or backslash '\', depending on the operating system) portion of that string to serve as local external file for sgs_frc input, exiting
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0493 --gaa remap_version=5.1.9    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc" "trash/v2.LR.historical_0201_08_185008_185108_climo.nc" "output/v2.LR.historical_0201_08_185008_185108_climo.nc"
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0493 --gaa remap_version=5.1.9    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc" "trash/v2.LR.historical_0201_10_185010_185110_climo.nc" "output/v2.LR.historical_0201_10_185010_185110_climo.nc"
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0493 --gaa remap_version=5.1.9    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc" "trash/v2.LR.historical_0201_06_185006_185106_climo.nc" "output/v2.LR.historical_0201_06_185006_185106_climo.nc"
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0493 --gaa remap_version=5.1.9    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc" "trash/v2.LR.historical_0201_02_185002_185102_climo.nc" "output/v2.LR.historical_0201_02_185002_185102_climo.nc"
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0493 --gaa remap_version=5.1.9    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc" "trash/v2.LR.historical_0201_11_185011_185111_climo.nc" "output/v2.LR.historical_0201_11_185011_185111_climo.nc"
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0493 --gaa remap_version=5.1.9    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc" "trash/v2.LR.historical_0201_03_185003_185103_climo.nc" "output/v2.LR.historical_0201_03_185003_185103_climo.nc"
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0493 --gaa remap_version=5.1.9    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc" "trash/v2.LR.historical_0201_07_185007_185107_climo.nc" "output/v2.LR.historical_0201_07_185007_185107_climo.nc"
ncks: ERROR nco_rgr_wgt() (aka "the regridder") reports unable to find sgs_frc_nm = landfrac in current input file, and unable to identify filename (ending with slash '/' or backslash '\', depending on the operating system) portion of that string to serve as local external file for sgs_frc input, exiting
ncks: ERROR nco_rgr_wgt() (aka "the regridder") reports unable to find sgs_frc_nm = landfrac in current input file, and unable to identify filename (ending with slash '/' or backslash '\', depending on the operating system) portion of that string to serve as local external file for sgs_frc input, exiting
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0493 --gaa remap_version=5.1.9    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc" "trash/v2.LR.historical_0201_04_185004_185104_climo.nc" "output/v2.LR.historical_0201_04_185004_185104_climo.nc"
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0493 --gaa remap_version=5.1.9    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc" "trash/v2.LR.historical_0201_12_185012_185112_climo.nc" "output/v2.LR.historical_0201_12_185012_185112_climo.nc"
srun: error: chr-0496: task 0: Exited with exit code 1
srun: error: chr-0495: task 0: Exited with exit code 1
srun: error: chr-0496: task 0: Exited with exit code 1
srun: error: chr-0493: task 0: Exited with exit code 1
srun: error: chr-0493: task 0: Exited with exit code 1
srun: error: chr-0496: task 0: Exited with exit code 1
srun: error: chr-0493: task 0: Exited with exit code 1
ncclimo: ERROR monthly regrid cmd_rgr[1] failed. Debug this:
srun --nodelist chr-0496 --nodes=1 ncremap -u .pid2141167.climo.1.tmp --nco_opt="--no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000" -t 1 -P elm  --map=/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc trash/v2.LR.historical_0201_01_185001_185101_climo.nc output/v2.LR.historical_0201_01_185001_185101_climo.nc
srun: error: chr-0494: task 0: Exited with exit code 1
srun: error: chr-0494: task 0: Exited with exit code 1
srun: error: chr-0494: task 0: Exited with exit code 1
srun: error: chr-0495: task 0: Exited with exit code 1
srun: error: chr-0495: task 0: Exited with exit code 1

It looks like the landfrac isn't being found, which is causing an issue in the horizontal regridding.

@chengzhuzhang do you have any input on the following errors?
This is my cfg:
cfg
[default]
case = v2.LR.historical_0201
constraint = ""
dry_run = "False"
environment_commands = ""
input = "/lcrc/group/e3sm/ac.forsyth2//E3SMv2/v2.LR.historical_0201"
input_subdir = archive/atm/hist
mapping_file = "map_ne30pg2_to_cmip6_180x360_aave.20200201.nc"
# To run this test, edit `output` and `www` in this file, along with `actual_images_dir` in test_complete_run.py
output = "/lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201"
partition = "debug"
qos = "regular"
www = "/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_test_complete_run_www/issue_531"

[climo]
active = True
walltime = "00:30:00"
years = "1850:1854:2", "1850:1854:4",

  [[ atm_monthly_180x360_aave ]]
  frequency = "monthly"
  # From https://acme-climate.atlassian.net/wiki/spaces/EIDMG/pages/3476979729/zppy+New+Feature+Requests
  #vars = "AODVIS,FLNS,FSNTOA,PRECSL,TGCLDLWP,TS,CCN3,FLNT,LINOZ_O3COL,QFLX,TMCO2,U10,CLDHGH,FLUT,LWCF,RESTOM,TMCO2,TMCO2_FFF,U850,CLDLOW,FSNS,PRECC,RHREFHT,TMCO2_LND,CLDMED,FSNT,PRECL,SHFLX,TMCO2_OCN,CO2,CO2_FFF,CO2_OCN,CO2_LND,FSNTC,PRECSC,SWCF,TREFHT"
  # 1. Remove LINOZ_O3COL
  # 2. Remove TMCO2
  # 3. Remove RESTOM
  # 4. Remove TMCO2
  # 5. Remove TMCO2_FFF
  # 6. Remove U850
  # 7. Remove RHREFHT
  # 8. Remove TMCO2_LND
  # 9. Remove TMCO2_OCN
  # 10. Remove CO2
  # 11. Remove CO2_FFF
  # 12. Remove CO2_OCN
  # 13. Remove CO2_LND
  vars = "AODVIS,FLNS,FSNTOA,PRECSL,TGCLDLWP,TS,CCN3,FLNT,QFLX,U10,CLDHGH,FLUT,LWCF,CLDLOW,FSNS,PRECC,CLDMED,FSNT,PRECL,SHFLX,FSNTC,PRECSC,SWCF,TREFHT"

  [[ land_monthly_180x360_aave ]]
  frequency = "monthly"
  input_files = "elm.h0"
  input_subdir = archive/lnd/hist
  # From https://bitbucket.org/minxu/alm2ilamb_wkflow/src/8b8fb98911911d930cc7f598ea64b83fdb199d4c/elm_singlevar_ts.bash#lines-304
  #vars = "ALT,AR,BTRAN,CH4PROD,DENIT,EFLX_LH_TOT,ELAI,ER,ESAI,FAREA_BURNED,FCEV,FCH4,FCH4TOCO2,FCOV,FCTR,FGEV,FGR,FGR12,FH2OSFC,FINUNDATED,FIRA,FIRE,FLDS,FPG,FPI,FPSN,FROST_TABLE,FSA,FSAT,FSDS,FSH,FSM,FSNO,FSR,F_DENIT,F_NIT,GPP,GROSS_NMIN,H2OSFC,H2OSNO,HR,HTOP,LAND_USE_FLUX,LEAFC,FROOTC,NDEP_TO_SMINN,NBP,NEE,NEP,NET_NMIN,NFIX_TO_SMINN,NPP,Q2M,QCHARGE,QDRAI,QOVER,QRUNOFF,QRGWL,QSNOMELT,QSOIL,QVEGE,QVEGT,RAIN,RH2M,SMIN_NO3,SMIN_NH4,SNOW,SNOWDP,SNOWICE,SNOWLIQ,SNOW_DEPTH,SNOW_SINKS,SNOW_SOURCES,SOMHR,TG,TSA,TREFMXAV,TREFMNAV,TSAI,TLAI,TV,QBOT,TBOT,AGNPP,FROOTC_ALLOC,LEAFC_ALLOC,WOODC_ALLOC,WOOD_HARVESTC,CH4_SURF_AERE_SAT,CH4_SURF_AERE_UNSAT,CH4_SURF_DIFF_SAT,CH4_SURF_DIFF_UNSAT,CH4_SURF_EBUL_SAT,CONC_CH4_SAT,CONC_CH4_UNSAT,FCH4_DFSAT,MR,TOTCOLCH4,ZWT_CH4_UNSAT,FSDSND,FSDSNI,FSDSVD,FSDSVI,TWS,VOLR,WA,ZWT_PERCH,ZWT,WIND,COL_FIRE_CLOSS,F_DENIT_vr,F_NIT_vr,H2OSOI,O_SCALAR,SOILICE,SOILLIQ,SOILPSI,TLAKE,TSOI,T_SCALAR,W_SCALAR,SOIL1N,SOIL2N,SOIL3N,SOIL1C,SOIL2C,SOIL3C,TOTVEGC,TOTVEGN,TOTECOSYSC,TOTLITC,TOTLITC_1m,TOTLITN_1m,TOTSOMC,TOTSOMC_1m,TOTSOMN_1m,CWDC,PBOT"
  # From https://acme-climate.atlassian.net/wiki/spaces/EIDMG/pages/3476979729/zppy+New+Feature+Requests
  #vars = "NBP,PCO2,GPP,NPP,NEE,TOTECOSYSC,TOTVEGC,TOTSOMC,TOTSOMN,TOTSOMP,TOTLITC,CPOOL,FAREA_BURNED,NFIRES,FPI,FP_UPTAKE,PLANT_NDEMAND_COL,PLANT_PDEMAND_COL,SMINN_TO_PLANT,SMINP_TO_PLANT"
  # vars = "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO"
  vars = "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO"

[ts]
active = True
walltime = "00:30:00"
years = "1850:1854:2",

  [[ atm_monthly_180x360_aave ]]
  frequency = "monthly"
  input_files = "eam.h0"
  input_subdir = "archive/atm/hist"
  # From https://acme-climate.atlassian.net/wiki/spaces/EIDMG/pages/3476979729/zppy+New+Feature+Requests
  #vars = "AODVIS,FLNS,FSNTOA,PRECSL,TGCLDLWP,TS,CCN3,FLNT,LINOZ_O3COL,QFLX,TMCO2,U10,CLDHGH,FLUT,LWCF,RESTOM,TMCO2,TMCO2_FFF,U850,CLDLOW,FSNS,PRECC,RHREFHT,TMCO2_LND,CLDMED,FSNT,PRECL,SHFLX,TMCO2_OCN,CO2,CO2_FFF,CO2_OCN,CO2_LND,FSNTC,PRECSC,SWCF,TREFHT"
  # 1. Remove LINOZ_O3COL,TMCO2_FFF,TMCO2,CO2_OCN,CO2_LND,CO2_FFF,CO2,U850,TMCO2_OCN,TMCO2_LND,RESTOM,RHREFHT
  vars = "AODVIS,FLNS,FSNTOA,PRECSL,TGCLDLWP,TS,CCN3,FLNT,QFLX,U10,CLDHGH,FLUT,LWCF,CLDLOW,FSNS,PRECC,CLDMED,FSNT,PRECL,SHFLX,FSNTC,PRECSC,SWCF,TREFHT"
  ts_fmt = "cmip"

  [[ atm_monthly_glb ]]
  frequency = "monthly"
  input_files = "eam.h0"
  input_subdir = "archive/atm/hist"
  mapping_file = "glb"
  # From https://acme-climate.atlassian.net/wiki/spaces/EIDMG/pages/3476979729/zppy+New+Feature+Requests
  #vars = "AODVIS,FLNS,FSNTOA,PRECSL,TGCLDLWP,TS,CCN3,FLNT,LINOZ_O3COL,QFLX,TMCO2,U10,CLDHGH,FLUT,LWCF,RESTOM,TMCO2,TMCO2_FFF,U850,CLDLOW,FSNS,PRECC,RHREFHT,TMCO2_LND,CLDMED,FSNT,PRECL,SHFLX,TMCO2_OCN,CO2,CO2_FFF,CO2_OCN,CO2_LND,FSNTC,PRECSC,SWCF,TREFHT"
  # 1. Remove RESTOM,LINOZ_O3COL,CO2,CO2_FFF,CO2_LND,TMCO2_OCN,RHREFHT,TMCO2,CO2_OCN,TMCO2_LND
  # 2. Remove U10,U850 as they appear to be adding on to variables names: TMCO2_FFF850,TMCO210
  # 3. Remove TMCO2
  # 4. Remove TMCO2_FFF
  # 5. ERROR Failed to globally and regionally average -- try removing CCN3
  vars = "AODVIS,FLNS,FSNTOA,PRECSL,TGCLDLWP,TS,FLNT,QFLX,CLDHGH,FLUT,LWCF,CLDLOW,FSNS,PRECC,CLDMED,FSNT,PRECL,SHFLX,FSNTC,PRECSC,SWCF,TREFHT"
  years = "1850:1860:5",

  [[ land_monthly ]]
  extra_vars = "landfrac"
  frequency = "monthly"
  input_files = "elm.h0"
  input_subdir = "archive/lnd/hist"
  # From https://bitbucket.org/minxu/alm2ilamb_wkflow/src/8b8fb98911911d930cc7f598ea64b83fdb199d4c/elm_singlevar_ts.bash#lines-304
  #vars = "ALT,AR,BTRAN,CH4PROD,DENIT,EFLX_LH_TOT,ELAI,ER,ESAI,FAREA_BURNED,FCEV,FCH4,FCH4TOCO2,FCOV,FCTR,FGEV,FGR,FGR12,FH2OSFC,FINUNDATED,FIRA,FIRE,FLDS,FPG,FPI,FPSN,FROST_TABLE,FSA,FSAT,FSDS,FSH,FSM,FSNO,FSR,F_DENIT,F_NIT,GPP,GROSS_NMIN,H2OSFC,H2OSNO,HR,HTOP,LAND_USE_FLUX,LEAFC,FROOTC,NDEP_TO_SMINN,NBP,NEE,NEP,NET_NMIN,NFIX_TO_SMINN,NPP,Q2M,QCHARGE,QDRAI,QOVER,QRUNOFF,QRGWL,QSNOMELT,QSOIL,QVEGE,QVEGT,RAIN,RH2M,SMIN_NO3,SMIN_NH4,SNOW,SNOWDP,SNOWICE,SNOWLIQ,SNOW_DEPTH,SNOW_SINKS,SNOW_SOURCES,SOMHR,TG,TSA,TREFMXAV,TREFMNAV,TSAI,TLAI,TV,QBOT,TBOT,AGNPP,FROOTC_ALLOC,LEAFC_ALLOC,WOODC_ALLOC,WOOD_HARVESTC,CH4_SURF_AERE_SAT,CH4_SURF_AERE_UNSAT,CH4_SURF_DIFF_SAT,CH4_SURF_DIFF_UNSAT,CH4_SURF_EBUL_SAT,CONC_CH4_SAT,CONC_CH4_UNSAT,FCH4_DFSAT,MR,TOTCOLCH4,ZWT_CH4_UNSAT,FSDSND,FSDSNI,FSDSVD,FSDSVI,TWS,VOLR,WA,ZWT_PERCH,ZWT,WIND,COL_FIRE_CLOSS,F_DENIT_vr,F_NIT_vr,H2OSOI,O_SCALAR,SOILICE,SOILLIQ,SOILPSI,TLAKE,TSOI,T_SCALAR,W_SCALAR,SOIL1N,SOIL2N,SOIL3N,SOIL1C,SOIL2C,SOIL3C,TOTVEGC,TOTVEGN,TOTECOSYSC,TOTLITC,TOTLITC_1m,TOTLITN_1m,TOTSOMC,TOTSOMC_1m,TOTSOMN_1m,CWDC,PBOT"
  # From https://acme-climate.atlassian.net/wiki/spaces/EIDMG/pages/3476979729/zppy+New+Feature+Requests
  #vars = "NBP,PCO2,GPP,NPP,NEE,TOTECOSYSC,TOTVEGC,TOTSOMC,TOTSOMN,TOTSOMP,TOTLITC,CPOOL,FAREA_BURNED,NFIRES,FPI,FP_UPTAKE,PLANT_NDEMAND_COL,PLANT_PDEMAND_COL,SMINN_TO_PLANT,SMINP_TO_PLANT"
  vars = "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO"
  ts_fmt = "cmip"

  [[ land_monthly_glb ]]
  extra_vars = "landfrac"
  frequency = "monthly"
  input_files = "elm.h0"
  input_subdir = "archive/lnd/hist"
  mapping_file = "glb"
  # From https://bitbucket.org/minxu/alm2ilamb_wkflow/src/8b8fb98911911d930cc7f598ea64b83fdb199d4c/elm_singlevar_ts.bash#lines-304
  #vars = "ALT,AR,BTRAN,CH4PROD,DENIT,EFLX_LH_TOT,ELAI,ER,ESAI,FAREA_BURNED,FCEV,FCH4,FCH4TOCO2,FCOV,FCTR,FGEV,FGR,FGR12,FH2OSFC,FINUNDATED,FIRA,FIRE,FLDS,FPG,FPI,FPSN,FROST_TABLE,FSA,FSAT,FSDS,FSH,FSM,FSNO,FSR,F_DENIT,F_NIT,GPP,GROSS_NMIN,H2OSFC,H2OSNO,HR,HTOP,LAND_USE_FLUX,LEAFC,FROOTC,NDEP_TO_SMINN,NBP,NEE,NEP,NET_NMIN,NFIX_TO_SMINN,NPP,Q2M,QCHARGE,QDRAI,QOVER,QRUNOFF,QRGWL,QSNOMELT,QSOIL,QVEGE,QVEGT,RAIN,RH2M,SMIN_NO3,SMIN_NH4,SNOW,SNOWDP,SNOWICE,SNOWLIQ,SNOW_DEPTH,SNOW_SINKS,SNOW_SOURCES,SOMHR,TG,TSA,TREFMXAV,TREFMNAV,TSAI,TLAI,TV,QBOT,TBOT,AGNPP,FROOTC_ALLOC,LEAFC_ALLOC,WOODC_ALLOC,WOOD_HARVESTC,CH4_SURF_AERE_SAT,CH4_SURF_AERE_UNSAT,CH4_SURF_DIFF_SAT,CH4_SURF_DIFF_UNSAT,CH4_SURF_EBUL_SAT,CONC_CH4_SAT,CONC_CH4_UNSAT,FCH4_DFSAT,MR,TOTCOLCH4,ZWT_CH4_UNSAT,FSDSND,FSDSNI,FSDSVD,FSDSVI,TWS,VOLR,WA,ZWT_PERCH,ZWT,WIND,COL_FIRE_CLOSS,F_DENIT_vr,F_NIT_vr,H2OSOI,O_SCALAR,SOILICE,SOILLIQ,SOILPSI,TLAKE,TSOI,T_SCALAR,W_SCALAR,SOIL1N,SOIL2N,SOIL3N,SOIL1C,SOIL2C,SOIL3C,TOTVEGC,TOTVEGN,TOTECOSYSC,TOTLITC,TOTLITC_1m,TOTLITN_1m,TOTSOMC,TOTSOMC_1m,TOTSOMN_1m,CWDC,PBOT"
  # From https://acme-climate.atlassian.net/wiki/spaces/EIDMG/pages/3476979729/zppy+New+Feature+Requests
  #vars = "NBP,PCO2,GPP,NPP,NEE,TOTECOSYSC,TOTVEGC,TOTSOMC,TOTSOMN,TOTSOMP,TOTLITC,CPOOL,FAREA_BURNED,NFIRES,FPI,FP_UPTAKE,PLANT_NDEMAND_COL,PLANT_PDEMAND_COL,SMINN_TO_PLANT,SMINP_TO_PLANT"
  #vars = "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO"
  # 1. Remove QOVER,QVEGE
  # 2. Remove QRUNOFF
  # 3. Still errors on QOVER
  # 4. Still errors on QVEGE
  # 5. Still errors on QOVER
  # 6. Remove QVEGT
  # 7. Still errors on QVEGT
  # 8. Remove TSOI
  # 9. Remove QSOIL
  # 10. Remove SOILICE
  # 11. Still errors on SOILICE
  # 12. Remove LAISHA
  # 13. Remove QINTR
  # 14. Still errors on QVEGE
  # 15. Still errors on LAISHA
  # 16. Still errors on QVEGE
  # 17. Still errors on SOILICE
  # 18. Still errors on TSOI
  vars = "FSH,RH2M,LAISUN,SOILLIQ,SOILWATER_10CM,TSA,H2OSNO"
  ts_fmt = "cmip"
  years = "1850:1860:5",

(1) From /lcrc/group/e3sm/ac.forsyth2/E3SMv2/v2.LR.historical_0201/archive/lnd/hist, I tried running  ncdump -h v2.LR.historical_0201.elm.h0.1909-08.nc | grep <var> for QOVER,QVEGE,QRUNOFF,QVEGT,TSOI,QSOIL,SOILICE,LAISHA,QINTR. In all cases, the variables were found. So, I'm not sure why those variables caused failures.
$ cd /lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/scripts
$ grep -v "OK" *status
climo_land_monthly_180x360_aave_1850-1851.status:ERROR (3)
climo_land_monthly_180x360_aave_1850-1853.status:ERROR (3)
climo_land_monthly_180x360_aave_1852-1853.status:ERROR (3)
ts_land_monthly_glb_1850-1854-0005.status:ERROR (5)
ts_land_monthly_glb_1855-1859-0005.status:ERROR (5)

(2) climo_land_monthly errors:
ncks: ERROR nco_rgr_wgt() (aka "the regridder") reports unable to find sgs_frc_nm = landfrac in current input file, and unable to identify filename (ending with slash '/' or backslash '\', depending on the operating system) portion of that string to serve as local external file for sgs_frc input, exiting
ncks: ERROR nco_rgr_wgt() (aka "the regridder") reports unable to find sgs_frc_nm = landfrac in current input file, and unable to identify filename (ending with slash '/' or backslash '\', depending on the operating system) portion of that string to serve as local external file for sgs_frc input, exiting
ncks: ERROR nco_rgr_wgt() (aka "the regridder") reports unable to find sgs_frc_nm = landfrac in current input file, and unable to identify filename (ending with slash '/' or backslash '\', depending on the operating system) portion of that string to serve as local external file for sgs_frc input, exiting
ncks: ERROR nco_rgr_wgt() (aka "the regridder") reports unable to find sgs_frc_nm = landfrac in current input file, and unable to identify filename (ending with slash '/' or backslash '\', depending on the operating system) portion of that string to serve as local external file for sgs_frc input, exiting
ncks: ERROR nco_rgr_wgt() (aka "the regridder") reports unable to find sgs_frc_nm = landfrac in current input file, and unable to identify filename (ending with slash '/' or backslash '\', depending on the operating system) portion of that string to serve as local external file for sgs_frc input, exiting
ncks: ERROR nco_rgr_wgt() (aka "the regridder") reports unable to find sgs_frc_nm = landfrac in current input file, and unable to identify filename (ending with slash '/' or backslash '\', depending on the operating system) portion of that string to serve as local external file for sgs_frc input, exiting
ncks: ERROR nco_rgr_wgt() (aka "the regridder") reports unable to find sgs_frc_nm = landfrac in current input file, and unable to identify filename (ending with slash '/' or backslash '\', depending on the operating system) portion of that string to serve as local external file for sgs_frc input, exiting
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0506 --gaa remap_version=5.1.9    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc" "trash/v2.LR.historical_0201_10_185010_185110_climo.nc" "output/v2.LR.historical_0201_10_185010_185110_climo.nc"
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0506 --gaa remap_version=5.1.9    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc" "trash/v2.LR.historical_0201_07_185007_185107_climo.nc" "output/v2.LR.historical_0201_07_185007_185107_climo.nc"
ncks: ERROR nco_rgr_wgt() (aka "the regridder") reports unable to find sgs_frc_nm = landfrac in current input file, and unable to identify filename (ending with slash '/' or backslash '\', depending on the operating system) portion of that string to serve as local external file for sgs_frc input, exiting
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0506 --gaa remap_version=5.1.9    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc" "trash/v2.LR.historical_0201_02_185002_185102_climo.nc" "output/v2.LR.historical_0201_02_185002_185102_climo.nc"
ncks: ERROR nco_rgr_wgt() (aka "the regridder") reports unable to find sgs_frc_nm = landfrac in current input file, and unable to identify filename (ending with slash '/' or backslash '\', depending on the operating system) portion of that string to serve as local external file for sgs_frc input, exiting
ncks: ERROR nco_rgr_wgt() (aka "the regridder") reports unable to find sgs_frc_nm = landfrac in current input file, and unable to identify filename (ending with slash '/' or backslash '\', depending on the operating system) portion of that string to serve as local external file for sgs_frc input, exiting
ncks: ERROR nco_rgr_wgt() (aka "the regridder") reports unable to find sgs_frc_nm = landfrac in current input file, and unable to identify filename (ending with slash '/' or backslash '\', depending on the operating system) portion of that string to serve as local external file for sgs_frc input, exiting
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0506 --gaa remap_version=5.1.9    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc" "trash/v2.LR.historical_0201_06_185006_185106_climo.nc" "output/v2.LR.historical_0201_06_185006_185106_climo.nc"
ncks: ERROR nco_rgr_wgt() (aka "the regridder") reports unable to find sgs_frc_nm = landfrac in current input file, and unable to identify filename (ending with slash '/' or backslash '\', depending on the operating system) portion of that string to serve as local external file for sgs_frc input, exiting
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0506 --gaa remap_version=5.1.9    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc" "trash/v2.LR.historical_0201_05_185005_185105_climo.nc" "output/v2.LR.historical_0201_05_185005_185105_climo.nc"
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0506 --gaa remap_version=5.1.9    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc" "trash/v2.LR.historical_0201_04_185004_185104_climo.nc" "output/v2.LR.historical_0201_04_185004_185104_climo.nc"
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0506 --gaa remap_version=5.1.9    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc" "trash/v2.LR.historical_0201_09_185009_185109_climo.nc" "output/v2.LR.historical_0201_09_185009_185109_climo.nc"
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0506 --gaa remap_version=5.1.9    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc" "trash/v2.LR.historical_0201_08_185008_185108_climo.nc" "output/v2.LR.historical_0201_08_185008_185108_climo.nc"
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0506 --gaa remap_version=5.1.9    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc" "trash/v2.LR.historical_0201_01_185001_185101_climo.nc" "output/v2.LR.historical_0201_01_185001_185101_climo.nc"
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0506 --gaa remap_version=5.1.9    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc" "trash/v2.LR.historical_0201_11_185011_185111_climo.nc" "output/v2.LR.historical_0201_11_185011_185111_climo.nc"
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0506 --gaa remap_version=5.1.9    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc" "trash/v2.LR.historical_0201_12_185012_185112_climo.nc" "output/v2.LR.historical_0201_12_185012_185112_climo.nc"
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0506 --gaa remap_version=5.1.9    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc" "trash/v2.LR.historical_0201_03_185003_185103_climo.nc" "output/v2.LR.historical_0201_03_185003_185103_climo.nc"
srun: error: chr-0507: task 0: Exited with exit code 1
srun: error: chr-0507: task 0: Exited with exit code 1
srun: error: chr-0507: task 0: Exited with exit code 1
srun: error: chr-0509: task 0: Exited with exit code 1
srun: error: chr-0506: task 0: Exited with exit code 1
srun: error: chr-0509: task 0: Exited with exit code 1
srun: error: chr-0509: task 0: Exited with exit code 1
srun: error: chr-0506: task 0: Exited with exit code 1
srun: error: chr-0506: task 0: Exited with exit code 1
srun: error: chr-0508: task 0: Exited with exit code 1
srun: error: chr-0508: task 0: Exited with exit code 1
srun: error: chr-0508: task 0: Exited with exit code 1
ncclimo: ERROR monthly regrid cmd_rgr[1] failed. Debug this:
srun --nodelist chr-0509 --nodes=1 ncremap -u .pid1820050.climo.1.tmp --nco_opt="--no_tmp_fl -v FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO --hdr_pad=10000" -t 1 -P elm  --map=/lcrc/group/e3sm/diagnostics/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc trash/v2.LR.historical_0201_01_185001_185101_climo.nc output/v2.LR.historical_0201_01_185001_185101_climo.nc

As mentioned in https://github.com/E3SM-Project/zppy/pull/533#issuecomment-1832819123, it looks like landfrac isn't being found. That said, ncdump -h v2.LR.historical_0201.elm.h0.1909-08.nc | grep landfrac also shows landfrac as available. I tried running the Debug this line but get srun: error: Unable to allocate resources: Requested node configuration is not available.
(3) ts_land_monthly_glb errors:
2023-12-12 21:00:24,283_283:INFO:__init__:| E3SM to CMIP Configuration
2023-12-12 21:00:24,283_283:INFO:__init__:--------------------------------------
2023-12-12 21:00:24,283_283:INFO:__init__:    * var_list='['mrsos', 'mrso', 'mrfso', 'mrros', 'mrro', 'prveg', 'evspsblveg', 'evspsblsoi', 'tran', 'tsl', 'lai', 'cLitter', 'cProduct', 'cSoilFast', 'cSoilMedium', 'cSoilSlow', 'fFire', 'fHarvest', 'cVeg', 'nbp', 'gpp', 'ra', 'rh']'
2023-12-12 21:00:24,283_283:INFO:__init__:    * input_path='/lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/lnd/glb/ts/monthly/5yr'
2023-12-12 21:00:24,283_283:INFO:__init__:    * output_path='/lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/lnd/glb/cmip_ts/monthly/tmp_ts_land_monthly_glb_1850-1854-0005'
2023-12-12 21:00:24,283_283:INFO:__init__:    * precheck_path='None'
2023-12-12 21:00:24,283_283:INFO:__init__:    * freq='mon'
2023-12-12 21:00:24,283_283:INFO:__init__:    * realm='lnd'
2023-12-12 21:00:24,283_283:INFO:__init__:    * Writing log output file to: logs/20231212_210024_276122
2023-12-12 21:00:38,592_592:WARNING:derive_handlers:No handlers could be derived for the variables: ['mrso', 'mrfso', 'mrros', 'mrro', 'prveg', 'evspsblveg', 'evspsblsoi', 'tran', 'tsl', 'lai', 'cLitter', 'cProduct', 'cSoilFast', 'cSoilMedium', 'cSoilSlow', 'fFire', 'fHarvest', 'cVeg', 'nbp', 'gpp', 'ra', 'rh']. Make sure the input E3SM datasets have the variables needed derivation.
2023-12-12 21:00:38,593_593:INFO:_get_handlers:--------------------------------------
2023-12-12 21:00:38,593_593:INFO:_get_handlers:| Derived CMIP6 Variable Handlers
2023-12-12 21:00:38,593_593:INFO:_get_handlers:--------------------------------------
2023-12-12 21:00:38,593_593:INFO:_get_handlers:    * 'mrsos' -> ['SOILWATER_10CM']
2023-12-12 21:00:38,598_598:INFO:run:--------------------------------------
2023-12-12 21:00:38,598_598:INFO:run:| Running E3SM to CMIP in Parallel
2023-12-12 21:00:38,598_598:INFO:run:--------------------------------------

  0%|          | 0/1 [00:00<?, ?it/s]2023-12-12 21:00:38,736_736:INFO:handle_variables:mrsos: Starting
2023-12-12 21:00:38,840_840:INFO:handle_variables:mrsos: CMOR setup complete
2023-12-12 21:00:38,840_840:INFO:handle_variables:mrsos: loading SOILWATER_10CM
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.1_chrysalis/lib/python3.10/site-packages/e3sm_to_cmip/lib.py", line 116, in run_parallel
    out = res.result()
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.1_chrysalis/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.1_chrysalis/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception

100%|██████████| 1/1 [00:00<00:00,  4.81it/s]
100%|██████████| 1/1 [00:00<00:00,  4.81it/s]
2023-12-12 21:00:38,939_939:INFO:run_parallel:0 of 1 handlers complete
2023-12-12 21:00:38,939_939:ERROR:run_parallel:mrsos failed to complete
2023-12-12 21:00:38,939_939:ERROR:run_parallel:0 of 1 handlers complete
'lat'
mv: cannot stat '/lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/lnd/glb/cmip_ts/monthly/tmp_ts_land_monthly_glb_1850-1854-0005/CMIP6/CMIP/*/*/*/*/*/*/*/*/*.nc': No such file or directory

It looks like /lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/lnd/glb/cmip_ts/monthly/tmp_ts_land_monthly_glb_1850-1854-0005 doesn't have a CMIP6 directory, possibly because of an error loading "loading SOILWATER_10CM"?

The climo_land_monthly error seems to complain about missing landfrac. I suppose adding: extra_vars = "landfrac" like those ts tasks should work?

Hmm, that's going to require a code change. That's only implemented for ts:
$ git grep -n "extra_vars"
templates/default.ini:92:extra_vars = string(default="")
templates/default.ini:101:  extra_vars = string(default=None)
templates/ts.bash:88:{%- if extra_vars != '' %}
templates/ts.bash:89:--var_xtr={{extra_vars}} \

I'll see if that fixes it though.

@forsyth2 For testing purpose, if -v is not list, climatology will be generated for all variables including "landfrac".This can give an idea if missing landfrac caused the issue.

for ts_land_monthly_glb, i was originally confused with glb in the task name. Usually it was just called ts_land_xx. I believe the issue is the same as what was brought up here:https://github.com/E3SM-Project/zppy/issues/523#issuecomment-1778268281. In this case supply with a mapping file to remap first would bring the cell bounds to the datasets. @tomvothecoder has updated e3sm_to_cmip to add cell bounds. I believe with newly released e3sm_to_cmip, it can support the native r05 grids land data that don't have lat-lon bounds.

for ts_land_monthly_glb, i was originally confused with glb in the task name. Usually it was just called ts_land_xx. I believe the issue is the same as what was brought up here:#523 (comment). In this case supply with a mapping file to remap first would bring the cell bounds to the datasets. @tomvothecoder has updated e3sm_to_cmip to add cell bounds. I believe with newly released e3sm_to_cmip, it can support the native r05 grids land data that don't have lat-lon bounds.

Actually this problem is related to glb. it seems like, the glb (global mean files) data path is listed as input file for e3sm_to_cmip. For the v2 data being tested, it is not r05, so my comment about cell bounds above is irrelevant.

if -v is not list, climatology will be generated for all variables including "landfrac".

Yes, omitting the variable list does in fact make climo_land_monthly pass.

if -v is not list, climatology will be generated for all variables including "landfrac".

Yes, omitting the variable list does in fact make climo_land_monthly pass.

I'm wondering if there is a zppy configuration that can omit the variable list?

I'm wondering if there is a zppy configuration that can omit the variable list?

I think this is possible by setting vars="" in the configuration file. (If you don't include vars at all, a default list is provided, which isn't what you would want).
In zppy/templates/ts.bash:
{%- if vars != '' %}
-v ${vars} \

If vars is set to the empty string, then no -v is added. That means all the variables will be processed.

I think this is possible by setting vars="" in the configuration file. (If you don't include vars at all, a default list is provided, which isn't what you would want).

In zppy/templates/ts.bash:
{%- if vars != '' %}
-v ${vars} \

If vars is set to the empty string, then no -v is added. That means all the variables will be processed.

This is good to know, either  using vars='' to omitting the variable list (including all variables) or including landfrac could fix this problem.

Latest run:
$ cd /lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/scripts
$ grep -v "OK" *status
ts_land_monthly_glb_1850-1854-0005.status:ERROR (5)
ts_land_monthly_glb_1855-1859-0005.status:ERROR (5)
$ tail -n 19 ts_land_monthly_glb_1850-1854-0005.o460938
2024-01-23 18:37:43,164_164:INFO:cmorize:mrsos: creating CMOR variable with CMOR axis objects.
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/e3sm_to_cmip/__main__.py", line 912, in _run_parallel
    out = res.result()
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
100%|██████████| 1/1 [00:00<00:00,  3.76it/s]
2024-01-23 18:37:43,253 [INFO]: __main__.py(_run_parallel:930) >> 0 of 1 handlers complete
2024-01-23 18:37:43,253 [INFO]: __main__.py(_run_parallel:930) >> 0 of 1 handlers complete
2024-01-23 18:37:43,253_253:INFO:_run_parallel:0 of 1 handlers complete
2024-01-23 18:37:43,254 [ERROR]: __main__.py(_run_parallel:934) >> mrsos failed to complete
2024-01-23 18:37:43,254 [ERROR]: __main__.py(_run_parallel:934) >> mrsos failed to complete
2024-01-23 18:37:43,254_254:ERROR:_run_parallel:mrsos failed to complete
2024-01-23 18:37:43,254 [ERROR]: __main__.py(_run_parallel:935) >> 0 of 1 handlers complete
2024-01-23 18:37:43,254 [ERROR]: __main__.py(_run_parallel:935) >> 0 of 1 handlers complete
2024-01-23 18:37:43,254_254:ERROR:_run_parallel:0 of 1 handlers complete
'lat'
mv: cannot stat '/lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/lnd/glb/cmip_ts/monthly/tmp_ts_land_monthly_glb_1850-1854-0005/CMIP6/CMIP/*/*/*/*/*/*/*/*/*.nc': No such file or directory
$ ls /lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/lnd/glb/cmip_ts/monthly/tmp_ts_land_monthly_glb_1850-1854-0005
user_metadata.json
# No CMIP6 directory present.

That matches the issue in https://github.com/E3SM-Project/zppy/pull/534#issuecomment-1906552473.
I'll rerun with Unified 1.9.1.

$ tail -n 18 ts_land_monthly_glb_1850-1854-0005.o460943
2024-01-23 19:02:37,115_115:INFO:run:--------------------------------------
2024-01-23 19:02:37,116_116:INFO:run:| Running E3SM to CMIP in Parallel
2024-01-23 19:02:37,116_116:INFO:run:--------------------------------------
  0%|          | 0/1 [00:00<?, ?it/s]2024-01-23 19:02:37,237_237:INFO:handle_variables:mrsos: Starting
2024-01-23 19:02:37,403_403:INFO:handle_variables:mrsos: CMOR setup complete
2024-01-23 19:02:37,403_403:INFO:handle_variables:mrsos: loading SOILWATER_10CM
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.1_chrysalis/lib/python3.10/site-packages/e3sm_to_cmip/lib.py", line 116, in run_parallel
    out = res.result()
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.1_chrysalis/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.1_chrysalis/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
100%|██████████| 1/1 [00:00<00:00,  3.94it/s]
2024-01-23 19:02:37,507_507:INFO:run_parallel:0 of 1 handlers complete
2024-01-23 19:02:37,507_507:ERROR:run_parallel:mrsos failed to complete
'lat'
2024-01-23 19:02:37,507_507:ERROR:run_parallel:0 of 1 handlers complete
mv: cannot stat '/lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/lnd/glb/cmip_ts/monthly/tmp_ts_land_monthly_glb_1850-1854-0005/CMIP6/CMIP/*/*/*/*/*/*/*/*/*.nc': No such file or directory
$  ls /lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/lnd/glb/cmip_ts/monthly/tmp_ts_land_monthly_glb_1850-1854-0005
user_metadata.json
# No CMIP6 directory present.

There appears to be a different e3sm_to_cmip error. Unified 1.9.1 errors after INFO:handle_variables:mrsos: loading SOILWATER_10CM whereas Unified 1.9.2 errors after INFO:cmorize:mrsos: creating CMOR variable with CMOR axis objects.. The Unified 1.9.1 error matches error (3) in https://github.com/E3SM-Project/zppy/pull/533#issuecomment-1852970411 above.

Actually this problem is related to glb. it seems like, the glb (global mean files) data path is listed as input file for e3sm_to_cmip. For the v2 data being tested, it is not r05, so my comment about cell bounds above is irrelevant

@chengzhuzhang So, do we have any idea what's causing the e3sm_to_cmip errors in ts_land_monthly_glb?
Regarding the other errors mentioned in the comment linked above:

(1) appears to have been resolved.
(2) climo_land_monthly errors: fixed by omitting the variable list -- see https://github.com/E3SM-Project/zppy/pull/533#issuecomment-1853053760.

either using vars='' to omitting the variable list (including all variables) or including landfrac could fix this problem.

vars = string(default="") in zppy/templates/default.ini. In my cfg for this run, I didn't define vars at all, so by default vars='' and thus all variables were used. So, omitting the variable list is actually the default behavior already. That said, I think adding landfrac with a new extra_vars parameter (as in ts) would be a more robust solution, since it wouldn't require users to run climo on all variables. I suppose that could be addressed in a later issue/PR though. @chengzhuzhang thoughts?

For trouble shooting, could you exam the input files for e3sm_to_cmip/cmor, to see if the dimension of the variables are remapped to lat lon coordinate, as expected?

could you exam the input files for e3sm_to_cmip/cmor, to see if the dimension of the variables are remapped to lat lon coordinate, as expected?

In ts_land_monthly_glb_1850-1854-0005.bash:
  export cmortables_dir=/lcrc/group/e3sm/diagnostics/cmip6-cmor-tables/Tables
  input_dir=/lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/lnd/glb/ts/monthly/5yr
  dest_cmip=/lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/lnd/glb/cmip_ts/monthly
  mkdir -p ${dest_cmip}
  srun -N 1 e3sm_to_cmip \

Then:
# Go to `input_dir`:
$ cd /lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/lnd/glb/ts/monthly/5yr
$ ls
FSH_185001_185412.nc     H2OSNO_185501_185912.nc  RH2M_185001_185412.nc     SOILLIQ_185501_185912.nc         TSA_185001_185412.nc
FSH_185501_185912.nc     LAISUN_185001_185412.nc  RH2M_185501_185912.nc     SOILWATER_10CM_185001_185412.nc  TSA_185501_185912.nc
H2OSNO_185001_185412.nc  LAISUN_185501_185912.nc  SOILLIQ_185001_185412.nc  SOILWATER_10CM_185501_185912.nc
# Looking at SOILWATER_10CM, since that caused the Unified 1.9.1 error:
$ ncdump -h SOILWATER_10CM_185001_185412.nc
# Potentially relevant output:
dimensions:
	rgn = 3 ;
	time = UNLIMITED ; // (60 currently)
	lndgrid = 21600 ;
	rgn_len = 19 ;
# That matches the dimensions for `FSH_185001_185412.nc`, which seems to have worked fine.

@chengzhuzhang How can I tell if it's remapped?

Potentially relevant output:
dimensions:
rgn = 3 ;
time = UNLIMITED ; // (60 currently)
lndgrid = 21600 ;
rgn_len = 19 ;

Based on the dimensions, it looks like global time series files are used as input for e3sm_to_cmip. The expected files should come from regrided files with lat lon dimensions, or native land output is also supported for v3 tri-grid.

Looking at ts_land_monthly_1850-1851-0002.bash (i.e., the not-global land time-series), I see the following:
  export cmortables_dir=/lcrc/group/e3sm/diagnostics/cmip6-cmor-tables/Tables
  input_dir=/lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/lnd/180x360_aave/ts/monthly/2yr
  dest_cmip=/lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/lnd/180x360_aave/cmip_ts/monthly
  mkdir -p ${dest_cmip}
  srun -N 1 e3sm_to_cmip \

Then:
$ cd /lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/issue_531/v2.LR.historical_0201/post/lnd/180x360_aave/ts/monthly/2yr
$ ls
FSH_185001_185112.nc     LAISUN_185201_185312.nc   QSOIL_185001_185112.nc  RH2M_185201_185312.nc            TSA_185001_185112.nc
FSH_185201_185312.nc     QINTR_185001_185112.nc    QSOIL_185201_185312.nc  SOILICE_185001_185112.nc         TSA_185201_185312.nc
H2OSNO_185001_185112.nc  QINTR_185201_185312.nc    QVEGE_185001_185112.nc  SOILICE_185201_185312.nc         TSOI_185001_185112.nc
H2OSNO_185201_185312.nc  QOVER_185001_185112.nc    QVEGE_185201_185312.nc  SOILLIQ_185001_185112.nc         TSOI_185201_185312.nc
LAISHA_185001_185112.nc  QOVER_185201_185312.nc    QVEGT_185001_185112.nc  SOILLIQ_185201_185312.nc
LAISHA_185201_185312.nc  QRUNOFF_185001_185112.nc  QVEGT_185201_185312.nc  SOILWATER_10CM_185001_185112.nc
LAISUN_185001_185112.nc  QRUNOFF_185201_185312.nc  RH2M_185001_185112.nc   SOILWATER_10CM_185201_185312.nc
$ ncdump -h SOILWATER_10CM_185001_185112.nc | head -n 8
netcdf SOILWATER_10CM_185001_185112 {
dimensions:
	lat = 180 ;
	lon = 360 ;
	nbnd = 2 ;
	time = UNLIMITED ; // (24 currently)
	hist_interval = 2 ;
variables:

Indeed, the dimensions are very different. So, we want the input_dir in the glb cases to have these dimensions too.

it looks like global time series files are used as input for e3sm_to_cmip. The expected files should come from regrided files with lat lon dimensions

The only way I can think to do that would be to feed in the output of the ts_land_monthly task on the same years, rather than the pre-e3sm_to_cmip output of the ts_land_monthly_glb task.... That would then introduce a dependency: ts_land_monthly_glb would depend on ts_land_monthly. That seems a bit convoluted. Is that what we need?
I also argue that such a design would elevate #467/#496 in priority. The dependency chain should be "e3sm_to_cmip_glb (i.e., as its own task) depends on both ts_land_monthly and ts_land_monthly_glb" rather than "ts_land_monthly_glb (with e3sm_to_cmip) depends on ts_land_monthly."
@chengzhuzhang I feel like I'm not following your intended design here. Can you clarify further? Thanks!

That said, I think adding landfrac with a new extra_vars parameter (as in ts) would be a more robust solution, since it wouldn't require users to run climo on all variables. I suppose that could be addressed in a later issue/PR though.

I created #542 to track this. We can choose to do that as part of this PR or separately, if we're ok with always setting vars='' for running climo on land for a while.

I created #542 to track this. We can choose to do that as part of this PR or separately, if we're ok with always setting vars='' for running climo on land for a while.

Just to clarify, setting vars='' or vars= a list of variables that including LANDFRAC, both would work?

vars= a list of variables that including LANDFRAC,

No, I don't think so. I think it would need the extra_vars = 'landfrac', as is currently done in ts. That said, I haven't specifically tested vars as a list including landfrac.

@chengzhuzhang  I set vars = "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO,landfrac" for [climo] > [[ land_monthly_180x360_aave ]] and it does in fact complete successfully. I guess we don't need #542 after all.
Although, I am now puzzled why we needed extra_vars for the ts task. Looking at #120:

For example, if the variables are specified with a mix of wildcards and variable names like
ncclimo -v area,'^DF_', it will fail.
But ncclimo -v '^DF_' --var_xtr=area will work.

So it looks like the issues were wildcards?

it looks like global time series files are used as input for e3sm_to_cmip.

@chengzhuzhang Following up from this week's meeting, in which you said that atm and land should be more or less the same for glb:
I did a diff of the atm and land glb files and the difference that stood out to me was that land had ts_fmt = "cmip" whereas atm did not. I was under the mistaken assumption that land required the e3sm_to_cmip step whereas atm did not. I ran it with that option turned off, and the script succeeded.

@chengzhuzhang Given the issues in the above 2 comments have been resolved, I think we have proof that zppy can handle land post-processing through the NCO (climo and ts) steps.

land required the e3sm_to_cmip step whereas atm did not. I ran it with that option turned off, and the script succeeded.

The e3sm_to_cmip step is required by the ilamb task. ilamb can produce results on both atm and land lat-lon time series.

@chengzhuzhang Given the issues in the above 2 comments have been resolved, I think we have proof that zppy can handle land post-processing through the NCO (climo and ts) steps.

I think so. It would be useful to starting a configuration file to summarize findings from this issue discussion.

It would be useful to starting a configuration file to summarize findings from this issue discussion.

#545

Given the issues in the above 2 comments have been resolved, I think we have proof that zppy can handle land post-processing through the NCO (climo and ts) steps.

Therefore, I am closing this issue. As mentioned above, a sample land configuration can be found in #545.


ninePointCircle-RishabhR

This is a visualization of the nine-point circle for any generic triangle. It is a special triangle because it contains the midpoints of each side of the triangle, the points at the feet of the altitudes, and the midpoints located between these feet and the orthocenter. I am experienced in Java for app design and robotics and Python ML but this was my first time using JS for a project, and I have learned a little about using Processing but apart from that I have not created digital art before.

[x] I have read the steps to getting a blot
[x] I have read all the submission requirements and follow them
[x] I am submitting art that...

[x] is algorithmically generated (will meaningfully change each time the program is run)
[x] is your own work that is not copied from somewhere else
[x] is drawable through the editor(does not use outside tools)
[x] doesn't call Math.random() (See the documentation on randomness)
[x] is drawable on Blot itself (doesn't have lines overlap more than 5 times)

[x] Doesn't overlap lines more than 5 times
[x] Fits in the working area of 125x125mm
[x] Doesn't rely on colour to look good

If you used based your art on something else, used a tutorial, or remixed it from something else, please link it here:
https://en.wikipedia.org/wiki/Nine-point_circle
Lastly, link a picture of your art piece here:

Sorry about the mistake! It should be in the correct folder now.


Change from Audit Mode to Enforce Mode

How is the Change from Audit Mode to Enforce Mode to detected?  How long does it take?  Can the state be seen on the Domain Controller?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 799247b7-b3f3-807f-2183-1775f47500f9
Version Independent ID: 8c242698-eaa1-4132-497f-e6ac336e94f6
Content: Azure AD Password Protection operations and reporting - Azure Active Directory
Content Source: articles/active-directory/authentication/howto-password-ban-bad-on-premises-operations.md
Service: active-directory
Sub-service: authentication
GitHub Login: @MicrosoftGuyJFlo
Microsoft Alias: joflore

@alicain
Thanks for your feedback! We will investigate and update as appropriate.

@alicain States are only visible on the Azure portal.  For your other questions I am checking with the product team and get back to you at the earliest.

See the section linked below. It provides the information you are looking for.
https://docs.microsoft.com/azure/active-directory/authentication/howto-password-ban-bad-on-premises-monitor#dc-agent-admin-event-log

#please-close

Thanks @MicrosoftGuyJFlo
@alicain We will now proceed to close this thread. If there are further questions regarding this matter, please reopen it and we will gladly continue the discussion.


Can't successfully run seldon deployment with tensorflow image (wrong port)

Creating seldon deployment that is run using tensorflow/serving image.
Here is the yaml file of the job - https://pastebin.ubuntu.com/p/KpNyz8qDf9/
Essentially the job is instructing the manager to probe model container on port 9000 (which is not defined anywhere), while the container is listening on the port 8501 (API).
Consequently the manager never claims the model to be ready, though model container starts successfully.
Log output from the model container: https://pastebin.ubuntu.com/p/8PqmgQC3mV/
Pod description: https://pastebin.ubuntu.com/p/Qnmfj26Jmw/

Closing, as I think this is a dupe of #42, let me know if that's not the case, and I can reopen.


Add option to set split_words:true globally

Closes https://github.com/kelseyhightower/envconfig/issues/190

Any chance this will be merged? Adding split_words:true everywhere is kind of ridiculous.

Also @ndrewnee would you mind adding a MustProcessWithOptions func too?

Also @ndrewnee would you mind adding a MustProcessWithOptions func too?

I already added it here

Missed it. My bad!

@kelseyhightower any idea if this PR would be merged?


[BUG] Images with a name starting with "blog" is not found

Due to the following route, pictures with a name starting with blog will not be found in the /public/images/ folder.
get "/blog":
    createTFD()
    resp genMain(c, genBlogAllPages(c))

  get re"/blog/*.":
    createTFD()
    let blogid = getValue(db, sql"SELECT id FROM blog WHERE url = ?", c.req.path.replace("/blog/", ""))
    resp genPageBlog(c, blogid)

Fixed in commit d4be66bf5445a8449fc9c90935548ed6e25e77e3.