Skip to content

Update bash ex-scripts to be linter compliant #4151

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 18 commits into
base: develop
Choose a base branch
from
Open

Update bash ex-scripts to be linter compliant #4151

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
b0ee5dd
Update exscripts for shellcheck except unused variables [SC2034]
WalterKolczynski-NOAA Oct 7, 2025
f4e93b5
Apply shfmt to exscripts
WalterKolczynski-NOAA Oct 7, 2025
85357db
Remove unneeded APRUNCFP in anlcalc
WalterKolczynski-NOAA Oct 7, 2025
3f23e96
Export "unused" variables
WalterKolczynski-NOAA Oct 8, 2025
c658195
Fix errors introduced during formatting updates
WalterKolczynski-NOAA Oct 11, 2025
d39104f
Fix error in skew-t script
WalterKolczynski-NOAA Oct 11, 2025
51d49d3
Shfmt skew-T
WalterKolczynski-NOAA Oct 15, 2025
f1743c0
Merge develop into feature/shfmt_exscripts
WalterKolczynski-NOAA Oct 17, 2025
61419d3
Merge develop' into feature/shfmt_exscripts
WalterKolczynski-NOAA Oct 20, 2025
7799ca5
Merge branch 'develop' into feature/shfmt_exscripts
WalterKolczynski-NOAA Oct 21, 2025
cd4f1ea
Apply shfmt to analysis changes
WalterKolczynski-NOAA Oct 21, 2025
006ab6c
Merge branch 'develop' into feature/shfmt_exscripts
WalterKolczynski-NOAA Oct 22, 2025
407affd
Merge origin/develop into feature/shfmt_exscripts
WalterKolczynski-NOAA Nov 4, 2025
e19b80e
Merge develop into feature/shfmt_exscripts
WalterKolczynski-NOAA Nov 5, 2025
c7e91f2
Merge develop into feature/shfmt_exscripts
WalterKolczynski-NOAA Nov 10, 2025
f59183c
Merge develop into feature/shfmt_exscripts
WalterKolczynski-NOAA Nov 10, 2025
0cf2240
Make minor updates based on PR review
WalterKolczynski-NOAA Nov 13, 2025
3f56b21
Make missed correction based on review comment
WalterKolczynski-NOAA Nov 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
154 changes: 62 additions & 92 deletions scripts/exgdas_atmos_chgres_forenkf.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,10 @@ GDUMP=${GDUMP:-"gdas"}
# Derived base variables
# shellcheck disable=SC2153
GDATE=$(date --utc +%Y%m%d%H -d "${PDY} ${cyc} - ${assim_freq} hours")
export GDATE
BDATE=$(date --utc +%Y%m%d%H -d "${PDY} ${cyc} - 3 hours")
bPDY=${BDATE:0:8}
bcyc=${BDATE:8:2}
export bPDY=${BDATE:0:8}
export bcyc=${BDATE:8:2}

# Utilities
export CHGRP_CMD=${CHGRP_CMD:-"chgrp ${group_name:-rstprod}"}
Expand Down Expand Up @@ -73,111 +74,80 @@ DOHYBVAR=${DOHYBVAR:-"NO"}
lrun_subdirs=${lrun_subdirs:-".true."}
USE_CFP=${USE_CFP:-"NO"}
CFP_MP=${CFP_MP:-"NO"}
nm=""
if [ $CFP_MP = "YES" ]; then
nm=0
fi
Comment on lines -77 to -79
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA Nov 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this set at the JJob level?

Copy link
Contributor Author

@WalterKolczynski-NOAA WalterKolczynski-NOAA Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not used anymore with the switch to using run_mpmd.sh.

if [ $DOHYBVAR = "YES" ]; then
l_hyb_ens=.true.
export l4densvar=${l4densvar:-".false."}
export lwrite4danl=${lwrite4danl:-".false."}

if [[ "${DOHYBVAR}" == "YES" ]]; then
export l_hyb_ens=.true.
export l4densvar=${l4densvar:-".false."}
export lwrite4danl=${lwrite4danl:-".false."}
else
echo "DOHYBVAR != YES, this script will exit without regridding deterministic forecast"
exit 0
echo "DOHYBVAR != YES, this script will exit without regridding deterministic forecast"
exit 0
fi

################################################################################
################################################################################

# get resolution information
LONB_ENKF=${LONB_ENKF:-$($NCLEN $ATMFCST_ENSRES grid_xt)} # get LONB_ENKF
LATB_ENKF=${LATB_ENKF:-$($NCLEN $ATMFCST_ENSRES grid_yt)} # get LATB_ENFK
LEVS_ENKF=${LEVS_ENKF:-$($NCLEN $ATMFCST_ENSRES pfull)} # get LATB_ENFK
LONB_ENKF=${LONB_ENKF:-$(${NCLEN} "${ATMFCST_ENSRES}" grid_xt)} # get LONB_ENKF
LATB_ENKF=${LATB_ENKF:-$(${NCLEN} "${ATMFCST_ENSRES}" grid_yt)} # get LATB_ENFK
LEVS_ENKF=${LEVS_ENKF:-$(${NCLEN} "${ATMFCST_ENSRES}" pfull)} # get LATB_ENFK

##############################################################
# If analysis increment is written by GSI, regrid forecasts to increment resolution
if [ $DO_CALC_ANALYSIS == "YES" ]; then
$NLN $ATMF06 fcst.06
$NLN $ATMF06ENS fcst.ensres.06
$NLN $ATMFCST_ENSRES atmens_fcst
if [ $DOHYBVAR = "YES" -a $l4densvar = ".true." -a $lwrite4danl = ".true." ]; then
$NLN $ATMF03 fcst.03
$NLN $ATMF03ENS fcst.ensres.03
$NLN $ATMF04 fcst.04
$NLN $ATMF04ENS fcst.ensres.04
$NLN $ATMF05 fcst.05
$NLN $ATMF05ENS fcst.ensres.05
$NLN $ATMF07 fcst.07
$NLN $ATMF07ENS fcst.ensres.07
$NLN $ATMF08 fcst.08
$NLN $ATMF08ENS fcst.ensres.08
$NLN $ATMF09 fcst.09
$NLN $ATMF09ENS fcst.ensres.09
fi
export OMP_NUM_THREADS=$NTHREADS_CHGRES
SIGLEVEL=${SIGLEVEL:-${FIXgfs}/am/global_hyblev.l${LEVS_ENKF}.txt}

if [[ "${USE_CFP}" == "YES" ]]; then
rm -f "${DATA}/mp_chgres.sh"
fi

nfhrs=$(echo "${IAUFHRS_ENKF}" | sed 's/,/ /g')
for FHR in $nfhrs; do
echo "Regridding deterministic forecast for forecast hour $FHR"
rm -f chgres_nc_gauss0$FHR.nml
cat > chgres_nc_gauss0$FHR.nml << EOF
if [[ ${DO_CALC_ANALYSIS} == "YES" ]]; then
${NLN} "${ATMF06}" fcst.06
${NLN} "${ATMF06ENS}" fcst.ensres.06
${NLN} "${ATMFCST_ENSRES}" atmens_fcst
if [[ "${DOHYBVAR}" == "YES" ]] && [[ "${l4densvar}" == ".true." ]] && [[ "${lwrite4danl}" == ".true." ]]; then
${NLN} "${ATMF03}" fcst.03
${NLN} "${ATMF03ENS}" fcst.ensres.03
${NLN} "${ATMF04}" fcst.04
${NLN} "${ATMF04ENS}" fcst.ensres.04
${NLN} "${ATMF05}" fcst.05
${NLN} "${ATMF05ENS}" fcst.ensres.05
${NLN} "${ATMF07}" fcst.07
${NLN} "${ATMF07ENS}" fcst.ensres.07
${NLN} "${ATMF08}" fcst.08
${NLN} "${ATMF08ENS}" fcst.ensres.08
${NLN} "${ATMF09}" fcst.09
${NLN} "${ATMF09ENS}" fcst.ensres.09
fi
export OMP_NUM_THREADS=${NTHREADS_CHGRES}
SIGLEVEL=${SIGLEVEL:-${FIXgfs}/am/global_hyblev.l${LEVS_ENKF}.txt}

if [[ "${USE_CFP}" == "YES" ]]; then
rm -f "${DATA}/mp_chgres.sh"
fi

nfhrs="${IAUFHRS_ENKF//,/ }"
for FHR in ${nfhrs}; do
echo "Regridding deterministic forecast for forecast hour ${FHR}"
rm -f "chgres_nc_gauss0${FHR}.nml"
cat > "chgres_nc_gauss0${FHR}.nml" << EOF
&chgres_setup
i_output=$LONB_ENKF
j_output=$LATB_ENKF
input_file="fcst.0$FHR"
output_file="fcst.ensres.0$FHR"
terrain_file="atmens_fcst"
ref_file="atmens_fcst"
i_output=${LONB_ENKF}
j_output=${LATB_ENKF}
input_file="fcst.0${FHR}"
output_file="fcst.ensres.0${FHR}"
terrain_file="atmens_fcst"
ref_file="atmens_fcst"
/
EOF
if [[ $USE_CFP == "YES" ]]; then
echo "$nm $APRUN_CHGRES $CHGRESNCEXEC chgres_nc_gauss0$FHR.nml" | tee -a $DATA/mp_chgres.sh
if [[ ${CFP_MP:-"NO"} = "YES" ]]; then
nm=$((nm+1))
fi
else

export pgm=$CHGRESNCEXEC
. prep_step

${APRUN_CHGRES} "${CHGRESNCEXEC}" "chgres_nc_gauss0${FHR}.nml" && true
export err=$?
if [[ ${err} -ne 0 ]]; then
err_exit
fi
fi
done

if [[ ${USE_CFP} == "YES" ]]; then
chmod 755 ${DATA}/mp_chgres.sh
ncmd=$(wc -l < "${DATA}/mp_chgres.sh")
if [[ ${ncmd} -gt 0 ]]; then
ncmd_max=$((ncmd < max_tasks_per_node ? ncmd : max_tasks_per_node))
APRUNCFP_CHGRES=$(eval echo "${APRUNCFP}")

export pgm=${CHGRESNCEXEC}
source prep_step

${APRUNCFP_CHGRES} "${DATA}/mp_chgres.sh" && true
export err=$?
if [[ ${err} -ne 0 ]]; then
err_exit
fi
fi
fi

echo "${APRUN_CHGRES} ${CHGRESNCEXEC} chgres_nc_gauss0${FHR}.nml" | tee -a "${DATA}/mp_chgres.sh"

done

# Run with MPMD
"${USHgfs}/run_mpmd.sh" "${DATA}/mp_chgres.sh" && true
export err=$?
if [[ ${err} -ne 0 ]]; then
err_exit
fi
else
echo "DO_CALC_ANALYSIS != YES, doing nothing"
echo "DO_CALC_ANALYSIS != YES, doing nothing"
fi

cd "${pwd}" || exit

################################################################################
# Postprocessing
cd $pwd

exit $err
exit "${err}"
8 changes: 4 additions & 4 deletions scripts/exgdas_atmos_gempak_gif_ncdc.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,17 +18,17 @@ if [[ ${MODEL} == GDAS ]]; then
export fhr3
for fhr3 in ${fcsthrs}; do
gempak_file="${COMIN_ATMOS_GEMPAK_1p00}/${RUN}_1p00_${PDY}${cyc}f${fhr3}"
if ! wait_for_file "${gempak_file}" "${sleep_interval}" "${max_tries}" ; then
if ! wait_for_file "${gempak_file}" "${sleep_interval}" "${max_tries}"; then
export err=10
if [[ ${err} -ne 0 ]]; then
err_exit "${gempak_file} not found after ${max_tries} iterations"
err_exit "${gempak_file} not found after ${max_tries} iterations"
fi
fi

if [[ ! -f "${gempak_file}" ]]; then
export err=1
if [[ ${err} -ne 0 ]]; then
err_exit "Could not copy ${gempak_file}"
err_exit "Could not copy ${gempak_file}"
fi
fi

Expand All @@ -37,7 +37,7 @@ if [[ ${MODEL} == GDAS ]]; then
"${HOMEgfs}/gempak/ush/gempak_${RUN}_f${fhr3}_gif.sh" && true
export err=$?
if [[ ${err} -ne 0 ]]; then
err_exit
err_exit
fi
done
fi
Expand Down
22 changes: 10 additions & 12 deletions scripts/exgdas_atmos_nawips.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,11 @@ cd "${DATA_RUN}" || exit 1
source "${USHgfs}/product_functions.sh"

for table in g2varswmo2.tbl g2vcrdwmo2.tbl g2varsncep1.tbl g2vcrdncep1.tbl; do
source_table="${HOMEgfs}/gempak/fix/${table}"
if [[ ! -f "${source_table}" ]]; then
err_exit "${table} is missing"
fi
cpreq "${source_table}" "${table}"
source_table="${HOMEgfs}/gempak/fix/${table}"
if [[ ! -f "${source_table}" ]]; then
err_exit "${table} is missing"
fi
cpreq "${source_table}" "${table}"
done

NAGRIB="${GEMEXE}/nagrib2"
Expand All @@ -37,15 +37,13 @@ proj=
output=T
pdsext=no



GEMGRD="${RUN}_${grid}_${PDY}${cyc}f${fhr3}"
source_dirvar="COMOUT_ATMOS_GRIB_${grid}"
export GRIBIN="${!source_dirvar}/${RUN}.${cycle}.pres_a.${grid}.f${fhr3}.grib2"
GRIBIN_chk="${GRIBIN}.idx"

if [[ ! -r "${GRIBIN_chk}" ]]; then
err_exit "GRIB index file ${GRIBIN_chk} not found!"
err_exit "GRIB index file ${GRIBIN_chk} not found!"
fi

cpreq "${GRIBIN}" "grib${fhr3}"
Expand Down Expand Up @@ -73,13 +71,13 @@ EOF

export err=$?
if [[ ${err} -ne 0 ]]; then
err_exit "${NAGRIB} failed to create ${GEMGRD}!"
err_exit "${NAGRIB} failed to create ${GEMGRD}!"
fi

cpfs "${GEMGRD}" "${destination}/${GEMGRD}"
if [[ "${SENDDBN}" == "YES" ]] ; then
"${DBNROOT}/bin/dbn_alert" MODEL "${DBN_ALERT_TYPE}" "${job}" \
"${destination}/${GEMGRD}"
if [[ "${SENDDBN}" == "YES" ]]; then
"${DBNROOT}/bin/dbn_alert" MODEL "${DBN_ALERT_TYPE}" "${job}" \
"${destination}/${GEMGRD}"
fi

"${GEMEXE}/gpend"
Expand Down
53 changes: 23 additions & 30 deletions scripts/exgdas_atmos_verfozn.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,42 +9,35 @@
################################################################################
export err=0

data_available=0
if [[ -s "${oznstat}" ]]; then
#------------------------------------------------------------------
# Copy data files file to local data directory.
# Untar oznstat file.
#------------------------------------------------------------------

if [[ -s ${oznstat} ]]; then
data_available=1
cpreq "${oznstat}" "./oznstat.${PDY}${cyc}"

#------------------------------------------------------------------
# Copy data files file to local data directory.
# Untar oznstat file.
#------------------------------------------------------------------
tar -xvf "oznstat.${PDY}${cyc}"
rm -f "oznstat.${PDY}${cyc}"

cpreq "${oznstat}" "./oznstat.${PDY}${cyc}"
netcdf=0
for filenc4 in diag*nc4.gz; do
netcdf=1
file=$(echo "${filenc4}" | cut -d'.' -f1-2).gz
mv "${filenc4}" "${file}"
done
Comment on lines +24 to +28
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA Nov 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs a file existence check. If there are no diag*nc4.gz files, then filenc4 will be assigned the literal diag*nc4.gz.

Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA Nov 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I think this should be reverted to the previous logic. There are some variables that could be removed (data_available, for instance), but otherwise I don't think this script should change logic.

Copy link
Contributor Author

@WalterKolczynski-NOAA WalterKolczynski-NOAA Nov 13, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shellcheck doesn't like looping over ls: https://www.shellcheck.net/wiki/SC2045

Recommend we turn on shopt -s nullglob (as noted on that page) anyway for all our code to prevent * being treated as literal anywhere.


tar -xvf "oznstat.${PDY}${cyc}"
rm -f "oznstat.${PDY}${cyc}"
export OZNMON_NETCDF=${netcdf}

netcdf=0
count=$(ls diag* | grep ".nc4" | wc -l)
if [ "${count}" -gt 0 ] ; then
netcdf=1
for filenc4 in $(ls diag*nc4.gz); do
file=$(echo "${filenc4}" | cut -d'.' -f1-2).gz
mv "${filenc4}" "${file}"
done
fi

export OZNMON_NETCDF=${netcdf}

"${USHgfs}/ozn_xtrct.sh" && true
export err=$?
if [[ ${err} -ne 0 ]]; then
err_exit "ozn_xtrct.sh failed!"
fi
"${USHgfs}/ozn_xtrct.sh" && true
export err=$?
if [[ ${err} -ne 0 ]]; then
err_exit "ozn_xtrct.sh failed!"
fi

else
# oznstat file not found
export err=1
err_exit "${oznstat} does not exist!"
# oznstat file not found
export err=1
err_exit "${oznstat} does not exist!"
fi
exit 0
Loading