WRF 4.2.1 crashes at CALL cldfra1

[msullivan-cornell]

I'm trying to downscale GFS .25deg to 10km within the default CONUS projection in the default namelists (740x610). I am getting a segmentation fault a few minutes into the run. It seems to do several time steps but then crashes at the same point in the model no matter what timestep I have set (tried between 5 seconds and 30 seconds). In this case, it is crashing at exactly the 30 minute time step, which may or may not be relevant, at CALL cldfra1.

I'm using the CONUS physics suite

Some versioning details:

WRF: 4.2.1 (gfortran, dmpar)
WPS: 4.2 (gfortran, serial)
netcdf-c 4.7.4
netcdf-fortran 4.5.3
hdf5: 1.12.0
mpich: 3.3 (included with debian buster)

I set debug_level to 50 and the following is a snippet of rsl.out.0000 where it shows the previous time step and the time step in which it crashes. This is where it crashes:

Code: Select all

d01 2020-08-06_12:30:00 Top of Radiation Driver
d01 2020-08-06_12:30:00 CALL cldfra1

this particular run was with 16 cores, but i've tried running even serially and see the same result.

Code: Select all

Timing for main: time 2020-08-06_12:29:30 on domain   1:    4.12892 elapsed seconds
d01 2020-08-06_12:29:30  grid spacing, dt, time_step_sound=   10000.0000       30.0000000               4
d01 2020-08-06_12:29:30 calling inc/HALO_EM_MOIST_OLD_E_7_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_MOIST_OLD_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_A_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_A_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_PHYS_A_inline.inc
d01 2020-08-06_12:29:30 Top of Radiation Driver
d01 2020-08-06_12:29:30 calling inc/HALO_PWP_inline.inc
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
 Flerchinger USEd in NEW version. Iterations=          10
d01 2020-08-06_12:29:30 calling inc/HALO_EM_FDDA_SFC_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_TKE_C_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_A1_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_HELICITY_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_TKE_D_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_TKE_E_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_PHY_BC_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_CHEM_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_PHYS_PBL_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_PHYS_CU_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_PHYS_DIFFUSION_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_TKE_5_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_B_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_B_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_B_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_C_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_C2_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_B3_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_D_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_EM_DA_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_D2_5_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_D_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_MOIST2_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_CHEM2_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_TRACER2_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_SCALAR2_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_TKE_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_TKE_5_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_MOIST_E_5_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_SCALAR_E_5_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_A_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_A_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_B_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_B_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_B_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_C_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_C2_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_B3_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_B_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_C_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_C2_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_B3_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_D_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_EM_DA_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_D2_5_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_D_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_MOIST2_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_CHEM2_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_TRACER2_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_SCALAR2_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_TKE_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_TKE_7_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_MOIST_E_7_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_SCALAR_E_7_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_A_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_A_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_B_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_B_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_B_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_C_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_C2_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_B3_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_B_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_C_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_C2_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_B3_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_B_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_C_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_C2_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_B3_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_B_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_C_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_C2_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_B3_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_MOIST_OLD_E_7_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_MOIST_OLD_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_SCALAR_OLD_E_7_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_SCALAR_OLD_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_CHEM_OLD_E_7_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_CHEM_OLD_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_TRACER_OLD_E_7_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_TRACER_OLD_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_D_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_EM_DA_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_SBM_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_THETAM_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_EM_THETAM_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_D3_5_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_D3_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_MOIST_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_CHEM_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_TRACER_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_SCALAR_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_C_inline.inc
d01 2020-08-06_12:29:30 calling inc/PERIOD_BDY_EM_E_inline.inc
d01 2020-08-06_12:29:30  call set_w_surface
d01 2020-08-06_12:29:30 calling inc/HALO_EM_PHYS_W_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_PHYS_HCW_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_E_5_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_MOIST_E_5_inline.inc
d01 2020-08-06_12:29:30 calling inc/HALO_EM_SCALAR_E_5_inline.inc
Timing for main: time 2020-08-06_12:30:00 on domain   1:    4.21246 elapsed seconds
d01 2020-08-06_12:30:00  grid spacing, dt, time_step_sound=   10000.0000       30.0000000               4
d01 2020-08-06_12:30:00 calling inc/HALO_EM_MOIST_OLD_E_7_inline.inc
d01 2020-08-06_12:30:00 calling inc/PERIOD_BDY_EM_MOIST_OLD_inline.inc
d01 2020-08-06_12:30:00 calling inc/HALO_EM_A_inline.inc
d01 2020-08-06_12:30:00 calling inc/PERIOD_BDY_EM_A_inline.inc
d01 2020-08-06_12:30:00 calling inc/HALO_EM_PHYS_A_inline.inc
d01 2020-08-06_12:30:00 Top of Radiation Driver
d01 2020-08-06_12:30:00 CALL cldfra1


namelist.input

Code: Select all

 &time_control
 run_days                            = 0,
 run_hours                           = 0,
 run_minutes                         = 0,
 run_seconds                         = 0,
 start_year                          = 2020, 2000, 2000,
 start_month                         = 08,   01,   01,
 start_day                           = 06,   24,   24,
 start_hour                          = 12,   12,   12,
 end_year                            = 2020, 2000, 2000,
 end_month                           = 08,   01,   01,
 end_day                             = 07,   25,   25,
 end_hour                            = 12,   12,   12,
 interval_seconds                    = 3600
 input_from_file                     = .true.,.true.,.true.,
 history_interval                    = 60,  60,   60,
 frames_per_outfile                  = 1000, 1000, 1000,
 restart                             = .false.,
 restart_interval                    = 7200,
 io_form_history                     = 2
 io_form_restart                     = 2
 io_form_input                       = 2
 io_form_boundary                    = 2
 debug_level                         = 50
 /

 &domains
 time_step                           = 30,
 time_step_fract_num                 = 0,
 time_step_fract_den                 = 1,
 max_dom                             = 1,
 e_we                                = 740,    112,   94,
 e_sn                                = 610,    97,    91,
 e_vert                              = 34,    33,    33,
 p_top_requested                     = 5000,
 num_metgrid_levels                  = 34,
 num_metgrid_soil_levels             = 4,
 dx                                  = 10000,
 dy                                  = 10000,
 grid_id                             = 1,     2,     3,
 parent_id                           = 0,     1,     2,
 i_parent_start                      = 1,     31,    30,
 j_parent_start                      = 1,     17,    30,
 parent_grid_ratio                   = 1,     3,     3,
 parent_time_step_ratio              = 1,     3,     3,
 feedback                            = 1,
 smooth_option                       = 0
 /

 &physics
 physics_suite                       = 'CONUS'
 mp_physics                          = -1,    -1,    -1,
 cu_physics                          = -1,    -1,     0,
 ra_lw_physics                       = -1,    -1,    -1,
 ra_sw_physics                       = -1,    -1,    -1,
 bl_pbl_physics                      = -1,    -1,    -1,
 sf_sfclay_physics                   = -1,    -1,    -1,
 sf_surface_physics                  = -1,    -1,    -1,
 radt                                = 30,    30,    30,
 bldt                                = 0,     0,     0,
 cudt                                = 5,     5,     5,
 icloud                              = 1,
 num_land_cat                        = 21,
 sf_urban_physics                    = 0,     0,     0,
 /

 &fdda
 /

 &dynamics
 hybrid_opt                          = 2,
 w_damping                           = 0,
 diff_opt                            = 1,      1,      1,
 km_opt                              = 4,      4,      4,
 diff_6th_opt                        = 0,      0,      0,
 diff_6th_factor                     = 0.12,   0.12,   0.12,
 base_temp                           = 290.
 damp_opt                            = 3,
 zdamp                               = 5000.,  5000.,  5000.,
 dampcoef                            = 0.2,    0.2,    0.2
 khdif                               = 0,      0,      0,
 kvdif                               = 0,      0,      0,
 non_hydrostatic                     = .true., .true., .true.,
 moist_adv_opt                       = 1,      1,      1,     
 scalar_adv_opt                      = 1,      1,      1,     
 gwd_opt                             = 1,      1,      0,
 /

 &bdy_control
 spec_bdy_width                      = 5,
 specified                           = .true.
 /

 &grib2
 /

 &namelist_quilt
 nio_tasks_per_group = 0,
 nio_groups = 1,
 /


namelist.wps

Code: Select all

&share
 wrf_core = 'ARW',
 max_dom = 1,
 start_date = '2020-08-06_12:00:00','2006-08-16_12:00:00',
 end_date   = '2020-08-07_12:00:00','2006-08-16_12:00:00',
 interval_seconds = 3600,
 io_form_geogrid = 2,
/

&geogrid
 parent_id         =   1,   1,
 parent_grid_ratio =   1,   3,
 i_parent_start    =   1,  31,
 j_parent_start    =   1,  17,
 e_we              =  740, 112,
 e_sn              =  610,  97,
 geog_data_res = 'default','default',
 dx = 10000,
 dy = 10000,
 map_proj = 'lambert',
 ref_lat   =  45.49,
 ref_lon   = -89.46,
 truelat1  =  30.0,
 truelat2  =  60.0,
 stand_lon = -76.46,
 geog_data_path = '/root/local/WPS_GEOG/'
/

&ungrib
 out_format = 'WPS',
 prefix = 'FILE',
/

&metgrid
 fg_name = 'FILE'
 io_form_metgrid = 2,
/


[peter]

I am using WRF 4.2 and I am having exactly the same problem.

[msullivan-cornell]

Glad to hear someone else at least sees this issue. I also tried running the same domain with WRF 4.1.5 and 4.0.3 and get the exact same behavior.

WRF seems to do an entire core dump to disk (like ~25GB) when this error happens, don't know if that would be useful but I could provide it to someone.

Any insight or ideas to try would be much appreciated.

[msullivan-cornell]

I also just upped the debug_level to 300 and got one additional message, it seems to get past CALL cldfra1 to CALL rrtmg_lw

Code: Select all

Timing for main: time 2020-08-06_12:30:00 on domain   1:    2.83465 elapsed seconds
d01 2020-08-06_12:30:00 in med_latbound_in
d01 2020-08-06_12:30:00 module_integrate: calling solve interface
d01 2020-08-06_12:30:00  grid spacing, dt, time_step_sound=   10000.0000       30.0000000               4
d01 2020-08-06_12:30:00 calling inc/HALO_EM_MOIST_OLD_E_7_inline.inc
d01 2020-08-06_12:30:00 calling inc/PERIOD_BDY_EM_MOIST_OLD_inline.inc
d01 2020-08-06_12:30:00  call rk_step_prep
d01 2020-08-06_12:30:00 calling inc/HALO_EM_A_inline.inc
d01 2020-08-06_12:30:00 calling inc/PERIOD_BDY_EM_A_inline.inc
d01 2020-08-06_12:30:00  call rk_phys_bc_dry_1
d01 2020-08-06_12:30:00  call init_zero_tendency
d01 2020-08-06_12:30:00 calling inc/HALO_EM_PHYS_A_inline.inc
d01 2020-08-06_12:30:00  call phy_prep
d01 2020-08-06_12:30:00  DEBUG wrf_timetoa():  returning with str = [2020-08-06_12:30:00]
d01 2020-08-06_12:30:00  call radiation_driver
d01 2020-08-06_12:30:00 Top of Radiation Driver
d01 2020-08-06_12:30:00 CALL cldfra1
d01 2020-08-06_12:30:00 CALL rrtmg_lw

Any thoughts?

[raj123456]

Dear all,
I am also facing the same issue with WRF 4.0.0 and CONUS configuration.
The model is crashing at exactly 30 minutes.
If any of you found the solution or suggestions please let me know.

Thank you.

[ipercy406]

Hi,

I know this reply is very late to the question. But I happened to have the same problem running WRF4.2. And the code stops at CALL cldfra1 or any other radiation related lines, which giving no error message on rsl.error.

I started to run 2 nested domain (9km 3km) and use the 'CONUS' suite physics, the code runs perfectly well. But when I just change the res to 5km and 5/3km, the issue happens. The code crash at 30min. I changed the radt from 30 to 5, the model will crash at 5min. So I guess the problem is related to radiation physics.

The solution is to change the ra_lw_physics and ra_sw_physics from -1 (the default value from CONUS suite is 4) to 3.

This problem bothered me so much lately. So just posted here to have a record.

Hope this will help someone in the future.