WRF-Chem Simulation Stalling

Any issues with running WRF Chemistry

WRF-Chem Simulation Stalling

Postby adprest » Sat May 14, 2016 10:32 pm

Dear Users,

I have successfully run a 3-domain (27-9-3 km) WRF simulation for a Tropical Cyclone over a 10 day period on my university's computing cluster. However, when I try running just my outermost domain (27 km) coupled with the chemistry module, my WRF-Chem simulation stalls only 6-12 hours into the run. I receive no error and cannot figure out why this keeps occurring. Here are the last few lines of my rsl.out.0000 file:

Timing for main: time 1991-09-16_20:49:00 on domain 1: 9.20224 elapsed seconds
Timing for main: time 1991-09-16_20:50:00 on domain 1: 9.13931 elapsed seconds
Timing for main: time 1991-09-16_20:51:00 on domain 1: 9.08160 elapsed seconds
Timing for main: time 1991-09-16_20:52:00 on domain 1: 9.30161 elapsed seconds
Timing for main: time 1991-09-16_20:53:00 on domain 1: 9.56595 elapsed seconds
Timing for main: time 1991-09-16_20:54:00 on domain 1: 9.59824 elapsed seconds
Timing for main: time 1991-09-16_20:55:00 on domain 1: 9.03539 elapsed seconds
Timing for main: time 1991-09-16_20:56:00 on domain 1: 9.08624 elapsed seconds
Timing for main: time 1991-09-16_20:57:00 on domain 1: 9.27863 elapsed seconds
Timing for main: time 1991-09-16_20:58:00 on domain 1: 9.27565 elapsed seconds
Timing for main: time 1991-09-16_20:59:00 on domain 1: 9.10902 elapsed seconds
calculate MEGAN emissions at ktau, gmtp, tmidh = 540 20.00000 20.99167
photolysis_driver: called for domain 1
Timing for main: time 1991-09-16_21:00:00 on domain 1: 10.99391 elapsed seconds
Timing for main: time 1991-09-16_21:01:00 on domain 1: 19.07616 elapsed seconds


Has anyone had this issue before? I am able to view my wrfout file using the Ncview tool and all of the anthropogenic and biogenic variables appear in the wrfout file. I look forward to hearing any recommendations/solutions you might have!

Thanks,
Ari
adprest
 
Posts: 18
Joined: Tue Mar 04, 2014 1:51 pm
Location: Florida State University

Re: WRF-Chem Simulation Stalling

Postby adprest » Sun May 22, 2016 6:27 pm

Using the debug = 100 option, my WRF-Chem simulation (WRF v3.7) seems to stall right after the following lines in my rsl.out/error files:

d01 1991-09-17_03:31:00 calling inc/HALO_EM_PHYS_PBL_inline.inc
d01 1991-09-17_03:31:00 calling inc/HALO_EM_FDDA_inline.inc
d01 1991-09-17_03:31:00 calling inc/HALO_EM_PHYS_DIFFUSION_inline.inc


Depending on the number of processors I request, the simulation will stall at different time periods into the WRF-Chem simulation. The only "errors" I receive throughout my entire rsl.out/error files are below:

Timing for Writing wrfout_d01_1991-09-17_00:00:00 for domain 1: 29.77804 elapsed seconds
d01 1991-09-17_00:00:00 NetCDF error: NetCDF: Attribute not found
d01 1991-09-17_00:00:00 NetCDF error in ext_ncd_get_dom_ti.code REAL, line 83 Element GMT
d01 1991-09-17_00:00:00 NetCDF error: NetCDF: Attribute not found
d01 1991-09-17_00:00:00 NetCDF error in ext_ncd_get_dom_ti.code INTEGER, line 83 Element JULYR
d01 1991-09-17_00:00:00 NetCDF error: NetCDF: Attribute not found
d01 1991-09-17_00:00:00 NetCDF error in ext_ncd_get_dom_ti.code INTEGER, line 83 Element JULDAY


Has anyone come across this before? I would greatly appreciate any type of advice or recommendation.

Thanks,
Ari
adprest
 
Posts: 18
Joined: Tue Mar 04, 2014 1:51 pm
Location: Florida State University

Re: WRF-Chem Simulation Stalling

Postby easy » Sun Jun 26, 2016 9:20 pm

Hi, Ari,

Have you solved this problem. My simulation will stall earlier at photolysis_driver: called for domain 1.
If you have solved it, could you please post the method here. Thanks.

easy
easy
 
Posts: 10
Joined: Sun Jun 07, 2015 10:26 pm

Re: WRF-Chem Simulation Stalling

Postby adprest » Wed Jun 29, 2016 10:31 pm

I seem to have mostly solved it by using newer (2015-16) intel processors at my university's computing center. Furthermore, I have implemented a new "exclusive" command that gives me exclusive privileges to each node. That is, I don't share the nodes with any other user. This seems to have fixed the problem. I did just have a "stall" issue the other week but that seems to be an exception. All of my other simulations have been able to run to completion.

Hope this helps!

Ari
adprest
 
Posts: 18
Joined: Tue Mar 04, 2014 1:51 pm
Location: Florida State University


Return to Runtime Problems

Who is online

Users browsing this forum: No registered users and 1 guest