External (Ruby) Workflow Manager

Questions or comments about the WRF Portal

External (Ruby) Workflow Manager

Postby KevinVW » Tue Oct 14, 2008 10:28 am

Hi,

I'd like to install and run the External Workflow Manager on our cluster so that my external users can submit jobs and run WRF/WPS on our clusters from their desktop. Is there any documentation on how to configure and run the External Workflow Manger? We're using Torque rather than SGE or LSF, but I believe I can simply write my own "pbsbatchsystem.rb" script to handle it. Or maybe someone already has? Any guidance would be appreciated.

Thanks,

Kevin
KevinVW
 
Posts: 4
Joined: Tue Oct 14, 2008 10:23 am

Re: External (Ruby) Workflow Manager

Postby jeffssmith1 » Tue Oct 14, 2008 12:20 pm

Do you want to use the external workflow manager with WRF Portal? If so, you just need to download the zip file at the bottom of this page:
http://wrfportal.org/WRFPortal.html

Download the file, unzip in a directory, and change the .rb files to executable (chmod +x *.rb).

By the way, for more immediate response, you can use our web form:
http://wrfportal.org/RequestInfoOrBugReport.html

Best regards,
Jeff Smith

Earth System Research Lab / NOAA
Boulder, Colorado
jeffssmith1
 
Posts: 25
Joined: Tue Apr 22, 2008 5:26 pm

Re: External (Ruby) Workflow Manager

Postby KevinVW » Tue Oct 14, 2008 1:17 pm

Yes, I want to use the external workflow manager with the WRF Portal. I've downloaded it, unziped, and chmod'd it. Now what? Is there any documentation on it?

Thanks,

Kevin
KevinVW
 
Posts: 4
Joined: Tue Oct 14, 2008 10:23 am

Re: External (Ruby) Workflow Manager

Postby jeffssmith1 » Tue Oct 14, 2008 1:33 pm

Since you are using it with WRF Portal, all you need to do is set the path to the external workflow manager from within Portal Wizard. Here is a short video tutorial on how to do that:
http://wrfportal.org/tutorial-flash/tut ... izard.html

When you run a workflow, you'll have the option (from the drop down combo box) of selecting the "Internal" or "External" workflow manager.

Regarding documentation, the author of the external workflow manager hasn't created it yet. I did, however, just add a little more information to the bottom of this page:
http://wrfportal.org/WRFPortal.html

Best regards,
Jeff Smith

Earth System Research Lab / NOAA
Boulder, Colorado
jeffssmith1
 
Posts: 25
Joined: Tue Apr 22, 2008 5:26 pm

Re: External (Ruby) Workflow Manager

Postby KevinVW » Tue Oct 14, 2008 1:57 pm

OK. I guess it was easier than I thought. It's going to take me some time to setup my cluster to do this.

Next questions:

Can the Portal be configured to use ssh key-pair authentication, or is password the only option?

Does the Portal use ssh to connect to the workflow manager or does it use some other TCP connection?

Thanks again for your help.
KevinVW
 
Posts: 4
Joined: Tue Oct 14, 2008 10:23 am

Re: External (Ruby) Workflow Manager

Postby jeffssmith1 » Tue Oct 14, 2008 2:20 pm

No, the portal requires the SSH password to be typed in.

The portal can connect to the workflow manager via SSH, or, if the workflow manager and WRF are installed on the local machine, it can just use a local connection in that case.

Best regards,
Jeff Smith

Earth System Research Lab / NOAA
Boulder, Colorado
jeffssmith1
 
Posts: 25
Joined: Tue Apr 22, 2008 5:26 pm

Re: External (Ruby) Workflow Manager

Postby KevinVW » Thu Oct 16, 2008 2:30 pm

So, I'm guessing that I have to run 'workflowmgr.rb' as a daemon with --loop. The problem I have now is that I don't have an XML workflow description file. How do I generate an initial file? Or am I completely on the wrong track here?
KevinVW
 
Posts: 4
Joined: Tue Oct 14, 2008 10:23 am

Re: External (Ruby) Workflow Manager

Postby jeffssmith1 » Thu Oct 16, 2008 2:38 pm

WRF Portal automatically creates the XML workflow description file for you. As far as you're concerned, all you need to do is select "External Workflow Manager" and then WRF Portal just uses it when you run your work flow.

Best regards,
Jeff Smith

Earth System Research Lab / NOAA
Boulder, Colorado
jeffssmith1
 
Posts: 25
Joined: Tue Apr 22, 2008 5:26 pm

Re: External (Ruby) Workflow Manager

Postby belgaroo » Thu Jun 04, 2009 4:51 am

I am running Torque also and would greatly appreciate a copy of a working Ruby script to port the external workflow manager i.e. the 'pbsbatchsystem.rb' KevinVW mentioned.

Much appreciated.

Brad Evans
School of Environmental Science
MURDOCH UNIVERSITY AUSTRALIA
belgaroo
 
Posts: 2
Joined: Thu Jun 04, 2009 4:48 am


Return to WRF Portal (GUI for running WRF)

Who is online

Users browsing this forum: No registered users and 1 guest