At this point we wish to wait for all submitted tasks to complete. So long as the queue is not empty, we continue to call. Wait waiting for the result of a task we submitted. Here we call. Wait(5) which takes a timeout argument. The call to wait will return a finished task which allows us to analyze the return_status or output.
M - the best Place to find live
Setting the port to 0 instructs Work queue to pick an arbitrary port to bind. We create a story task which takes a shell command argument. In the first run, we will have the command: t task python 1 file.00 file.01 each task usually depends on a number of files to run. These resume include the executable and any input files. Here we specify the executable and its input file infile. Notice that we specify both and infile twice when calling specify_file. The first argument is the name of the file on the master and the second argument is the name of the file we want created on the worker. Usually these filenames will be the same as in this example. This the output file, outfile, we want transferred back to the master. At this point we have finished the description of our task and it is ready to be submitted to work queue workers. Bmit submits this task.
Txt" outfile "file.0.2x" i command "python d s s" (i, infile, outfile) t task(command) work_queue_input). Specify_file(infile, infile, work_queue_input). Specify_file(outfile, outfile, work_queue_output) taskid bmit(T) print "done." print "Waiting for tasks to complete." while not. Wait(5) if T: print "task (id d) complete: s (return code d mmand, turn_status) print "done." Here we load the work_queue python binding. Python will look in pythonpath which is set when you setup your environment. This instantiates a workqueue master to which you may submit work. We give this master a project name of fg-tutorial and turn on catalog mode. This allows workers to locate the hostname and port our master is running on dynamically via the catalog server.
The output includes various data on the tasks and workers the master is managing. Here we see there are 4 tasks currently but no workers are connected, yet. It is simple to write a program in C/Perl/Python (or any language with appropriate work queue bindings) which can generate tasks to be queued for Work queue. In this example, we will create 100 simulation tasks using the same code. We will continue to use torque for this example. Setup mkdir /cctools-tutorial/wq-100 cd /cctools-tutorial/wq-100 wget wget wget Analysis we will analyze the code for this example to prepare you for the following section which is more advanced.!/usr/bin/env python from work_queue import * import sys try: q workqueue(port 0, name "fg-tutorial catalog retrolisthesis True) except: print "could not instantiate work queue master" sys. Exit(1) print "Listening on port.". Port print "Submitting 100 simulation tasks." where for i in range(0, 100 infile "input.
When the tasks are finished, the worker should quit due to the 5 second timeout. Running with Work queue workers on Torque our final goal is to setup Work queue workers on Torque compute nodes. Here we submit the worker tasks using the torque_submit_workers executable. We still use the catalog server and a project name so workers can find the master. Makeflow -c torque_submit_workers -a -n fg-tutorial -t 120 2 makeflow -t wq -a -n fg-tutorial -d all -o bug /dev/null 2 /dev/null while makeflow is running, you can look at the current Work queue projects. work_queue_status project name port waiting busy complete workers. Tacc work_queue_status is a useful tool for checking on your various Work queue projects.
Assignment, calendar - 4teachers
Work queue excels at handling low latency and short turn-around time jobs. So, for food the output of the simulation, we can confirm that the job was run on another host: cat d running on host uster Starting 2012 30 Jul 13:18:18.0. Finished 2012 30 Jul 13:18:23 Here we see that the worker ran on node uster. It took 5 seconds to complete. Running with Work queue we will start makeflow which will setup a work queue master on an arbitrary port using -p. Next, we'll examine the debug output saved to bug to see what port it's listening on so we can start a worker. Makeflow -c makeflow -t wq -p 0 -d all -o bug /dev/null 2 /dev/null we can now inspect the output to see the port (in bold cat bug 2012/07/30 albert 12:18:54.30 10131 makeflow: debug: checking for duplicate targets.
2012/07/30 12:18:54.30 10131 makeflow: debug: checking rules for consistency. 2012/07/30 12:18:54.30 10131 makeflow: tcp: listening on port xxxx 2012/07/30 12:18:54.30 10131 makeflow: wq: Work queue is listening on port xxxx. 2012/07/30 12:18:54.30 10131 makeflow: batch: started process 10132: /bin/echo hello world input. Txt 2012/07/30 12:18:54.30 10131 makeflow: debug: node 0 waiting - running 2012/07/30 12:18:54.30 10131 makeflow: debug: Job 10132 has returned. 2012/07/30 12:18:54.30 10131 makeflow: debug: node 0 running - complete 2012/07/30 12:18:54.30 10131 makeflow: debug: node 4 waiting - running 2012/07/30 12:18:54.30 10131 makeflow: debug: node 3 waiting - running 2012/07/30 12:18:54.31 10131 makeflow: debug: node 2 waiting - running 2012/07/30 12:18:54.31 10131 makeflow: debug. Work_queue_worker -t 5 -d all localhost xxxx the debug output for the worker will appear on your terminal.
This is basically the same as regular Unix make using the -j flag. Makeflow -t local If everything worked out correctly, you should see: makeflow -t local /bin/echo hello world input. Txt d python 3 input. Txt c python 2 input. Txt b python 1 input.
Txt a nothing left. Running with FutureGrid's Torque the following code tells makeflow to dispatch jobs using the torque batch submission system (qsub, qdel, qstat, etc.). Makeflow -t torque you will get as output: makeflow -t torque nothing left. We need to clean out the generated output files and logs so makeflow starts from a clean slate again: makeflow -c we see it deleted the files we generated in the last run: makeflow -c deleted file d deleted file c deleted file b deleted. Txt deleted file./keflowlog Now let's try again: makeflow -t torque we get the output we expect: makeflow -t torque /bin/echo hello world input. Notice that the output is no different from using local execution. Makeflow is built to be execution engine agnostic. There is no difference between executing the task locally or remotely. However, submission and wait times will definitely lag compared to work queue because of the latency of the underlying batch job submission platform.
Fraser Valley regional Library
Txt python rainbow 1 input. Txt a b: input. Txt python 2 input. Txt b c: input. Txt python 3 input. Txt c d: input. Txt python 4 input. Txt d running with Local (Multiprocess) Execution Here we're going to tell makeflow to dispatch the jobs using regular local processes (no distributed computing!).
Setup a sandbox for this Tutorial and Download a copy of cctools.5.3 mkdir /cctools-tutorial wget tar xzf build and Install cctools cd /cctools-3.5.3-RC1-source./configure -prefix /cctools make install. Set Environment Variables, you will also hotel want to add your cctools directory to your path export path/cctools/bin:path, we will use python in this tutorial. It will need to be able to find our installed packages. Export setup mkdir cd download which is our application executable for this exercise. Download this makeflow file which defines the workflow. Wget wget The makeflow should look like: cat makeflow input. Txt: local /bin/echo hello world input.
homework tutorial will have you install cctools into your FutureGrid home directory and will take you through some distributed computation examples using. Makeflow and, work queue. Login to the future Grid head Node. For this homework, we assume you have an open. Ssh connection to the future Grid alamo login node. If you do not have an account with. Future Grid, then you may register here, download, build and Install cctools, navigate to the download page in your browser to review the most recent versions: tml.
Hw 2: Bug algorithm, due thursday 21 February, do a lit search on navigation algorithms and pick one of interest. For example, look for the family of Bug robot navigation algorithms. After picking an algorithm, write up a brief wiki page describing it along with links to appropriate resources or papers. Please put the label cs363s13hw02 on your writeup. Getting Started, login to the future Grid head Node. Download, build and Install cctools, set Environment Variables, simple makeflow Example. Setup, running with Local (Multiprocess) Execution, running with FutureGrid's Torque. Running with Work queue, running with Work queue workers on Torque.
Welcome to, teen Web!
HW 1: your favorite robot, due tuesday 12 February by class time. Take a look at information about real robot research projects. Find one that you like and write a 1-page wiki entry on where it, with links to the project and relevant papers. Identify and describe the mechanisms or joints, the sensors, computational abilities, and software capabilities. What about the robot makes it interesting? A good place to start is the cmu, robotics Institute. Please put the label cs363s13hw01 on your writeup.