Example 1

In this example, we have:

• one Manager that own a task Queue
• one Boss that pushes tasks in that Queue
• Minions that get things done

Test locally

In one shell: ./manager.py In another: ./boss.py And in many others: ./minion.py

Check how many tasks are still in the queue

> from manager import QueueClient
> c = QueueClient()
> c.queue.qsize()
100

Test on multiple computers

Network architecture

The architecture will run 3 very different processes: the manager, the boss and the minions.

You should first decide which computers your are going to use. There is no official list of available host @ gepetto. You should decide which one to use with Guilhem. In the following example, we are using nyusan.laas.fr . DONT directly use nyusan without thinking or asking the team (nyusan might be busy when you sudently decide to use it).

Most of the load is taken by the minions, so they should go on the computers where you have CPU time available.

The manager should be "seen" by all the processes. So all the computers for minions and boss should be able to ping the computer where the manager is running.

The boss is going to send the requests to the manager and wait to collect the results. The boss typically runs on your personal computer. This computer can be off the main network (the manager does not have to be able to ping the boss), and can even be outside of the lab if the network port is open to ssh). See port forwarding below.

Port forwarding

The minions, manager and boss typicall runs on the same local network. You may consider to run the boss on another network, for example from home, with manager and minions at the lab. Yet, the manager/boss code is written to use port 7481 which is not open for remote connection at the lab. You must then forward connection to port 7481 using:

bash
ssh -L 7481:localhost:7481 $USER@nyusan.laas.fr

Now, the boss can contact the manager buy simply connecting to localhost (and this will be forwarded by ssh to nyusan).

Prepare your container

The docker container should be rebuilt explicitely for the host of your manager.

After being rebuilt, it has to be pushed on a container registery. Several option might be consider. The most convenient way is likely to use the registery associated with the gitlab repository where this code is stored. For example, clone this repository on your gitlab.laas.fr account, and use the registery available at gitlab.laas.fr:4567/USER/multiprocessing-examples . Then, the full docker name for your container will be gitlab.laas.fr:4567/USER/multiprocessing-examples:EXAMPLE-NAME . (change $USER to your login e.g. gsaurel or nmansard, and EXAMPLE_NAME to anything relevant to you e.g. nmansard-example-1)

You likely will have first to docker-log on the registery. For example, if using gitlab.laas.fr registery, type:

docker login gitlab.laas.fr:4567

(don't forget to docker-login on each computer where you will use this registery)

The built and push of the container is then to be run from this directory (on any computer your want, for example your personal computer).

docker build --build-arg HOST=nyusan -t gitlab.laas.fr:4567/$USER/multiprocessing-examples:EXAMPLE-NAME .
docker push gitlab.laas.fr:4567/$USER/multiprocessing-examples:EXAMPLE-NAME

Deploy the boss-minon architercture

You can now start the minions from the docker container, on several computers. Log on the computers (e.g. by ssh), and run (docker login first, if not done yet):

docker run --rm -it gitlab.laas.fr:4567/$USER/multiprocessing-examples:EXAMPLE-NAME

The manager should be started similarly, but explicitely launching the manager script at docker-run. When doing so, you must tell docker to expose the port 7481 to the host with -p 7481:7481

docker run --rm -p 7481:7481 -it gitlab.laas.fr:4567/$USER/multiprocessing-examples:EXAMPLE-NAME /app/manager.py

Finally, run the boss. You can do it with docker or directly natively on your computer.

./boss.py