并行执行

1.3 新版功能.

What it does¶

Because Fabric 1.x is not fully threadsafe (and because in general use, task functions do not typically interact with one another) this functionality is implemented via the Python multiprocessing module. It creates one new process for each host and task combination, optionally using a (configurable) sliding window to prevent too many processes from running at the same time.

For example, imagine a scenario where you want to update Web application code on a number of Web servers, and then reload the servers once the code has been distributed everywhere (to allow for easier rollback if code updates fail.) One could implement this with the following fabfile:

from fabric.api import *

def update():
    with cd("/srv/django/myapp"):
        run("git pull")

def reload():
    sudo("service apache2 reload")

and execute it on a set of 3 servers, in serial, like so:

$ fab -H web1,web2,web3 update reload

Normally, without any parallel execution options activated, Fabric would run in order:

1. update on web1
2. update on web2
3. update on web3
4. reload on web1
5. reload on web2
6. reload on web3

With parallel execution activated (via -P – see below for details), this turns into:

1. update on web1, web2, and web3
2. reload on web1, web2, and web3

Hopefully the benefits of this are obvious – if update took 5 seconds to run and reload took 2 seconds, serial execution takes (5+2)*3 = 21 seconds to run, while parallel execution takes only a third of the time, (5+2) = 7 seconds on average.

How to use it¶

from fabric.api import *

@parallel
def runs_in_parallel():
    pass

def runs_serially():
    pass

$ fab -H host1,host2,host3 runs_in_parallel runs_serially

from fabric.api import *

def runs_in_parallel():
    pass

@serial
def runs_serially():
    pass

$ fab -H host1,host2,host3 -P runs_in_parallel runs_serially

Bubble size¶

With large host lists, a user’s local machine can get overwhelmed by running too many concurrent Fabric processes. Because of this, you may opt to use a moving bubble approach that limits Fabric to a specific number of concurrently active processes.

By default, no bubble is used and all hosts are run in one concurrent pool. You can override this on a per-task level by specifying the pool_size keyword argument to parallel, or globally via -z.

For example, to run on 5 hosts at a time:

from fabric.api import *

@parallel(pool_size=5)
def heavy_task():
    # lots of heavy local lifting or lots of IO here

Or skip the pool_size kwarg and instead:

$ fab -P -z 5 heavy_task

Linewise vs bytewise output¶

Fabric’s default mode of printing to the terminal is byte-by-byte, in order to support 与远程程序集成. This often gives poor results when running in parallel mode, as the multiple processes may write to your terminal’s standard out stream simultaneously.

To help offset this problem, Fabric’s option for linewise output is automatically enabled whenever parallelism is active. This will cause you to lose most of the benefits outlined in the above link Fabric’s remote interactivity features, but as those do not map well to parallel invocations, it’s typically a fair trade.

There’s no way to avoid the multiple processes mixing up on a line-by-line basis, but you will at least be able to tell them apart by the host-string line prefix.