1. What is schedula?
Schedula implements a intelligent function scheduler, which selects and
executes functions. The order (workflow) is calculated from the provided inputs
and the requested outputs. A function is executed when all its dependencies
(i.e., inputs, input domain) are satisfied and when at least one of its outputs
has to be calculated.
Note
Schedula is performing the runtime selection of the minimum-workflow to
be invoked. A workflow describes the overall process - i.e., the order of
function execution - and it is defined by a directed acyclic graph (DAG).
The minimum-workflow is the DAG where each output is calculated using the
shortest path from the provided inputs. The path is calculated on the basis
of a weighed directed graph (data-flow diagram) with a modified Dijkstra
algorithm.
2. Installation
To install it use (with root privileges):
Or download the last git version and use (with root privileges):
$ python setup.py install
3. Why may I use schedula?
Imagine we have a system of interdependent functions - i.e. the inputs
of a function are the output for one or more function(s), and we do not know
which input the user will provide and which output will request. With a normal
scheduler you would have to code all possible implementations. I’m bored to
think and code all possible combinations of inputs and outputs from a model.
3.1. Solution
Schedula allows to write a simple model
(Dispatcher
) with just the basic functions, then
the Dispatcher
will select and execute the proper
functions for the given inputs and the requested outputs.
Moreover, schedula provides a flexible framework for structuring code. It
allows to extract sub-models from a bigger one and to run your model
asynchronously or in parallel without extra coding.
Note
A successful application is CO2 MPAS, where schedula has been used
to model an entire vehicle .
4. Very simple example
Let’s assume that we have to extract some filesystem attributes and we do not
know which inputs the user will provide. The code below shows how to create a
Dispatcher
adding the functions that define your
system.
Note that with this simple system the maximum number of inputs combinations is
31 (\((2^n - 1)\) , where n is the number of data).
>>> import schedula as sh
>>> import os.path as osp
>>> dsp = sh . Dispatcher ()
>>> dsp . add_data ( data_id = 'dirname' , default_value = '.' , initial_dist = 2 )
'dirname'
>>> dsp . add_function ( function = osp . split , inputs = [ 'path' ],
... outputs = [ 'dirname' , 'basename' ])
'split'
>>> dsp . add_function ( function = osp . splitext , inputs = [ 'basename' ],
... outputs = [ 'fname' , 'suffix' ])
'splitext'
>>> dsp . add_function ( function = osp . join , inputs = [ 'dirname' , 'basename' ],
... outputs = [ 'path' ])
'join'
>>> dsp . add_function ( function_id = 'union' , function = lambda * a : '' . join ( a ),
... inputs = [ 'fname' , 'suffix' ], outputs = [ 'basename' ])
'union'
digraph dmap {
graph [ratio=1]
node [style=filled]
label = <dmap>
splines = ortho
style = filled
587 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">basename</TD></TR></TABLE>> fillcolor=cyan id=3 shape=box style="rounded,filled" tooltip="\"basename\""]
588 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">dirname</TD></TR><TR><TD align="RIGHT" border="1">default</TD><TD align="LEFT" border="1">.</TD></TR><TR><TD align="RIGHT" border="1">initial_dist</TD><TD align="LEFT" border="1">2</TD></TR></TABLE>> fillcolor=cyan id=0 shape=box style="rounded,filled" tooltip="\"dirname\""]
589 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">fname</TD></TR></TABLE>> fillcolor=cyan id=5 shape=box style="rounded,filled" tooltip="\"fname\""]
590 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-bc3b4407269331934f472c55b7d6801b624c975c/join.html">join</TD></TR></TABLE>> fillcolor=springgreen id=7 shape=box tooltip="\"Join two or more pathname components, inserting '/' as needed.\""]
591 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">path</TD></TR></TABLE>> fillcolor=cyan id=2 shape=box style="rounded,filled" tooltip="\"path\""]
592 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-bc3b4407269331934f472c55b7d6801b624c975c/split.html">split</TD></TR></TABLE>> fillcolor=springgreen id=1 shape=box tooltip="\"Split a pathname.\""]
593 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-bc3b4407269331934f472c55b7d6801b624c975c/splitext.html">splitext</TD></TR></TABLE>> fillcolor=springgreen id=4 shape=box tooltip="\"Split the extension from a pathname.\""]
594 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">suffix</TD></TR></TABLE>> fillcolor=cyan id=6 shape=box style="rounded,filled" tooltip="\"suffix\""]
595 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-bc3b4407269331934f472c55b7d6801b624c975c/lambda.html">union</TD></TR></TABLE>> fillcolor=springgreen id=8 shape=box tooltip="\"union\""]
588 -> 590
592 -> 588
592 -> 587
591 -> 592
587 -> 593
587 -> 590
593 -> 589
593 -> 594
589 -> 595
594 -> 595
590 -> 591
595 -> 587
}
Tip
You can explore the diagram by clicking on it.
The next step to calculate the outputs would be just to run the
dispatch()
method. You can invoke it with
just the inputs, so it will calculate all reachable outputs:
>>> inputs = { 'path' : 'schedula/_version.py' }
>>> o = dsp . dispatch ( inputs = inputs )
>>> o
Solution([('path', 'schedula/_version.py'),
('basename', '_version.py'),
('dirname', 'schedula'),
('fname', '_version'),
('suffix', '.py')])
digraph workflow {
graph [ratio=1]
node [style=filled]
label = <workflow>
splines = ortho
style = filled
608 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-fddc1ba9e634eb6a59f61af7eb46078afb299a12/basename-output.html">basename</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">2.0</TD></TR></TABLE>> fillcolor=cyan id=3 shape=box style="rounded,filled" tooltip="\"basename\""]
609 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-fddc1ba9e634eb6a59f61af7eb46078afb299a12/dirname-output.html">dirname</TD></TR><TR><TD align="RIGHT" border="1">default</TD><TD align="LEFT" border="1">.</TD></TR><TR><TD align="RIGHT" border="1">initial_dist</TD><TD align="LEFT" border="1">2</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">2.0</TD></TR></TABLE>> fillcolor=cyan id=0 shape=box style="rounded,filled" tooltip="\"dirname\""]
610 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-fddc1ba9e634eb6a59f61af7eb46078afb299a12/fname-output.html">fname</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">4.0</TD></TR></TABLE>> fillcolor=cyan id=5 shape=box style="rounded,filled" tooltip="\"fname\""]
611 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-fddc1ba9e634eb6a59f61af7eb46078afb299a12/path-output.html">path</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">0.0</TD></TR></TABLE>> fillcolor=cyan id=2 shape=box style="rounded,filled" tooltip="\"path\""]
612 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-fddc1ba9e634eb6a59f61af7eb46078afb299a12/split.html">split</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">1.0</TD></TR><TR><TD align="RIGHT" border="1">started</TD><TD align="LEFT" border="1">2019-12-06T17:55:02.941879</TD></TR><TR><TD align="RIGHT" border="1">duration</TD><TD align="LEFT" border="1">0:00:00.000033</TD></TR></TABLE>> fillcolor=springgreen id=1 shape=box tooltip="\"Split a pathname.\""]
613 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-fddc1ba9e634eb6a59f61af7eb46078afb299a12/splitext.html">splitext</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">3.0</TD></TR><TR><TD align="RIGHT" border="1">started</TD><TD align="LEFT" border="1">2019-12-06T17:55:02.942063</TD></TR><TR><TD align="RIGHT" border="1">duration</TD><TD align="LEFT" border="1">0:00:00.000021</TD></TR></TABLE>> fillcolor=springgreen id=4 shape=box tooltip="\"Split the extension from a pathname.\""]
614 [label=start fillcolor=red id=start shape=egg]
615 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-fddc1ba9e634eb6a59f61af7eb46078afb299a12/suffix-output.html">suffix</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">4.0</TD></TR></TABLE>> fillcolor=cyan id=6 shape=box style="rounded,filled" tooltip="\"suffix\""]
614 -> 611
614 -> 609
611 -> 612
612 -> 609
612 -> 608
608 -> 613
613 -> 610
613 -> 615
}
or you can set also the outputs, so the dispatch will stop when it will find all
outputs:
>>> o = dsp . dispatch ( inputs = inputs , outputs = [ 'basename' ])
>>> o
Solution([('path', 'schedula/_version.py'), ('basename', '_version.py')])
digraph workflow {
graph [ratio=1]
node [style=filled]
label = <workflow>
splines = ortho
style = filled
624 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-44648e24f803bb1404af84a5e461f399753b9c66/basename-output.html">basename</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">2.0</TD></TR></TABLE>> fillcolor=cyan id=3 shape=box style="rounded,filled" tooltip="\"basename\""]
625 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-44648e24f803bb1404af84a5e461f399753b9c66/path-output.html">path</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">0.0</TD></TR></TABLE>> fillcolor=cyan id=2 shape=box style="rounded,filled" tooltip="\"path\""]
626 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-44648e24f803bb1404af84a5e461f399753b9c66/split.html">split</TD></TR><TR><TD align="RIGHT" border="1">M_outputs</TD><TD align="LEFT" border="1">('dirname',)</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">1.0</TD></TR><TR><TD align="RIGHT" border="1">started</TD><TD align="LEFT" border="1">2019-12-06T17:55:02.996090</TD></TR><TR><TD align="RIGHT" border="1">duration</TD><TD align="LEFT" border="1">0:00:00.000025</TD></TR></TABLE>> fillcolor=orange id=1 shape=box tooltip="\"Split a pathname.\""]
627 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-44648e24f803bb1404af84a5e461f399753b9c66/splitext.html">splitext</TD></TR><TR><TD align="RIGHT" border="1">M_outputs</TD><TD align="LEFT" border="1">('fname', 'suffix')</TD></TR></TABLE>> fillcolor=orange id=4 shape=box tooltip="\"Split the extension from a pathname.\""]
628 [label=start fillcolor=red id=start shape=egg]
628 -> 625
625 -> 626
626 -> 624
624 -> 627
}
5. Advanced example (circular system)
Systems of interdependent functions can be described by “graphs” and they might
contains circles . This kind of system can not be resolved by a normal
scheduler.
Suppose to have a system of sequential functions in circle - i.e., the input of
a function is the output of the previous function. The maximum number of input
and output permutations is \((2^n - 1)^2\) , where n is the number of
functions. Thus, with a normal scheduler you have to code all possible
implementations, so \((2^n - 1)^2\) functions (IMPOSSIBLE!!!).
Schedula will simplify your life. You just create a
Dispatcher
, that contains all functions that link
your data:
>>> import schedula as sh
>>> dsp = sh . Dispatcher ()
>>> increment = lambda x : x + 1
>>> for k , ( i , j ) in enumerate ( sh . pairwise ([ 1 , 2 , 3 , 4 , 5 , 6 , 1 ])):
... dsp . add_function ( 'f %d ' % k , increment , [ 'v %d ' % i ], [ 'v %d ' % j ])
'...'
digraph dmap {
graph [ratio=1]
node [style=filled]
label = <dmap>
splines = curves
style = filled
633 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b4899571ee5e4178fa6ff7ffac3d13ed8ad48f20/lambda.html">f0</TD></TR></TABLE>> fillcolor=springgreen id=0 shape=box tooltip="\"f0\""]
634 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b4899571ee5e4178fa6ff7ffac3d13ed8ad48f20/lambda-0.html">f1</TD></TR></TABLE>> fillcolor=springgreen id=3 shape=box tooltip="\"f1\""]
635 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b4899571ee5e4178fa6ff7ffac3d13ed8ad48f20/lambda-1.html">f2</TD></TR></TABLE>> fillcolor=springgreen id=5 shape=box tooltip="\"f2\""]
636 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b4899571ee5e4178fa6ff7ffac3d13ed8ad48f20/lambda-2.html">f3</TD></TR></TABLE>> fillcolor=springgreen id=7 shape=box tooltip="\"f3\""]
637 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b4899571ee5e4178fa6ff7ffac3d13ed8ad48f20/lambda-3.html">f4</TD></TR></TABLE>> fillcolor=springgreen id=9 shape=box tooltip="\"f4\""]
638 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b4899571ee5e4178fa6ff7ffac3d13ed8ad48f20/lambda-4.html">f5</TD></TR></TABLE>> fillcolor=springgreen id=11 shape=box tooltip="\"f5\""]
639 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v1</TD></TR></TABLE>> fillcolor=cyan id=1 shape=box style="rounded,filled" tooltip="\"v1\""]
640 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v2</TD></TR></TABLE>> fillcolor=cyan id=2 shape=box style="rounded,filled" tooltip="\"v2\""]
641 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v3</TD></TR></TABLE>> fillcolor=cyan id=4 shape=box style="rounded,filled" tooltip="\"v3\""]
642 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v4</TD></TR></TABLE>> fillcolor=cyan id=6 shape=box style="rounded,filled" tooltip="\"v4\""]
643 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v5</TD></TR></TABLE>> fillcolor=cyan id=8 shape=box style="rounded,filled" tooltip="\"v5\""]
644 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v6</TD></TR></TABLE>> fillcolor=cyan id=10 shape=box style="rounded,filled" tooltip="\"v6\""]
633 -> 640
639 -> 633
640 -> 634
634 -> 641
641 -> 635
635 -> 642
642 -> 636
636 -> 643
643 -> 637
637 -> 644
644 -> 638
638 -> 639
}
Then it will handle all possible combination of inputs and outputs
(\((2^n - 1)^2\) ) just invoking the
dispatch()
method, as follows:
>>> out = dsp . dispatch ( inputs = { 'v1' : 0 , 'v4' : 1 }, outputs = [ 'v2' , 'v6' ])
>>> out
Solution([('v1', 0), ('v4', 1), ('v2', 1), ('v5', 2), ('v6', 3)])
digraph workflow {
graph [bgcolor=transparent]
node [style=filled]
label = <workflow>
splines = ortho
style = filled
657 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b6f1955b896ae811e4ceb78e39de539282a5f967/lambda.html">f0</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">1.0</TD></TR><TR><TD align="RIGHT" border="1">started</TD><TD align="LEFT" border="1">2019-12-06T17:55:03.049857</TD></TR><TR><TD align="RIGHT" border="1">duration</TD><TD align="LEFT" border="1">0:00:00.000024</TD></TR></TABLE>> fillcolor=springgreen id=0 shape=box tooltip="\"f0\""]
658 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b6f1955b896ae811e4ceb78e39de539282a5f967/lambda-0.html">f3</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">1.0</TD></TR><TR><TD align="RIGHT" border="1">started</TD><TD align="LEFT" border="1">2019-12-06T17:55:03.049916</TD></TR><TR><TD align="RIGHT" border="1">duration</TD><TD align="LEFT" border="1">0:00:00.000015</TD></TR></TABLE>> fillcolor=springgreen id=7 shape=box tooltip="\"f3\""]
659 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b6f1955b896ae811e4ceb78e39de539282a5f967/lambda-1.html">f4</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">3.0</TD></TR><TR><TD align="RIGHT" border="1">started</TD><TD align="LEFT" border="1">2019-12-06T17:55:03.050010</TD></TR><TR><TD align="RIGHT" border="1">duration</TD><TD align="LEFT" border="1">0:00:00.000015</TD></TR></TABLE>> fillcolor=springgreen id=9 shape=box tooltip="\"f4\""]
660 [label=start fillcolor=red id=start shape=egg]
661 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b6f1955b896ae811e4ceb78e39de539282a5f967/v1-output.html">v1</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">0.0</TD></TR></TABLE>> fillcolor=cyan id=1 shape=box style="rounded,filled" tooltip="\"v1\""]
662 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b6f1955b896ae811e4ceb78e39de539282a5f967/v2-output.html">v2</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">2.0</TD></TR></TABLE>> fillcolor=cyan id=2 shape=box style="rounded,filled" tooltip="\"v2\""]
663 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b6f1955b896ae811e4ceb78e39de539282a5f967/v4-output.html">v4</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">0.0</TD></TR></TABLE>> fillcolor=cyan id=6 shape=box style="rounded,filled" tooltip="\"v4\""]
664 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b6f1955b896ae811e4ceb78e39de539282a5f967/v5-output.html">v5</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">2.0</TD></TR></TABLE>> fillcolor=cyan id=8 shape=box style="rounded,filled" tooltip="\"v5\""]
665 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b6f1955b896ae811e4ceb78e39de539282a5f967/v6-output.html">v6</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">4.0</TD></TR></TABLE>> fillcolor=cyan id=10 shape=box style="rounded,filled" tooltip="\"v6\""]
660 -> 661
660 -> 663
661 -> 657
663 -> 658
657 -> 662
658 -> 664
664 -> 659
659 -> 665
}
5.2. Iterated function
Schedula allows to build an iterated function, i.e. the input is recalculated.
This could be done easily with the DispatchPipe
,
as follows:
>>> func = sh . DispatchPipe ( dsp , 'func' , ( 'v1' , 'v4' ), ( 'v1' , 'v4' ))
>>> x = [[ 1 , 4 ]]
>>> for i in range ( 6 ):
... x . append ( func ( * x [ - 1 ]))
>>> x
[[1, 4], [7, 4], [7, 10], [13, 10], [13, 16], [19, 16], [19, 22]]
6. Asynchronous and Parallel dispatching
When there are heavy calculations which takes a significant amount of time, you
want to run your model asynchronously or in parallel. Generally, this is
difficult to achieve, because it requires an higher level of abstraction and a
deeper knowledge of python programming and the Global Interpreter Lock (GIL).
Schedula will simplify again your life. It has four default executors to
dispatch asynchronously or in parallel:
async : execute all functions asynchronously in the same process,
parallel : execute all functions in parallel excluding
SubDispatch
functions,
parallel-pool : execute all functions in parallel using a process pool
excluding SubDispatch
functions,
parallel-dispatch : execute all functions in parallel including
SubDispatch
.
Note
Running functions asynchronously or in parallel has a cost. Schedula
will spend time creating / deleting new threads / processes.
The code below shows an example of a time consuming code, that with the
concurrent execution it requires at least 6 seconds to run. Note that the slow
function return the process id.
>>> import schedula as sh
>>> dsp = sh . Dispatcher ()
>>> def slow ():
... import os , time
... time . sleep ( 1 )
... return os . getpid ()
>>> for o in 'abcdef' :
... dsp . add_function ( function = slow , outputs = [ o ])
'...'
digraph dmap {
graph [bgcolor=transparent]
node [style=filled]
label = <dmap>
splines = ortho
style = filled
689 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">a</TD></TR></TABLE>> fillcolor=cyan id=2 shape=box style="rounded,filled" tooltip="\"a\""]
690 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">b</TD></TR></TABLE>> fillcolor=cyan id=4 shape=box style="rounded,filled" tooltip="\"b\""]
691 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">c</TD></TR></TABLE>> fillcolor=cyan id=6 shape=box style="rounded,filled" tooltip="\"c\""]
692 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">d</TD></TR></TABLE>> fillcolor=cyan id=8 shape=box style="rounded,filled" tooltip="\"d\""]
693 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">e</TD></TR></TABLE>> fillcolor=cyan id=10 shape=box style="rounded,filled" tooltip="\"e\""]
694 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">f</TD></TR></TABLE>> fillcolor=cyan id=12 shape=box style="rounded,filled" tooltip="\"f\""]
695 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-8da5d19fb6a74d9d1512dc8ab616fba9f0cfa6f8/slow.html">slow</TD></TR></TABLE>> fillcolor=springgreen id=1 shape=box tooltip="\"slow\""]
696 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-8da5d19fb6a74d9d1512dc8ab616fba9f0cfa6f8/slow-0.html">slow<0></TD></TR></TABLE>> fillcolor=springgreen id=3 shape=box tooltip="\"slow<0>\""]
697 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-8da5d19fb6a74d9d1512dc8ab616fba9f0cfa6f8/slow-1.html">slow<1></TD></TR></TABLE>> fillcolor=springgreen id=5 shape=box tooltip="\"slow<1>\""]
698 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-8da5d19fb6a74d9d1512dc8ab616fba9f0cfa6f8/slow-2.html">slow<2></TD></TR></TABLE>> fillcolor=springgreen id=7 shape=box tooltip="\"slow<2>\""]
699 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-8da5d19fb6a74d9d1512dc8ab616fba9f0cfa6f8/slow-3.html">slow<3></TD></TR></TABLE>> fillcolor=springgreen id=9 shape=box tooltip="\"slow<3>\""]
700 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-8da5d19fb6a74d9d1512dc8ab616fba9f0cfa6f8/slow-4.html">slow<4></TD></TR></TABLE>> fillcolor=springgreen id=11 shape=box tooltip="\"slow<4>\""]
701 [label=start fillcolor=red id=start shape=egg]
701 -> 695
701 -> 696
701 -> 697
701 -> 698
701 -> 699
701 -> 700
695 -> 689
696 -> 690
697 -> 691
698 -> 692
699 -> 693
700 -> 694
}
while using the async executor, it lasts a bit more then 1 second:
>>> import time
>>> start = time . time ()
>>> sol = dsp ( executor = 'async' ) . result () # Asynchronous execution.
>>> ( time . time () - start ) < 2 # Faster then concurrent execution.
True
all functions have been executed asynchronously, but in the same process:
>>> import os
>>> pid = os . getpid () # Current process id.
>>> { sol [ k ] for k in 'abcdef' } == { pid } # Single process id.
True
if we use the parallel executor all functions are executed in different
processes:
>>> sol = dsp ( executor = 'parallel' ) . result () # Parallel execution.
>>> pids = { sol [ k ] for k in 'abcdef' } # Process ids returned by `slow`.
>>> len ( pids ) == 6 # Each function returns a different process id.
True
>>> pid not in pids # The current process id is not in the returned pids.
True
>>> sorted ( sh . shutdown_executors ())
['async', 'parallel']
7. Next moves
Things yet to do: utility to transform a dispatcher in a command line tool.