1. What is schedula?
Schedula implements a intelligent function scheduler, which selects and
executes functions. The order (workflow) is calculated from the provided inputs
and the requested outputs. A function is executed when all its dependencies
(i.e., inputs, input domain) are satisfied and when at least one of its outputs
has to be calculated.
Note
Schedula is performing the runtime selection of the minimum-workflow to
be invoked. A workflow describes the overall process - i.e., the order of
function execution - and it is defined by a directed acyclic graph (DAG).
The minimum-workflow is the DAG where each output is calculated using the
shortest path from the provided inputs. The path is calculated on the basis
of a weighed directed graph (data-flow diagram) with a modified Dijkstra
algorithm.
2. Installation
To install it use (with root privileges):
Or download the last git version and use (with root privileges):
$ python setup.py install
3. Why may I use schedula?
Imagine we have a system of interdependent functions - i.e. the inputs
of a function are the output for one or more function(s), and we do not know
which input the user will provide and which output will request. With a normal
scheduler you would have to code all possible implementations. I’m bored to
think and code all possible combinations of inputs and outputs from a model.
3.1. Solution
Schedula allows to write a simple model (Dispatcher()
) with
just the basic functions, then the Dispatcher()
will select and
execute the proper functions for the given inputs and the requested outputs.
Moreover, schedula provides a flexible framework for structuring code. It
allows to extract sub-models from a bigger one.
Note
A successful application is CO2 MPAS, where schedula has been used
to model an entire vehicle .
4. Very simple example
Let’s assume that we have to extract some filesystem attributes and we do not
know which inputs the user will provide. The code below shows how to create a
Dispatcher()
adding the functions that define your system.
Note that with this simple system the maximum number of inputs combinations is
31 (\((2^n - 1)\) , where n is the number of data).
>>> import schedula
>>> import os.path as osp
>>> dsp = schedula . Dispatcher ()
>>> dsp . add_data ( data_id = 'dirname' , default_value = '.' , initial_dist = 2 )
'dirname'
>>> dsp . add_function ( function = osp . split , inputs = [ 'path' ],
... outputs = [ 'dirname' , 'basename' ])
'split'
>>> dsp . add_function ( function = osp . splitext , inputs = [ 'basename' ],
... outputs = [ 'fname' , 'suffix' ])
'splitext'
>>> dsp . add_function ( function = osp . join , inputs = [ 'dirname' , 'basename' ],
... outputs = [ 'path' ])
'join'
>>> dsp . add_function ( function_id = 'union' , function = lambda * a : '' . join ( a ),
... inputs = [ 'fname' , 'suffix' ], outputs = [ 'basename' ])
'union'
digraph dmap {
graph [ratio=1]
node [style=filled]
label = <dmap>
splines = ortho
style = filled
493 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">basename</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=basename]
494 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">dirname</TD></TR><TR><TD align="RIGHT" border="1">default</TD><TD align="LEFT" border="1">.</TD></TR><TR><TD align="RIGHT" border="1">initial_dist</TD><TD align="LEFT" border="1">2</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=dirname]
495 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">fname</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=fname]
496 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-bda0a2f1d84ca3228debb5f4520122317594c6fe/join.html">join</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip="Join two or more pathname components, inserting '/' as needed."]
497 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">path</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=path]
498 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-bda0a2f1d84ca3228debb5f4520122317594c6fe/split.html">split</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip="Split a pathname."]
499 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-bda0a2f1d84ca3228debb5f4520122317594c6fe/splitext.html">splitext</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip="Split the extension from a pathname."]
500 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">suffix</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=suffix]
501 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-bda0a2f1d84ca3228debb5f4520122317594c6fe/lambda.html">union</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip=union]
495 -> 501
498 -> 494
499 -> 500
499 -> 495
500 -> 501
493 -> 496
494 -> 496
496 -> 497
497 -> 498
493 -> 499
498 -> 493
501 -> 493
}
Tip
You can explore the diagram by clicking on it.
The next step to calculate the outputs would be just to run the
dispatch()
method. You can invoke it with just the
inputs, so it will calculate all reachable outputs:
>>> inputs = { 'path' : 'schedula/_version.py' }
>>> o = dsp . dispatch ( inputs = inputs )
>>> o
Solution([('path', 'schedula/_version.py'),
('basename', '_version.py'),
('dirname', 'schedula'),
('fname', '_version'),
('suffix', '.py')])
digraph workflow {
graph [ratio=1]
node [style=filled]
label = <workflow>
splines = ortho
style = filled
514 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-806cc650ed876917f9cd7a0cf9ff30f237c4569d/basename-output.html">basename</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">2.0</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=basename]
515 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-806cc650ed876917f9cd7a0cf9ff30f237c4569d/dirname-output.html">dirname</TD></TR><TR><TD align="RIGHT" border="1">default</TD><TD align="LEFT" border="1">.</TD></TR><TR><TD align="RIGHT" border="1">initial_dist</TD><TD align="LEFT" border="1">2</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">2.0</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=dirname]
516 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-806cc650ed876917f9cd7a0cf9ff30f237c4569d/fname-output.html">fname</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">4.0</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=fname]
517 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-806cc650ed876917f9cd7a0cf9ff30f237c4569d/path-output.html">path</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">0.0</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=path]
518 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-806cc650ed876917f9cd7a0cf9ff30f237c4569d/split.html">split</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">1.0</TD></TR><TR><TD align="RIGHT" border="1">started</TD><TD align="LEFT" border="1">2018-10-08T22:02:44.458127</TD></TR><TR><TD align="RIGHT" border="1">duration</TD><TD align="LEFT" border="1">0:00:00.000011</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip="Split a pathname."]
519 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-806cc650ed876917f9cd7a0cf9ff30f237c4569d/splitext.html">splitext</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">3.0</TD></TR><TR><TD align="RIGHT" border="1">started</TD><TD align="LEFT" border="1">2018-10-08T22:02:44.458266</TD></TR><TR><TD align="RIGHT" border="1">duration</TD><TD align="LEFT" border="1">0:00:00.000008</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip="Split the extension from a pathname."]
520 [label=start fillcolor=red shape=egg]
521 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-806cc650ed876917f9cd7a0cf9ff30f237c4569d/suffix-output.html">suffix</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">4.0</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=suffix]
518 -> 515
517 -> 518
519 -> 521
520 -> 515
519 -> 516
514 -> 519
520 -> 517
518 -> 514
}
or you can set also the outputs, so the dispatch will stop when it will find all
outputs:
>>> o = dsp . dispatch ( inputs = inputs , outputs = [ 'basename' ])
>>> o
Solution([('path', 'schedula/_version.py'), ('basename', '_version.py')])
digraph workflow {
graph [ratio=1]
node [style=filled]
label = <workflow>
splines = ortho
style = filled
530 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-0311a927c48afe0c50f5bcf9ab2b65511411d6e9/basename-output.html">basename</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">2.0</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=basename]
531 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-0311a927c48afe0c50f5bcf9ab2b65511411d6e9/path-output.html">path</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">0.0</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=path]
532 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-0311a927c48afe0c50f5bcf9ab2b65511411d6e9/split.html">split</TD></TR><TR><TD align="RIGHT" border="1">M_outputs</TD><TD align="LEFT" border="1">('dirname',)</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">1.0</TD></TR><TR><TD align="RIGHT" border="1">started</TD><TD align="LEFT" border="1">2018-10-08T22:02:44.497116</TD></TR><TR><TD align="RIGHT" border="1">duration</TD><TD align="LEFT" border="1">0:00:00.000010</TD></TR></TABLE>> fillcolor=orange shape=box tooltip="Split a pathname."]
533 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-0311a927c48afe0c50f5bcf9ab2b65511411d6e9/splitext.html">splitext</TD></TR><TR><TD align="RIGHT" border="1">M_outputs</TD><TD align="LEFT" border="1">('fname', 'suffix')</TD></TR></TABLE>> fillcolor=orange shape=box tooltip="Split the extension from a pathname."]
534 [label=start fillcolor=red shape=egg]
530 -> 533
531 -> 532
534 -> 531
532 -> 530
}
5. Advanced example (circular system)
Systems of interdependent functions can be described by “graphs” and they might
contains circles . This kind of system can not be resolved by a normal
scheduler.
Suppose to have a system of sequential functions in circle - i.e., the input of
a function is the output of the previous function. The maximum number of input
and output permutations is \((2^n - 1)^2\) , where n is the number of
functions. Thus, with a normal scheduler you have to code all possible
implementations, so \((2^n - 1)^2\) functions (IMPOSSIBLE!!!).
Schedula will simplify your life. You just create a
Dispatcher()
, that contains all functions that link your data:
>>> import schedula
>>> dsp = schedula . Dispatcher ()
>>> plus , minus = lambda x : x + 1 , lambda x : x - 1
>>> n = j = 6
>>> for i in range ( 1 , n + 1 ):
... func = plus if i < ( n / 2 + 1 ) else minus
... f = dsp . add_function ( 'f %d ' % i , func , [ 'v %d ' % j ], [ 'v %d ' % i ])
... j = i
digraph dmap {
graph [ratio=1]
node [style=filled]
label = <dmap>
splines = curves
style = filled
539 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-9c7dd4b450501f09f46dc42451783c6aa5aaf77b/lambda.html">f1</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip=f1]
540 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-9c7dd4b450501f09f46dc42451783c6aa5aaf77b/lambda-0.html">f2</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip=f2]
541 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-9c7dd4b450501f09f46dc42451783c6aa5aaf77b/lambda-1.html">f3</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip=f3]
542 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-9c7dd4b450501f09f46dc42451783c6aa5aaf77b/lambda-2.html">f4</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip=f4]
543 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-9c7dd4b450501f09f46dc42451783c6aa5aaf77b/lambda-3.html">f5</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip=f5]
544 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-9c7dd4b450501f09f46dc42451783c6aa5aaf77b/lambda-4.html">f6</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip=f6]
545 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v1</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v1]
546 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v2</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v2]
547 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v3</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v3]
548 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v4</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v4]
549 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v5</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v5]
550 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v6</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v6]
545 -> 540
546 -> 541
542 -> 548
543 -> 549
540 -> 546
547 -> 542
539 -> 545
541 -> 547
544 -> 550
548 -> 543
549 -> 544
550 -> 539
}
Then it will handle all possible combination of inputs and outputs
(\((2^n - 1)^2\) ) just invoking the dispatch()
method, as follows:
>>> out = dsp . dispatch ( inputs = { 'v1' : 0 , 'v4' : 1 }, outputs = [ 'v2' , 'v6' ])
>>> out
Solution([('v1', 0), ('v4', 1), ('v2', 1), ('v5', 0), ('v6', -1)])
digraph workflow {
node [style=filled]
label = <workflow>
splines = ortho
style = filled
563 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-3c5653db8609422c1d79476e16b8fea9a04a0ead/lambda.html">f2</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">1.0</TD></TR><TR><TD align="RIGHT" border="1">started</TD><TD align="LEFT" border="1">2018-10-08T22:02:44.545694</TD></TR><TR><TD align="RIGHT" border="1">duration</TD><TD align="LEFT" border="1">0:00:00.000006</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip=f2]
564 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-3c5653db8609422c1d79476e16b8fea9a04a0ead/lambda-0.html">f5</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">1.0</TD></TR><TR><TD align="RIGHT" border="1">started</TD><TD align="LEFT" border="1">2018-10-08T22:02:44.545729</TD></TR><TR><TD align="RIGHT" border="1">duration</TD><TD align="LEFT" border="1">0:00:00.000004</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip=f5]
565 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-3c5653db8609422c1d79476e16b8fea9a04a0ead/lambda-1.html">f6</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">3.0</TD></TR><TR><TD align="RIGHT" border="1">started</TD><TD align="LEFT" border="1">2018-10-08T22:02:44.545806</TD></TR><TR><TD align="RIGHT" border="1">duration</TD><TD align="LEFT" border="1">0:00:00.000004</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip=f6]
566 [label=start fillcolor=red shape=egg]
567 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-3c5653db8609422c1d79476e16b8fea9a04a0ead/v1-output.html">v1</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">0.0</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v1]
568 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-3c5653db8609422c1d79476e16b8fea9a04a0ead/v2-output.html">v2</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">2.0</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v2]
569 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-3c5653db8609422c1d79476e16b8fea9a04a0ead/v4-output.html">v4</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">0.0</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v4]
570 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-3c5653db8609422c1d79476e16b8fea9a04a0ead/v5-output.html">v5</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">2.0</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v5]
571 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-3c5653db8609422c1d79476e16b8fea9a04a0ead/v6-output.html">v6</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">4.0</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v6]
567 -> 563
563 -> 568
566 -> 567
565 -> 571
569 -> 564
564 -> 570
570 -> 565
566 -> 569
}
6. Next moves
Things yet to do include a mechanism to allow the execution of functions in
parallel.