1. What is schedula?

Schedula implements a intelligent function scheduler, which selects and executes functions. The order (workflow) is calculated from the provided inputs and the requested outputs. A function is executed when all its dependencies (i.e., inputs, input domain) are satisfied and when at least one of its outputs has to be calculated.

Note

Schedula is performing the runtime selection of the minimum-workflow to be invoked. A workflow describes the overall process - i.e., the order of function execution - and it is defined by a directed acyclic graph (DAG). The minimum-workflow is the DAG where each output is calculated using the shortest path from the provided inputs. The path is calculated on the basis of a weighed directed graph (data-flow diagram) with a modified Dijkstra algorithm.

2. Installation

To install it use (with root privileges):

$ pip install schedula

Or download the last git version and use (with root privileges):

$ python setup.py install

3. Why may I use schedula?

Imagine we have a system of interdependent functions - i.e. the inputs of a function are the output for one or more function(s), and we do not know which input the user will provide and which output will request. With a normal scheduler you would have to code all possible implementations. I’m bored to think and code all possible combinations of inputs and outputs from a model.

3.1. Solution

Schedula allows to write a simple model (Dispatcher()) with just the basic functions, then the Dispatcher() will select and execute the proper functions for the given inputs and the requested outputs. Moreover, schedula provides a flexible framework for structuring code. It allows to extract sub-models from a bigger one.

Note

A successful application is CO2MPAS, where schedula has been used to model an entire vehicle.

4. Very simple example

Let’s assume that we have to extract some filesystem attributes and we do not know which inputs the user will provide. The code below shows how to create a Dispatcher() adding the functions that define your system. Note that with this simple system the maximum number of inputs combinations is 31 (\((2^n - 1)\), where n is the number of data).

>>> import schedula
>>> import os.path as osp
>>> dsp = schedula.Dispatcher()
>>> dsp.add_data(data_id='dirname', default_value='.', initial_dist=2)
'dirname'
>>> dsp.add_function(function=osp.split, inputs=['path'],
...                  outputs=['dirname', 'basename'])
'split'
>>> dsp.add_function(function=osp.splitext, inputs=['basename'],
...                  outputs=['fname', 'suffix'])
'splitext'
>>> dsp.add_function(function=osp.join, inputs=['dirname', 'basename'],
...                  outputs=['path'])
'join'
>>> dsp.add_function(function_id='union', function=lambda *a: ''.join(a),
...                  inputs=['fname', 'suffix'], outputs=['basename'])
'union'

digraph dmap { graph [ratio=1] node [style=filled] label = <dmap> splines = ortho style = filled 493 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">basename</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=basename] 494 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">dirname</TD></TR><TR><TD align="RIGHT" border="1">default</TD><TD align="LEFT" border="1">.</TD></TR><TR><TD align="RIGHT" border="1">initial_dist</TD><TD align="LEFT" border="1">2</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=dirname] 495 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">fname</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=fname] 496 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-bda0a2f1d84ca3228debb5f4520122317594c6fe/join.html">join</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip="Join two or more pathname components, inserting '/' as needed."] 497 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">path</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=path] 498 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-bda0a2f1d84ca3228debb5f4520122317594c6fe/split.html">split</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip="Split a pathname."] 499 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-bda0a2f1d84ca3228debb5f4520122317594c6fe/splitext.html">splitext</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip="Split the extension from a pathname."] 500 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">suffix</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=suffix] 501 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-bda0a2f1d84ca3228debb5f4520122317594c6fe/lambda.html">union</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip=union] 499 -> 500 499 -> 495 498 -> 494 494 -> 496 493 -> 496 496 -> 497 498 -> 493 495 -> 501 497 -> 498 500 -> 501 501 -> 493 493 -> 499 }

Tip

You can explore the diagram by clicking on it.

Note

For more details how to created a Dispatcher() see: add_data(), add_function(), add_dispatcher(), SubDispatch(), SubDispatchFunction(), SubDispatchPipe(), and DFun().

The next step to calculate the outputs would be just to run the dispatch() method. You can invoke it with just the inputs, so it will calculate all reachable outputs:

>>> inputs = {'path': 'schedula/_version.py'}
>>> o = dsp.dispatch(inputs=inputs)
>>> o
Solution([('path', 'schedula/_version.py'),
          ('basename', '_version.py'),
          ('dirname', 'schedula'),
          ('fname', '_version'),
          ('suffix', '.py')])

digraph workflow { graph [ratio=1] node [style=filled] label = <workflow> splines = ortho style = filled 514 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-806cc650ed876917f9cd7a0cf9ff30f237c4569d/basename-output.html">basename</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">2.0</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=basename] 515 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-806cc650ed876917f9cd7a0cf9ff30f237c4569d/dirname-output.html">dirname</TD></TR><TR><TD align="RIGHT" border="1">default</TD><TD align="LEFT" border="1">.</TD></TR><TR><TD align="RIGHT" border="1">initial_dist</TD><TD align="LEFT" border="1">2</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">2.0</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=dirname] 516 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-806cc650ed876917f9cd7a0cf9ff30f237c4569d/fname-output.html">fname</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">4.0</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=fname] 517 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-806cc650ed876917f9cd7a0cf9ff30f237c4569d/path-output.html">path</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">0.0</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=path] 518 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-806cc650ed876917f9cd7a0cf9ff30f237c4569d/split.html">split</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">1.0</TD></TR><TR><TD align="RIGHT" border="1">started</TD><TD align="LEFT" border="1">2018-06-05T10:44:26.301212</TD></TR><TR><TD align="RIGHT" border="1">duration</TD><TD align="LEFT" border="1">0:00:00.000022</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip="Split a pathname."] 519 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-806cc650ed876917f9cd7a0cf9ff30f237c4569d/splitext.html">splitext</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">3.0</TD></TR><TR><TD align="RIGHT" border="1">started</TD><TD align="LEFT" border="1">2018-06-05T10:44:26.301422</TD></TR><TR><TD align="RIGHT" border="1">duration</TD><TD align="LEFT" border="1">0:00:00.000019</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip="Split the extension from a pathname."] 520 [label=start fillcolor=red shape=egg] 521 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-806cc650ed876917f9cd7a0cf9ff30f237c4569d/suffix-output.html">suffix</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">4.0</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=suffix] 520 -> 515 519 -> 521 518 -> 514 519 -> 516 520 -> 517 517 -> 518 518 -> 515 514 -> 519 }

or you can set also the outputs, so the dispatch will stop when it will find all outputs:

>>> o = dsp.dispatch(inputs=inputs, outputs=['basename'])
>>> o
Solution([('path', 'schedula/_version.py'), ('basename', '_version.py')])

digraph workflow { graph [ratio=1] node [style=filled] label = <workflow> splines = ortho style = filled 530 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-0311a927c48afe0c50f5bcf9ab2b65511411d6e9/basename-output.html">basename</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">2.0</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=basename] 531 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-0311a927c48afe0c50f5bcf9ab2b65511411d6e9/path-output.html">path</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">0.0</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=path] 532 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-0311a927c48afe0c50f5bcf9ab2b65511411d6e9/split.html">split</TD></TR><TR><TD align="RIGHT" border="1">M_outputs</TD><TD align="LEFT" border="1">(&#x27;dirname&#x27;,)</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">1.0</TD></TR><TR><TD align="RIGHT" border="1">started</TD><TD align="LEFT" border="1">2018-06-05T10:44:26.349526</TD></TR><TR><TD align="RIGHT" border="1">duration</TD><TD align="LEFT" border="1">0:00:00.000021</TD></TR></TABLE>> fillcolor=orange shape=box tooltip="Split a pathname."] 533 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-0311a927c48afe0c50f5bcf9ab2b65511411d6e9/splitext.html">splitext</TD></TR><TR><TD align="RIGHT" border="1">M_outputs</TD><TD align="LEFT" border="1">(&#x27;fname&#x27;, &#x27;suffix&#x27;)</TD></TR></TABLE>> fillcolor=orange shape=box tooltip="Split the extension from a pathname."] 534 [label=start fillcolor=red shape=egg] 534 -> 531 531 -> 532 532 -> 530 530 -> 533 }

5. Advanced example (circular system)

Systems of interdependent functions can be described by “graphs” and they might contains circles. This kind of system can not be resolved by a normal scheduler.

Suppose to have a system of sequential functions in circle - i.e., the input of a function is the output of the previous function. The maximum number of input and output permutations is \((2^n - 1)^2\), where n is the number of functions. Thus, with a normal scheduler you have to code all possible implementations, so \((2^n - 1)^2\) functions (IMPOSSIBLE!!!).

Schedula will simplify your life. You just create a Dispatcher(), that contains all functions that link your data:

>>> import schedula
>>> dsp = schedula.Dispatcher()
>>> plus, minus = lambda x: x + 1, lambda x: x - 1
>>> n = j = 6
>>> for i in range(1, n + 1):
...     func = plus if i < (n / 2 + 1) else minus
...     f = dsp.add_function('f%d' % i, func, ['v%d' % j], ['v%d' % i])
...     j = i

digraph dmap { graph [ratio=1] node [style=filled] label = <dmap> splines = curves style = filled 539 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-6d91fd83e051f77353a2f42a2cebfd32702341fb/lambda.html">f1</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip=f1] 540 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-6d91fd83e051f77353a2f42a2cebfd32702341fb/lambda-0.html">f2</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip=f2] 541 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-6d91fd83e051f77353a2f42a2cebfd32702341fb/lambda-1.html">f3</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip=f3] 542 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-6d91fd83e051f77353a2f42a2cebfd32702341fb/lambda-2.html">f4</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip=f4] 543 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-6d91fd83e051f77353a2f42a2cebfd32702341fb/lambda-3.html">f5</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip=f5] 544 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-6d91fd83e051f77353a2f42a2cebfd32702341fb/lambda-4.html">f6</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip=f6] 545 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v1</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v1] 546 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v2</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v2] 547 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v3</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v3] 548 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v4</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v4] 549 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v5</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v5] 550 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v6</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v6] 546 -> 541 549 -> 544 539 -> 545 545 -> 540 550 -> 539 543 -> 549 542 -> 548 540 -> 546 541 -> 547 544 -> 550 547 -> 542 548 -> 543 }

Then it will handle all possible combination of inputs and outputs (\((2^n - 1)^2\)) just invoking the dispatch() method, as follows:

>>> out = dsp.dispatch(inputs={'v1': 0, 'v4': 1}, outputs=['v2', 'v6'])
>>> out
Solution([('v1', 0), ('v4', 1), ('v2', 1), ('v5', 0), ('v6', -1)])

digraph workflow { node [style=filled] label = <workflow> splines = ortho style = filled 563 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-3c5653db8609422c1d79476e16b8fea9a04a0ead/lambda.html">f2</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">1.0</TD></TR><TR><TD align="RIGHT" border="1">started</TD><TD align="LEFT" border="1">2018-06-05T10:44:26.412670</TD></TR><TR><TD align="RIGHT" border="1">duration</TD><TD align="LEFT" border="1">0:00:00.000016</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip=f2] 564 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-3c5653db8609422c1d79476e16b8fea9a04a0ead/lambda-0.html">f5</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">1.0</TD></TR><TR><TD align="RIGHT" border="1">started</TD><TD align="LEFT" border="1">2018-06-05T10:44:26.412746</TD></TR><TR><TD align="RIGHT" border="1">duration</TD><TD align="LEFT" border="1">0:00:00.000014</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip=f5] 565 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-3c5653db8609422c1d79476e16b8fea9a04a0ead/lambda-1.html">f6</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">3.0</TD></TR><TR><TD align="RIGHT" border="1">started</TD><TD align="LEFT" border="1">2018-06-05T10:44:26.412891</TD></TR><TR><TD align="RIGHT" border="1">duration</TD><TD align="LEFT" border="1">0:00:00.000015</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip=f6] 566 [label=start fillcolor=red shape=egg] 567 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-3c5653db8609422c1d79476e16b8fea9a04a0ead/v1-output.html">v1</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">0.0</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v1] 568 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-3c5653db8609422c1d79476e16b8fea9a04a0ead/v2-output.html">v2</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">2.0</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v2] 569 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-3c5653db8609422c1d79476e16b8fea9a04a0ead/v4-output.html">v4</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">0.0</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v4] 570 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-3c5653db8609422c1d79476e16b8fea9a04a0ead/v5-output.html">v5</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">2.0</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v5] 571 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-3c5653db8609422c1d79476e16b8fea9a04a0ead/v6-output.html">v6</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">4.0</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v6] 570 -> 565 567 -> 563 566 -> 567 563 -> 568 569 -> 564 565 -> 571 566 -> 569 564 -> 570 }

5.1. Sub-system extraction

Schedula allows to extract sub-models from a model. This could be done with the shrink_dsp() method, as follows:

>>> sub_dsp = dsp.shrink_dsp(('v1', 'v3', 'v5'), ('v2', 'v4', 'v6'))

digraph dmap { node [style=filled] label = <dmap> splines = ortho style = filled 580 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-a9c94206f0b2600eb28affe444809eb53ac109b0/lambda.html">f2</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip=f2] 581 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-a9c94206f0b2600eb28affe444809eb53ac109b0/lambda-0.html">f4</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip=f4] 582 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-a9c94206f0b2600eb28affe444809eb53ac109b0/lambda-1.html">f6</TD></TR></TABLE>> fillcolor=springgreen shape=box tooltip=f6] 583 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v1</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v1] 584 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v2</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v2] 585 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v3</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v3] 586 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v4</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v4] 587 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v5</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v5] 588 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v6</TD></TR></TABLE>> fillcolor=cyan shape=box style="rounded,filled" tooltip=v6] 583 -> 580 587 -> 582 580 -> 584 582 -> 588 581 -> 586 585 -> 581 }

Note

For more details how to extract a sub-model see: get_sub_dsp(), get_sub_dsp_from_workflow(), SubDispatch(), SubDispatchFunction(), and SubDispatchPipe().

6. Next moves

Things yet to do include a mechanism to allow the execution of functions in parallel.