schedula: An intelligent function scheduler

Latest Version in PyPI Travis build status Code coverage Documentation status Dependencies up-to-date? Issues count Supported Python versions Project License Live Demo

release:

0.3.7

date:

2019-12-06 15:50:00

repository:

https://github.com/vinci1it2000/schedula

pypi-repo:

https://pypi.org/project/schedula/

docs:

http://schedula.readthedocs.io/

wiki:

https://github.com/vinci1it2000/schedula/wiki/

download:

http://github.com/vinci1it2000/schedula/releases/

keywords:

scheduling, dispatch, dataflow, processing, calculation, dependencies, scientific, engineering, simulink, graph theory

developers:
license:

EUPL 1.1+

What is schedula?

Schedula implements a intelligent function scheduler, which selects and executes functions. The order (workflow) is calculated from the provided inputs and the requested outputs. A function is executed when all its dependencies (i.e., inputs, input domain) are satisfied and when at least one of its outputs has to be calculated.

Note

Schedula is performing the runtime selection of the minimum-workflow to be invoked. A workflow describes the overall process - i.e., the order of function execution - and it is defined by a directed acyclic graph (DAG). The minimum-workflow is the DAG where each output is calculated using the shortest path from the provided inputs. The path is calculated on the basis of a weighed directed graph (data-flow diagram) with a modified Dijkstra algorithm.

Installation

To install it use (with root privileges):

$ pip install schedula

Or download the last git version and use (with root privileges):

$ python setup.py install

Install extras

Some additional functionality is enabled installing the following extras:

  • plot: enables the plot of the Dispatcher model and workflow (see plot()).
  • web: enables to build a dispatcher Flask app (see web()).
  • sphinx: enables the sphinx extension directives (i.e., autosummary and dispatcher).
  • parallel: enables the parallel execution of Dispatcher model.

To install schedula and all extras, do:

$ pip install schedula[all]

Note

plot extra requires Graphviz. Make sure that the directory containing the dot executable is on your systems’ path. If you have not you can install it from its download page.

What is schedula?

Schedula implements a intelligent function scheduler, which selects and executes functions. The order (workflow) is calculated from the provided inputs and the requested outputs. A function is executed when all its dependencies (i.e., inputs, input domain) are satisfied and when at least one of its outputs has to be calculated.

Note

Schedula is performing the runtime selection of the minimum-workflow to be invoked. A workflow describes the overall process - i.e., the order of function execution - and it is defined by a directed acyclic graph (DAG). The minimum-workflow is the DAG where each output is calculated using the shortest path from the provided inputs. The path is calculated on the basis of a weighed directed graph (data-flow diagram) with a modified Dijkstra algorithm.

Installation

To install it use (with root privileges):

$ pip install schedula

Or download the last git version and use (with root privileges):

$ python setup.py install
Install extras

Some additional functionality is enabled installing the following extras:

  • plot: enables the plot of the Dispatcher model and workflow (see plot()).
  • web: enables to build a dispatcher Flask app (see web()).
  • sphinx: enables the sphinx extension directives (i.e., autosummary and dispatcher).
  • parallel: enables the parallel execution of Dispatcher model.

To install schedula and all extras, do:

$ pip install schedula[all]

Note

plot extra requires Graphviz. Make sure that the directory containing the dot executable is on your systems’ path. If you have not you can install it from its download page.

Why may I use schedula?

Imagine we have a system of interdependent functions - i.e. the inputs of a function are the output for one or more function(s), and we do not know which input the user will provide and which output will request. With a normal scheduler you would have to code all possible implementations. I’m bored to think and code all possible combinations of inputs and outputs from a model.

Solution

Schedula allows to write a simple model (Dispatcher) with just the basic functions, then the Dispatcher will select and execute the proper functions for the given inputs and the requested outputs. Moreover, schedula provides a flexible framework for structuring code. It allows to extract sub-models from a bigger one and to run your model asynchronously or in parallel without extra coding.

Note

A successful application is CO2MPAS, where schedula has been used to model an entire vehicle.

Very simple example

Let’s assume that we have to extract some filesystem attributes and we do not know which inputs the user will provide. The code below shows how to create a Dispatcher adding the functions that define your system. Note that with this simple system the maximum number of inputs combinations is 31 (\((2^n - 1)\), where n is the number of data).

>>> import schedula as sh
>>> import os.path as osp
>>> dsp = sh.Dispatcher()
>>> dsp.add_data(data_id='dirname', default_value='.', initial_dist=2)
'dirname'
>>> dsp.add_function(function=osp.split, inputs=['path'],
...                  outputs=['dirname', 'basename'])
'split'
>>> dsp.add_function(function=osp.splitext, inputs=['basename'],
...                  outputs=['fname', 'suffix'])
'splitext'
>>> dsp.add_function(function=osp.join, inputs=['dirname', 'basename'],
...                  outputs=['path'])
'join'
>>> dsp.add_function(function_id='union', function=lambda *a: ''.join(a),
...                  inputs=['fname', 'suffix'], outputs=['basename'])
'union'

digraph dmap { graph [ratio=1] node [style=filled] label = <dmap> splines = ortho style = filled 587 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">basename</TD></TR></TABLE>> fillcolor=cyan id=3 shape=box style="rounded,filled" tooltip="\"basename\""] 588 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">dirname</TD></TR><TR><TD align="RIGHT" border="1">default</TD><TD align="LEFT" border="1">.</TD></TR><TR><TD align="RIGHT" border="1">initial_dist</TD><TD align="LEFT" border="1">2</TD></TR></TABLE>> fillcolor=cyan id=0 shape=box style="rounded,filled" tooltip="\"dirname\""] 589 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">fname</TD></TR></TABLE>> fillcolor=cyan id=5 shape=box style="rounded,filled" tooltip="\"fname\""] 590 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-bc3b4407269331934f472c55b7d6801b624c975c/join.html">join</TD></TR></TABLE>> fillcolor=springgreen id=7 shape=box tooltip="\"Join two or more pathname components, inserting '/' as needed.\""] 591 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">path</TD></TR></TABLE>> fillcolor=cyan id=2 shape=box style="rounded,filled" tooltip="\"path\""] 592 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-bc3b4407269331934f472c55b7d6801b624c975c/split.html">split</TD></TR></TABLE>> fillcolor=springgreen id=1 shape=box tooltip="\"Split a pathname.\""] 593 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-bc3b4407269331934f472c55b7d6801b624c975c/splitext.html">splitext</TD></TR></TABLE>> fillcolor=springgreen id=4 shape=box tooltip="\"Split the extension from a pathname.\""] 594 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">suffix</TD></TR></TABLE>> fillcolor=cyan id=6 shape=box style="rounded,filled" tooltip="\"suffix\""] 595 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-bc3b4407269331934f472c55b7d6801b624c975c/lambda.html">union</TD></TR></TABLE>> fillcolor=springgreen id=8 shape=box tooltip="\"union\""] 588 -> 590 592 -> 588 592 -> 587 591 -> 592 587 -> 593 587 -> 590 593 -> 589 593 -> 594 589 -> 595 594 -> 595 590 -> 591 595 -> 587 }

Tip

You can explore the diagram by clicking on it.

The next step to calculate the outputs would be just to run the dispatch() method. You can invoke it with just the inputs, so it will calculate all reachable outputs:

>>> inputs = {'path': 'schedula/_version.py'}
>>> o = dsp.dispatch(inputs=inputs)
>>> o
Solution([('path', 'schedula/_version.py'),
          ('basename', '_version.py'),
          ('dirname', 'schedula'),
          ('fname', '_version'),
          ('suffix', '.py')])

digraph workflow { graph [ratio=1] node [style=filled] label = <workflow> splines = ortho style = filled 608 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-fddc1ba9e634eb6a59f61af7eb46078afb299a12/basename-output.html">basename</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">2.0</TD></TR></TABLE>> fillcolor=cyan id=3 shape=box style="rounded,filled" tooltip="\"basename\""] 609 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-fddc1ba9e634eb6a59f61af7eb46078afb299a12/dirname-output.html">dirname</TD></TR><TR><TD align="RIGHT" border="1">default</TD><TD align="LEFT" border="1">.</TD></TR><TR><TD align="RIGHT" border="1">initial_dist</TD><TD align="LEFT" border="1">2</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">2.0</TD></TR></TABLE>> fillcolor=cyan id=0 shape=box style="rounded,filled" tooltip="\"dirname\""] 610 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-fddc1ba9e634eb6a59f61af7eb46078afb299a12/fname-output.html">fname</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">4.0</TD></TR></TABLE>> fillcolor=cyan id=5 shape=box style="rounded,filled" tooltip="\"fname\""] 611 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-fddc1ba9e634eb6a59f61af7eb46078afb299a12/path-output.html">path</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">0.0</TD></TR></TABLE>> fillcolor=cyan id=2 shape=box style="rounded,filled" tooltip="\"path\""] 612 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-fddc1ba9e634eb6a59f61af7eb46078afb299a12/split.html">split</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">1.0</TD></TR><TR><TD align="RIGHT" border="1">started</TD><TD align="LEFT" border="1">2019-12-06T17:55:02.941879</TD></TR><TR><TD align="RIGHT" border="1">duration</TD><TD align="LEFT" border="1">0:00:00.000033</TD></TR></TABLE>> fillcolor=springgreen id=1 shape=box tooltip="\"Split a pathname.\""] 613 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-fddc1ba9e634eb6a59f61af7eb46078afb299a12/splitext.html">splitext</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">3.0</TD></TR><TR><TD align="RIGHT" border="1">started</TD><TD align="LEFT" border="1">2019-12-06T17:55:02.942063</TD></TR><TR><TD align="RIGHT" border="1">duration</TD><TD align="LEFT" border="1">0:00:00.000021</TD></TR></TABLE>> fillcolor=springgreen id=4 shape=box tooltip="\"Split the extension from a pathname.\""] 614 [label=start fillcolor=red id=start shape=egg] 615 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-fddc1ba9e634eb6a59f61af7eb46078afb299a12/suffix-output.html">suffix</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">4.0</TD></TR></TABLE>> fillcolor=cyan id=6 shape=box style="rounded,filled" tooltip="\"suffix\""] 614 -> 611 614 -> 609 611 -> 612 612 -> 609 612 -> 608 608 -> 613 613 -> 610 613 -> 615 }

or you can set also the outputs, so the dispatch will stop when it will find all outputs:

>>> o = dsp.dispatch(inputs=inputs, outputs=['basename'])
>>> o
Solution([('path', 'schedula/_version.py'), ('basename', '_version.py')])

digraph workflow { graph [ratio=1] node [style=filled] label = <workflow> splines = ortho style = filled 624 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-44648e24f803bb1404af84a5e461f399753b9c66/basename-output.html">basename</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">2.0</TD></TR></TABLE>> fillcolor=cyan id=3 shape=box style="rounded,filled" tooltip="\"basename\""] 625 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-44648e24f803bb1404af84a5e461f399753b9c66/path-output.html">path</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">0.0</TD></TR></TABLE>> fillcolor=cyan id=2 shape=box style="rounded,filled" tooltip="\"path\""] 626 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-44648e24f803bb1404af84a5e461f399753b9c66/split.html">split</TD></TR><TR><TD align="RIGHT" border="1">M_outputs</TD><TD align="LEFT" border="1">(&#x27;dirname&#x27;,)</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">1.0</TD></TR><TR><TD align="RIGHT" border="1">started</TD><TD align="LEFT" border="1">2019-12-06T17:55:02.996090</TD></TR><TR><TD align="RIGHT" border="1">duration</TD><TD align="LEFT" border="1">0:00:00.000025</TD></TR></TABLE>> fillcolor=orange id=1 shape=box tooltip="\"Split a pathname.\""] 627 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-44648e24f803bb1404af84a5e461f399753b9c66/splitext.html">splitext</TD></TR><TR><TD align="RIGHT" border="1">M_outputs</TD><TD align="LEFT" border="1">(&#x27;fname&#x27;, &#x27;suffix&#x27;)</TD></TR></TABLE>> fillcolor=orange id=4 shape=box tooltip="\"Split the extension from a pathname.\""] 628 [label=start fillcolor=red id=start shape=egg] 628 -> 625 625 -> 626 626 -> 624 624 -> 627 }

Advanced example (circular system)

Systems of interdependent functions can be described by “graphs” and they might contains circles. This kind of system can not be resolved by a normal scheduler.

Suppose to have a system of sequential functions in circle - i.e., the input of a function is the output of the previous function. The maximum number of input and output permutations is \((2^n - 1)^2\), where n is the number of functions. Thus, with a normal scheduler you have to code all possible implementations, so \((2^n - 1)^2\) functions (IMPOSSIBLE!!!).

Schedula will simplify your life. You just create a Dispatcher, that contains all functions that link your data:

>>> import schedula as sh
>>> dsp = sh.Dispatcher()
>>> increment = lambda x: x + 1
>>> for k, (i, j) in enumerate(sh.pairwise([1, 2, 3, 4, 5, 6, 1])):
...     dsp.add_function('f%d' % k, increment, ['v%d' % i], ['v%d' % j])
'...'

digraph dmap { graph [ratio=1] node [style=filled] label = <dmap> splines = curves style = filled 633 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b4899571ee5e4178fa6ff7ffac3d13ed8ad48f20/lambda.html">f0</TD></TR></TABLE>> fillcolor=springgreen id=0 shape=box tooltip="\"f0\""] 634 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b4899571ee5e4178fa6ff7ffac3d13ed8ad48f20/lambda-0.html">f1</TD></TR></TABLE>> fillcolor=springgreen id=3 shape=box tooltip="\"f1\""] 635 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b4899571ee5e4178fa6ff7ffac3d13ed8ad48f20/lambda-1.html">f2</TD></TR></TABLE>> fillcolor=springgreen id=5 shape=box tooltip="\"f2\""] 636 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b4899571ee5e4178fa6ff7ffac3d13ed8ad48f20/lambda-2.html">f3</TD></TR></TABLE>> fillcolor=springgreen id=7 shape=box tooltip="\"f3\""] 637 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b4899571ee5e4178fa6ff7ffac3d13ed8ad48f20/lambda-3.html">f4</TD></TR></TABLE>> fillcolor=springgreen id=9 shape=box tooltip="\"f4\""] 638 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b4899571ee5e4178fa6ff7ffac3d13ed8ad48f20/lambda-4.html">f5</TD></TR></TABLE>> fillcolor=springgreen id=11 shape=box tooltip="\"f5\""] 639 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v1</TD></TR></TABLE>> fillcolor=cyan id=1 shape=box style="rounded,filled" tooltip="\"v1\""] 640 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v2</TD></TR></TABLE>> fillcolor=cyan id=2 shape=box style="rounded,filled" tooltip="\"v2\""] 641 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v3</TD></TR></TABLE>> fillcolor=cyan id=4 shape=box style="rounded,filled" tooltip="\"v3\""] 642 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v4</TD></TR></TABLE>> fillcolor=cyan id=6 shape=box style="rounded,filled" tooltip="\"v4\""] 643 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v5</TD></TR></TABLE>> fillcolor=cyan id=8 shape=box style="rounded,filled" tooltip="\"v5\""] 644 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v6</TD></TR></TABLE>> fillcolor=cyan id=10 shape=box style="rounded,filled" tooltip="\"v6\""] 633 -> 640 639 -> 633 640 -> 634 634 -> 641 641 -> 635 635 -> 642 642 -> 636 636 -> 643 643 -> 637 637 -> 644 644 -> 638 638 -> 639 }

Then it will handle all possible combination of inputs and outputs (\((2^n - 1)^2\)) just invoking the dispatch() method, as follows:

>>> out = dsp.dispatch(inputs={'v1': 0, 'v4': 1}, outputs=['v2', 'v6'])
>>> out
Solution([('v1', 0), ('v4', 1), ('v2', 1), ('v5', 2), ('v6', 3)])

digraph workflow { graph [bgcolor=transparent] node [style=filled] label = <workflow> splines = ortho style = filled 657 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b6f1955b896ae811e4ceb78e39de539282a5f967/lambda.html">f0</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">1.0</TD></TR><TR><TD align="RIGHT" border="1">started</TD><TD align="LEFT" border="1">2019-12-06T17:55:03.049857</TD></TR><TR><TD align="RIGHT" border="1">duration</TD><TD align="LEFT" border="1">0:00:00.000024</TD></TR></TABLE>> fillcolor=springgreen id=0 shape=box tooltip="\"f0\""] 658 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b6f1955b896ae811e4ceb78e39de539282a5f967/lambda-0.html">f3</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">1.0</TD></TR><TR><TD align="RIGHT" border="1">started</TD><TD align="LEFT" border="1">2019-12-06T17:55:03.049916</TD></TR><TR><TD align="RIGHT" border="1">duration</TD><TD align="LEFT" border="1">0:00:00.000015</TD></TR></TABLE>> fillcolor=springgreen id=7 shape=box tooltip="\"f3\""] 659 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b6f1955b896ae811e4ceb78e39de539282a5f967/lambda-1.html">f4</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">3.0</TD></TR><TR><TD align="RIGHT" border="1">started</TD><TD align="LEFT" border="1">2019-12-06T17:55:03.050010</TD></TR><TR><TD align="RIGHT" border="1">duration</TD><TD align="LEFT" border="1">0:00:00.000015</TD></TR></TABLE>> fillcolor=springgreen id=9 shape=box tooltip="\"f4\""] 660 [label=start fillcolor=red id=start shape=egg] 661 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b6f1955b896ae811e4ceb78e39de539282a5f967/v1-output.html">v1</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">0.0</TD></TR></TABLE>> fillcolor=cyan id=1 shape=box style="rounded,filled" tooltip="\"v1\""] 662 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b6f1955b896ae811e4ceb78e39de539282a5f967/v2-output.html">v2</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">2.0</TD></TR></TABLE>> fillcolor=cyan id=2 shape=box style="rounded,filled" tooltip="\"v2\""] 663 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b6f1955b896ae811e4ceb78e39de539282a5f967/v4-output.html">v4</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">0.0</TD></TR></TABLE>> fillcolor=cyan id=6 shape=box style="rounded,filled" tooltip="\"v4\""] 664 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b6f1955b896ae811e4ceb78e39de539282a5f967/v5-output.html">v5</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">2.0</TD></TR></TABLE>> fillcolor=cyan id=8 shape=box style="rounded,filled" tooltip="\"v5\""] 665 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-b6f1955b896ae811e4ceb78e39de539282a5f967/v6-output.html">v6</TD></TR><TR><TD align="RIGHT" border="1">distance</TD><TD align="LEFT" border="1">4.0</TD></TR></TABLE>> fillcolor=cyan id=10 shape=box style="rounded,filled" tooltip="\"v6\""] 660 -> 661 660 -> 663 661 -> 657 663 -> 658 657 -> 662 658 -> 664 664 -> 659 659 -> 665 }

Sub-system extraction

Schedula allows to extract sub-models from a model. This could be done with the shrink_dsp() method, as follows:

>>> sub_dsp = dsp.shrink_dsp(('v1', 'v3', 'v5'), ('v2', 'v4', 'v6'))

digraph dmap { graph [bgcolor=transparent] node [style=filled] label = <dmap> splines = ortho style = filled 674 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-f67d4bf5a0df282b4a5de8b05dd59251943bf699/lambda.html">f0</TD></TR></TABLE>> fillcolor=springgreen id=0 shape=box tooltip="\"f0\""] 675 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-f67d4bf5a0df282b4a5de8b05dd59251943bf699/lambda-0.html">f2</TD></TR></TABLE>> fillcolor=springgreen id=5 shape=box tooltip="\"f2\""] 676 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-f67d4bf5a0df282b4a5de8b05dd59251943bf699/lambda-1.html">f4</TD></TR></TABLE>> fillcolor=springgreen id=9 shape=box tooltip="\"f4\""] 677 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v1</TD></TR></TABLE>> fillcolor=cyan id=1 shape=box style="rounded,filled" tooltip="\"v1\""] 678 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v2</TD></TR></TABLE>> fillcolor=cyan id=2 shape=box style="rounded,filled" tooltip="\"v2\""] 679 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v3</TD></TR></TABLE>> fillcolor=cyan id=4 shape=box style="rounded,filled" tooltip="\"v3\""] 680 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v4</TD></TR></TABLE>> fillcolor=cyan id=6 shape=box style="rounded,filled" tooltip="\"v4\""] 681 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v5</TD></TR></TABLE>> fillcolor=cyan id=8 shape=box style="rounded,filled" tooltip="\"v5\""] 682 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">v6</TD></TR></TABLE>> fillcolor=cyan id=10 shape=box style="rounded,filled" tooltip="\"v6\""] 674 -> 678 675 -> 680 676 -> 682 677 -> 674 679 -> 675 681 -> 676 }

Note

For more details how to extract a sub-model see: get_sub_dsp(), get_sub_dsp_from_workflow(), SubDispatch, SubDispatchFunction, DispatchPipe, and SubDispatchPipe.

Iterated function

Schedula allows to build an iterated function, i.e. the input is recalculated. This could be done easily with the DispatchPipe, as follows:

>>> func = sh.DispatchPipe(dsp, 'func', ('v1', 'v4'), ('v1', 'v4'))
>>> x = [[1, 4]]
>>> for i in range(6):
...     x.append(func(*x[-1]))
>>> x
[[1, 4], [7, 4], [7, 10], [13, 10], [13, 16], [19, 16], [19, 22]]

Asynchronous and Parallel dispatching

When there are heavy calculations which takes a significant amount of time, you want to run your model asynchronously or in parallel. Generally, this is difficult to achieve, because it requires an higher level of abstraction and a deeper knowledge of python programming and the Global Interpreter Lock (GIL). Schedula will simplify again your life. It has four default executors to dispatch asynchronously or in parallel:

  • async: execute all functions asynchronously in the same process,
  • parallel: execute all functions in parallel excluding SubDispatch functions,
  • parallel-pool: execute all functions in parallel using a process pool excluding SubDispatch functions,
  • parallel-dispatch: execute all functions in parallel including SubDispatch.

Note

Running functions asynchronously or in parallel has a cost. Schedula will spend time creating / deleting new threads / processes.

The code below shows an example of a time consuming code, that with the concurrent execution it requires at least 6 seconds to run. Note that the slow function return the process id.

>>> import schedula as sh
>>> dsp = sh.Dispatcher()
>>> def slow():
...     import os, time
...     time.sleep(1)
...     return os.getpid()
>>> for o in 'abcdef':
...     dsp.add_function(function=slow, outputs=[o])
'...'

digraph dmap { graph [bgcolor=transparent] node [style=filled] label = <dmap> splines = ortho style = filled 689 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">a</TD></TR></TABLE>> fillcolor=cyan id=2 shape=box style="rounded,filled" tooltip="\"a\""] 690 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">b</TD></TR></TABLE>> fillcolor=cyan id=4 shape=box style="rounded,filled" tooltip="\"b\""] 691 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">c</TD></TR></TABLE>> fillcolor=cyan id=6 shape=box style="rounded,filled" tooltip="\"c\""] 692 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">d</TD></TR></TABLE>> fillcolor=cyan id=8 shape=box style="rounded,filled" tooltip="\"d\""] 693 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">e</TD></TR></TABLE>> fillcolor=cyan id=10 shape=box style="rounded,filled" tooltip="\"e\""] 694 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2">f</TD></TR></TABLE>> fillcolor=cyan id=12 shape=box style="rounded,filled" tooltip="\"f\""] 695 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-8da5d19fb6a74d9d1512dc8ab616fba9f0cfa6f8/slow.html">slow</TD></TR></TABLE>> fillcolor=springgreen id=1 shape=box tooltip="\"slow\""] 696 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-8da5d19fb6a74d9d1512dc8ab616fba9f0cfa6f8/slow-0.html">slow&lt;0&gt;</TD></TR></TABLE>> fillcolor=springgreen id=3 shape=box tooltip="\"slow<0>\""] 697 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-8da5d19fb6a74d9d1512dc8ab616fba9f0cfa6f8/slow-1.html">slow&lt;1&gt;</TD></TR></TABLE>> fillcolor=springgreen id=5 shape=box tooltip="\"slow<1>\""] 698 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-8da5d19fb6a74d9d1512dc8ab616fba9f0cfa6f8/slow-2.html">slow&lt;2&gt;</TD></TR></TABLE>> fillcolor=springgreen id=7 shape=box tooltip="\"slow<2>\""] 699 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-8da5d19fb6a74d9d1512dc8ab616fba9f0cfa6f8/slow-3.html">slow&lt;3&gt;</TD></TR></TABLE>> fillcolor=springgreen id=9 shape=box tooltip="\"slow<3>\""] 700 [label=<<TABLE border="0" cellspacing="0"><TR><TD border="0" colspan="2" href="./dispatcher-8da5d19fb6a74d9d1512dc8ab616fba9f0cfa6f8/slow-4.html">slow&lt;4&gt;</TD></TR></TABLE>> fillcolor=springgreen id=11 shape=box tooltip="\"slow<4>\""] 701 [label=start fillcolor=red id=start shape=egg] 701 -> 695 701 -> 696 701 -> 697 701 -> 698 701 -> 699 701 -> 700 695 -> 689 696 -> 690 697 -> 691 698 -> 692 699 -> 693 700 -> 694 }

while using the async executor, it lasts a bit more then 1 second:

>>> import time
>>> start = time.time()
>>> sol = dsp(executor='async').result()  # Asynchronous execution.
>>> (time.time() - start) < 2  # Faster then concurrent execution.
True

all functions have been executed asynchronously, but in the same process:

>>> import os
>>> pid = os.getpid()  # Current process id.
>>> {sol[k] for k in 'abcdef'} == {pid}  # Single process id.
True

if we use the parallel executor all functions are executed in different processes:

>>> sol = dsp(executor='parallel').result()  # Parallel execution.
>>> pids = {sol[k] for k in 'abcdef'}  # Process ids returned by `slow`.
>>> len(pids) == 6  # Each function returns a different process id.
True
>>> pid not in pids  # The current process id is not in the returned pids.
True
>>> sorted(sh.shutdown_executors())
['async', 'parallel']

Next moves

Things yet to do: utility to transform a dispatcher in a command line tool.

API Reference

The core of the library is composed from the following modules:

It contains a comprehensive list of all modules and classes within schedula.

Docstrings should provide sufficient understanding for any individual function.

Modules:

dispatcher It provides Dispatcher class.
utils It contains utility classes and functions.
ext It provides sphinx extensions.

Changelog

v0.3.7 (2019-12-06)
Feat
  • (drw): Update the index GUI of the plot.
  • (appveyor): Drop appveyor in favor of travis.
  • (travis): Update travis configuration file.
  • (plot): Add node link and id in graph plot.
Fix
  • (drw): Render dot in temp folder.
  • (plot): Add quiet arg to _view method.
  • (doc): Correct missing gh links.
  • (core) #17: Correct deprecated Graph attribute.
v0.3.6 (2019-10-18)
Fix
  • (setup) #17: Update version networkx.
  • (setup) #13: Build universal wheel.
  • (alg) #15: Escape % in node id.
  • (setup) #14: Update tests requirements.
  • (setup): Add env ENABLE_SETUP_LONG_DESCRIPTION.
v0.3.4 (2019-07-15)
Feat
  • (binder): Add @jupyterlab/plotly-extension.
  • (binder): Customize Site._repr_html_ with env SCHEDULA_SITE_REPR_HTML.
  • (binder): Add jupyter-server-proxy.
  • (doc): Add binder examples.
  • (gen): Create super-class of Token.
  • (dsp): Improve error message.
Fix
  • (binder): Simplify processing_chain example.
  • (setup): Exclude binder and examples folders as packages.
  • (doc): Correct binder data.
  • (doc): Update examples for binder.
  • (doc): Add missing requirements binder.
  • (test): Add state to fake directive.
  • (import): Remove stub file to enable autocomplete.
  • Update to canonical pypi name of beautifulsoup4.
v0.3.3 (2019-04-02)
Feat
  • (dispatcher): Improve error message.
Fix
  • (doc): Correct bug for sphinx AutoDirective.
  • (dsp): Add dsp as kwargs for a new Blueprint.
  • (doc): Update PEP and copyright.
v0.3.2 (2019-02-23)
Feat
  • (core): Add stub file.
  • (sphinx): Add Blueprint in Dispatcher documenter.
  • (sphinx): Add BlueDispatcher in documenter.
  • (doc): Add examples.
  • (blue): Customizable memo registration of blueprints.
Fix
  • (sphinx): Correct bug when is in csv-table directive.
  • (core): Set module attribute when __getattr__ is invoked.
  • (doc): Correct utils description.
  • (setup): Improve keywords.
  • (drw): Correct tooltip string format.
  • (version): Correct import.
v0.3.1 (2018-12-10)
Fix
  • (setup): Correct long description for pypi.
  • (dsp): Correct bug DispatchPipe when dill.
v0.3.0 (2018-12-08)
Feat
  • (blue, dispatcher): Add method extend to extend Dispatcher or Blueprint with Dispatchers or Blueprints.
  • (blue, dsp): Add BlueDispatcher class + remove DFun util.
  • (core): Remove weight attribute from Dispatcher struc.
  • (dispatcher): Add method add_func to Dispatcher.
  • (core): Remove remote_links attribute from dispatcher data nodes.
  • (core): Implement callable raise option in Dispatcher.
  • (core): Add feature to dispatch asynchronously and in parallel.
  • (setup): Add python 3.7.
  • (dsp): Use the same dsp.solution class in SubDispatch functions.
Fix
  • (dsp): Do not copy solution when call DispatchPipe, but reset solution when copying the obj.
  • (alg): Correct and clean get_sub_dsp_from_workflow algorithm.
  • (sol): Ensure bool output from input_domain call.
  • (dsp): Parse arg and kw using SubDispatchFunction.__signature__.
  • (core): Do not support python 3.4.
  • (asy): Do not dill the Dispatcher solution.
  • (dispatcher): Correct bug in removing remote links.
  • (core): Simplify and correct Exception handling.
  • (dsp): Postpone __signature__ evaluation in add_args.
  • (gen): Make Token constant when pickled.
  • (sol): Move callback invocation in _evaluate_node.
  • (core) #11: Lazy import of modules.
  • (sphinx): Remove warnings.
  • (dsp): Add missing code option in add_function decorator.
Other
  • Refact: Update documentation.
v0.2.8 (2018-10-09)
Feat
  • (dsp): Add inf class to model infinite numbers.
v0.2.7 (2018-09-13)
Fix
  • (setup): Correct bug when long_description fails.
v0.2.6 (2018-09-13)
Feat
  • (setup): Patch to use sphinxcontrib.restbuilder in setup long_description.
v0.2.5 (2018-09-13)
Fix
  • (doc): Correct link docs_status.
  • (setup): Use text instead rst to compile long_description + add logging.
v0.2.4 (2018-09-13)
Fix
  • (sphinx): Correct bug sphinx==1.8.0.
  • (sphinx): Remove all sphinx warnings.
v0.2.3 (2018-08-02)
Fix
  • (des): Correct bug when SubDispatchFunction have no outputs.
v0.2.2 (2018-08-02)
Fix
  • (des): Correct bug of get_id when tuple ids nodes are given as input or outputs of a sub_dsp.
  • (des): Correct bug when tuple ids are given as inputs or outputs of add_dispatcher method.
v0.2.1 (2018-07-24)
Feat
  • (setup): Update Development Status to 5 - Production/Stable.
  • (setup): Add additional project_urls.
  • (doc): Add changelog to rtd.
Fix
  • (doc): Correct link docs_status.
  • (des): Correct bugs get_des.
v0.2.0 (2018-07-19)
Feat
  • (doc): Add changelog.
  • (travis): Test extras.
  • (des): Avoid using sphinx for getargspec.
  • (setup): Add extras_require to setup file.
Fix
  • (setup): Correct bug in get_long_description.
v0.1.19 (2018-06-05)
Fix
  • (dsp): Add missing content block in note directive.
  • (drw): Make sure to plot same sol as function and as node.
  • (drw): Correct format of started attribute.
v0.1.18 (2018-05-28)
Feat
  • (dsp): Add DispatchPipe class (faster pipe execution, it overwrite the existing solution).
  • (core): Improve performances replacing datetime.today() with time.time().
v0.1.17 (2018-05-18)
Feat
  • (travis): Run coveralls in python 3.6.
Fix
  • (web): Skip Flask logging for the doctest.
  • (ext.dispatcher): Update to the latest Sphinx 1.7.4.
  • (des): Use the proper dependency (i.e., sphinx.util.inspect) for getargspec.
  • (drw): Set socket option to reuse the address (host:port).
  • (setup): Correct dill requirements dill>=0.2.7.1 –> dill!=0.2.7.
v0.1.16 (2017-09-26)
Fix
  • (requirements): Update dill requirements.
v0.1.15 (2017-09-26)
Fix
  • (networkx): Update according to networkx 2.0.
v0.1.14 (2017-07-11)
Fix
  • (io): pin dill version <=0.2.6.
  • (abort): abort was setting Exception.args instead of sol attribute.
Other
  • Merge pull request #9 from ankostis/fixabortex.
v0.1.13 (2017-06-26)
Feat
  • (appveyor): Add python 3.6.
Fix
  • (install): Force update setuptools>=36.0.1.
  • (exc): Do not catch KeyboardInterrupt exception.
  • (doc) #7: Catch exception for sphinx 1.6.2 (listeners are moved in EventManager).
  • (test): Skip empty error message.
v0.1.12 (2017-05-04)
Fix
  • (drw): Catch dot error and log it.
v0.1.11 (2017-05-04)
Feat
  • (dsp): Add add_function decorator to add a function to a dsp.
  • (dispatcher) #4: Use kk_dict function to parse inputs and outputs of add_dispatcher method.
  • (dsp) #4: Add kk_dict function.
Fix
  • (doc): Replace type function with callable.
  • (drw): Folder name without ext.
  • (test): Avoid Documentation of DspPlot.
  • (doc): fix docstrings types.
v0.1.10 (2017-04-03)
Feat
  • (sol): Close sub-dispatcher solution when all outputs are satisfied.
Fix
  • (drw): Log error when dot is not able to render a graph.
v0.1.9 (2017-02-09)
Fix
  • (appveyor): Setup of lmxl.
  • (drw): Update plot index.
v0.1.8 (2017-02-09)
Feat
  • (drw): Update plot index + function code highlight + correct plot outputs.
v0.1.7 (2017-02-08)
Fix
  • (setup): Add missing package_data.
v0.1.6 (2017-02-08)
Fix
  • (setup): Avoid setup failure due to get_long_description.
  • (drw): Avoid to plot unneeded weight edges.
  • (dispatcher): get_sub_dsp_from_workflow set correctly the remote links.
v0.1.5 (2017-02-06)
Feat
  • (exl): Drop exl module because of formulas.
  • (sol): Add input value of filters in solution.
Fix
  • (drw): Plot just one time the filer attribute in workflow +filers|solution_filters .
v0.1.4 (2017-01-31)
Feat
  • (drw): Save autoplot output.
  • (sol): Add filters and function solutions to the workflow nodes.
  • (drw): Add filters to the plot node.
Fix
  • (dispatcher): Add missing function data inputs edge representation.
  • (sol): Correct value when apply filters on setting the node output.
  • (core): get_sub_dsp_from_workflow blockers can be applied to the sources.
v0.1.3 (2017-01-29)
Fix
  • (dsp): Raise a DispatcherError when the pipe workflow is not respected instead KeyError.
  • (dsp): Unresolved references.
v0.1.2 (2017-01-28)
Feat
  • (dsp): add_args _set_doc.
  • (dsp): Remove parse_args class.
  • (readme): Appveyor badge status == master.
  • (dsp): Add _format option to get_unused_node_id.
  • (dsp): Add wildcard option to SubDispatchFunction and SubDispatchPipe.
  • (drw): Create sub-package drw.
Fix
  • (dsp): combine nested dicts with different length.
  • (dsp): are_in_nested_dicts return false if nested_dict is not a dict.
  • (sol): Remove defaults when setting wildcards.
  • (drw): Misspelling outpus –> outputs.
  • (directive): Add exception on graphviz patch for sphinx 1.3.5.
v0.1.1 (2017-01-21)
Fix
  • (site): Fix ResourceWarning: unclosed socket.
  • (setup): Not log sphinx warnings for long_description.
  • (travis): Wait util the server is up.
  • (rtd): Missing requirement dill.
  • (travis): Install first - pip install -r dev-requirements.txt.
  • (directive): Tagname from _img to img.
  • (directive): Update minimum sphinx version.
  • (readme): Badge svg links.
Other
  • Add project descriptions.
  • (directive): Rename schedula.ext.dsp_directive –> schedula.ext.dispatcher.
  • Update minimum sphinx version and requests.

Indices and tables