CNSM2020 branch first commit

.gitignore

@@ -0,0 +1,112 @@
+.idea
+
+# VS Code files
+.vscode
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Result files
+results/
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/

.travis.yml

@@ -0,0 +1,11 @@
+language: python
+python:
+    - "3.6"
+
+install:
+    - pip install -r requirements.txt
+
+script:
+    - flake8 src
+    - nose2
+    - coord-sim -d 20 -n params/networks/triangle.graphml -sf params/services/abc.yaml -sfr params/services/resource_functions -c params/config/sim_config.yaml

README.md

@@ -0,0 +1,110 @@
+[![Build Status](https://travis-ci.com/RealVNF/coord-sim.svg?branch=master)](https://travis-ci.com/RealVNF/coord-sim)
+
+# Simulation: Coordination of chained virtual network functions
+
+Simulate flow-level, inter-node network coordination including scaling and placement of services and scheduling/balancing traffic between them.
+
+<p align="center">
+    <img src="docs/realvnf_logo.png" height="150" hspace="30"/>
+  <img src="docs/upb.png" width="200" hspace="30"/>
+  <img src="docs/huawei_horizontal.png" width="250" hspace="30"/>
+</p>
+
+**Features**:
+
+- Simulate any given network topology with arbitrary node and link capacities and link delays
+- Simulatie any given network service consisting of linearly chained SFs/VNFs
+- VNFs can specify arbitrary resource consumption as function of their load using Python modules. Also VNF delay can be specified individually and may be normally distributed.
+- Simulate network traffic in the form of flow arrivals at various ingress nodes with varying arrival rate, flow length, volume, etc according to stochastic distributions
+- Simple and clear interface to run algorithms for scaling, placement, and scheduling/load balancing of these incoming flows across the nodes in the network. Coordination within each node is out of scope.
+- Interface allows easy integration with OpenAI Gym to enable training and evaluating reinforcement learning algorithms
+- Collection of metrics like successful/dropped flows, end-to-end delay, resource consumption, etc over time. Easily extensible.
+- Discrete event simulation to evaluate coordination over time with SimPy
+- Gracefull adjustment of placements: When VNFs are removed from a placement by an algorithm. Currently processing flows are allowed to finish processing before the VNF is completely removed (see PR [#78](https://github.com/RealVNF/coordination-simulation/pull/78) and [#81](https://github.com/RealVNF/coordination-simulation/pull/81)).
+
+## Setup
+
+Requires Python 3.6. Install with (ideally using [virtualenv](https://virtualenv.pypa.io/en/stable/)):
+
+```bash
+pip install -r requirements.txt
+```
+
+## Usage
+
+Type `coord-sim -h` for help using the simulator. For now, this should print
+
+```
+$ coord-sim -h
+usage: coord-sim [-h] -d DURATION -sf SF [-sfr SFR] -n NETWORK -c CONFIG
+                 [-t TRACE] [-s SEED]
+
+Coordination-Simulation tool
+
+optional arguments:
+  -h, --help            show this help message and exit
+  -d DURATION, --duration DURATION
+                        The duration of the simulation (simulates
+                        milliseconds).
+  -sf SF, --sf SF       VNF file which contains the SFCs and their respective
+                        SFs and their properties.
+  -sfr SFR, --sfr SFR   Path which contains the SF resource consumption
+                        functions.
+  -n NETWORK, --network NETWORK
+                        The GraphML network file that specifies the nodes and
+                        edges of the network.
+  -c CONFIG, --config CONFIG
+                        Path to the simulator config file.
+  -t TRACE, --trace TRACE
+                        Provide a CSV trace file to configure the traffic the
+                        simulator is generating.
+  -s SEED, --seed SEED  Random seed
+```
+
+You can use the following command as an example (run from the root project folder)
+
+```bash
+coord-sim -d 20 -n params/networks/triangle.graphml -sf params/services/abc.yaml -sfr params/services/resource_functions -c params/config/sim_config.yaml
+```
+
+This will run a simulation on a provided GraphML network file and a YAML placement file for a duration of 20 timesteps.
+
+### Dynamic SF resource consumption
+
+By default, all SFs have a node resource consumption, which exactly equals the aggregated traffic that they have to handle.
+
+It is possible to specify arbitrary other resource consumption models simply by implementing a python module with a
+function `resource_function(load)` (see examples [here](https://github.com/RealVNF/coordination-simulation/tree/master/params/services/resource_functions)).
+
+To use these modules, they need to be referenced in the service file:
+
+```
+sf_list:
+  a:
+    processing_delay_mean: 5.0
+    processing_delay_stdev: 0.0
+    resource_function_id: A
+```
+
+And the path to the folder with the Python modules needs to be passed via the `-sfr` argument.
+
+See PR https://github.com/RealVNF/coordination-simulation/pull/78 for details.
+
+## Tests
+
+```bash
+# style check
+flake8 src
+
+# tests
+nose2
+```
+
+## Acknowledgement
+
+This project has received funding from German Federal Ministry of Education and Research ([BMBF](https://www.bmbf.de/)) through Software Campus grant 01IS17046 ([RealVNF](https://realvnf.github.io/)).
+
+<p align="center">
+  <img src="docs/software_campus.png" width="200"/>
+  <img src="docs/BMBF_sponsored_by.jpg" width="250"/>
+</p>

docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/make.bat

@@ -0,0 +1,35 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=source
+set BUILDDIR=build
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.http://sphinx-doc.org/
+	exit /b 1
+)
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd

docs/simulator.md

@@ -0,0 +1,87 @@
+# Coordination Simulator
+Disclaimer: This document is currently in **draft** mode and includes the documentation for the project and is updated as features are added or modified. 
+
+Last update: 20.06.2019 by Haydar Qarawlus
+___
+
+Table of contents: 
+- Introduction
+- Application structure
+- Operation flow
+- Input parameters
+- Running the simulator
+
+## Introduction
+This discrete-event flow-based simulator is a fast testbed for VNF coordination algorithms. It is designed to simulate small to medium sized networks with relatively close-to-reality accuracy. It can interact with coordination algorithms via an interface that can be adapted to by the developers of the algorithms. 
+
+The simulator is based on SimPy and is tested with Python 3.6. 
+
+Project owner: Stefan Schneider, MSc, University of Paderborn. 
+
+Developers: 
+- Haydar Qarawlus, MSc. CS student, University of Paderborn. 
+- Adnan Manzoor, MSc. CS student, University of Paderborn. 
+- Sven Uthe, MSc, University of Paderbron. 
+
+
+
+
+## Application Structure
+The file structure of the simulator is as follows:
+- docs (Folder): Contains the doumentation files of the project.
+- params (Folder): Contains sample parameter files (network file and VNF file) for testing purposes. 
+- src (Folder): Contains the source code for the simulator and the interface.
+    - coordsim (Folder): contains the source code of the simulator
+        - metrics (Folder): contains the metrics module.
+        - network (Folder): contains the network module.
+        - reader (Folder): contains the params file reader module.
+        - simulation (Folder): main simulator module.
+        - main (Python): main executable for running the simulator from CLI
+    - siminterface (Folder): contains the interface source code
+- tests (Folder): contains the unit tests for the simulator.
+
+The main modules of the application are the following: 
+
+1. Simulation Module: This module contains FlowSimulator and SimulatorParams files. FlowSimulator is the main simulator that imports and utilizes the other modules. SimulatorParams is the class used to hold all the simulator parameters, this allows the parameters to be changed mid-simulation by the calling coordination algorithm.
+2. Network Module: This module contains the Flow descriptor class. this class holds the main properties of the flow such as the data rate and the requested SFC, etc.
+3. Reader Module: This module holds the network reader file that parses the parameter YAML and GraphML files that identify the network and the available SFCs and SFs. 
+4. Simulation Module: This module contains the main flow simulator. The flow simulator obtains its parameters from the SimulatotParams class. The params class is created by the calling coordination algorithm. 
+
+
+## Operation Flow
+The simulator works as follows:
+The user (coordination algorithm or cli) provide two main inputs for the simulator: 
+- Network file: GraphML file using the Zoo format. This file contains the network nodes and the edges.
+- VNF file: YAML file that includes the list of SFCs and the list of SFs under each SFC in an ordered manner. The file can also include a specifiecd placement that can be used as a default placement. The SFs must include a processing delay mean and standard deviation values so that processing delays can be calculated for each flow passing through that SF.
+
+Once the parameters are provided, the flow of data through the simulator is as follows:
+
+1. The input network and VNF files are parsed producing a NetworkX object containing the list of nodes and edges, and the shortest paths of the network (using Floyd-Warshall). The parsing also produces dictionaries that contain the list of SFCs and the list of SFs and their respective values. Additionally, the list of ingress nodes (nodes at which flows arrive) are also calculated from the GraphML file. These parameters are then passed to a SimulatorParams object, which holds all the parameters of the simulator, the simulator is then started using the FlowSimulator object's `start()` function
+2. At each ingress node, the function `generate_flow()` is called as a SimPy process, this function creates `Flow` objects with exponentially distributed random inter arrival times. The flow's data rate and size are generated using normally distributed random variables. All of the inter arrival time, data rate, and flow size parameters are user configurable. The flow is also assigned a random SFC chosen from the list of available SFC given in the VNF file.
+3. Once the flow is generated, `init_flow()` is called as a SimPy process which initializes the handling of the flow within the simulator. The function then calls `pass_flow()`, which then handles the scheduling of the flow according to the defined load balancing rules (flow schedule). Once the next node has been determined, the forwarding of the flow is simulated by halting the flow for the path delay duration using the `forward_flow()` function. Once that is done, the processing of the flow is simulated by calling `process_flow()` as a SimPy process. If the requested SF was not found at the next node, the flow is then dropped. 
+4. In `process_flow()`, the processing delay for that particular SF is generated using given mean and standard deviation values using a normal distribution. The simulator checks the node's remaining processing capacity to check if the node can handle the data rate requested by the SF, if there is not enough capacity, then the flow is dropped. For the duration that the flow is being processed by the SF, the flow's data rate is deducted from the node's capacity, and returned after the flow finished processing completely. 
+5. Once the flow was processed completely at each SF, `depart_flow()` is called to register the flow's departure from the network. If the flow still has other SFs to be processed at in the network, `process_flow()` calls `pass_flow()` again in a mutually recursive manner. This allows the flow to stay in the SF for processing, while the parts of the flow that were processed already to be sent to the next SF.
+
+
+## Input Parameters
+The available input parameters that are configurable by the user are:
+- d: The duration of the simulation (simulates milliseconds).
+- s: The seed to use for the random number generator. 
+- n: The GraphML network file that specifies the nodes and edges of the network. 
+- sf: VNF file which contains the SFCs and their respective SFs and their properties. 
+- iam: Inter arrival mean of the flows' arrival at ingress nodes.
+- fdm: The mean value for the generation of data rate values for each flow.
+- fds: The standard deviation value for the generation of data rate values for each flow.
+- fss: The shape of the Pareto distribution for the generation of the flow size values.
+
+## Running the simulator
+
+The simulator application is called `coord-sim`
+To run the simulator, the following call may be executed:
+
+```bash
+coord-sim -d 20 -n params/networks/triangle.graphml -sf params/services/abc.yaml -c params/config/sim_config.yaml
+```
+
+This will run the coord-sim simulator to simulate the given network and vnf parameter files for 10ms (environment time).
+