diff --git a/ANNOUNCE.md b/ANNOUNCE.md new file mode 100644 index 0000000..4704c55 --- /dev/null +++ b/ANNOUNCE.md @@ -0,0 +1,29 @@ +Ansible Network: Network Engine Release +---------------------------------------- + +The Ansible Network team is pleased to announce that the initial release of the Network Engine Ansible role is now available in Ansible Galaxy! + +What is an Ansible Role? +---------------------------------- +An Ansible Role is a collection of related tasks, methods, plugins, and modules in a standard format. You can use Roles in tasks or playbooks. + +What does the Network Engine Role do? +---------------------------------- +The Network Engine Role provides the fundamental building blocks for a data-model-driven approach to automated network management. Network Engine: + + - extracts data about your network devices + - returns the data as Ansible facts in a JSON data structure, ready to be added to your inventory host facts and/or consumed by Ansible tasks and templates + - works on any network platform + +With the Network Engine role, and other Roles built around it, you can normalize your Ansible facts across your entire network. + +How do I get it? +---------------------------------- +Via Ansible Galaxy using the following Linux command: + +`ansible-galaxy install ansible-network.network-engine` + +How do I use it? +---------------------------------- +See the [User Guide](https://github.com/ansible-network/network-engine/blob/devel/docs/user_guide/main.md) for details and examples. + diff --git a/CHANGELOG.rst b/CHANGELOG.rst index d130a48..bb2a7c6 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -18,6 +18,25 @@ Bugfixes - Fix StringIO to work with Python3 in addition to Python2 (https://github.com/ansible-network/network-engine/pull/53) +.. _Ansible Network: network-engine_v2.5.1_Deprecated Features: + +Deprecated Features +------------------- + +- Module ``text_parser`` renamed to ``command_parser``; original name deprecated; legacy use supported; will be removed in 2.6.0. + +- Module ``textfsm`` renamed to ``textfsm_parser``; original name deprecated; legacy use supported; will be removed in 2.6.0. + + +.. _Ansible Network: network-engine_v2.5.1_New Modules: + +New Modules +----------- + +- New module ``command_parser`` (renamed from ``text_parser``) + +- New module ``textfsm_parser`` (renamed from ``textfsm``) + .. _Ansible Network: network-engine_v2.5.0: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b8585e9..b16aa36 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -7,7 +7,7 @@ role. ## Contributing -There are many ways you can contribute to this role. Adding new artefacts such +There are many ways you can contribute to this role. Adding new artifacts such as modules and plugins, testing and/or reviewing and updating documentation. ### Adding support for a new platform diff --git a/README.md b/README.md index 04db122..fa667c5 100644 --- a/README.md +++ b/README.md @@ -3,11 +3,15 @@ [![network-engine Ansible Galaxy Role](https://img.shields.io/ansible/role/25206.svg)](https://galaxy.ansible.com/ansible-network/network-engine/) This role provides the foundation for building network roles by providing -modules and plugins that are common to all Ansible Network roles. All of -the artifacts in this role can be used independent of the platform that is -being managed. +modules and plugins that are common to all Ansible Network roles. The role +is platform-agnostic - all of the artifacts in this role can be used on any +Ansible-managed network platform. -To use this role, install it from [Galaxy](https://galaxy.ansible.com/ansible-network/network-engine/). To find other roles maintained by the Ansible Network team, see our [Galaxy Profile](https://galaxy.ansible.com/ansible-network/). +To install this role: `ansible-galaxy install ansible-network.network-engine`. +To see the version of this role you currently have installed: `ansible-galaxy list`, then find `network-engine` in the results. +To ensure you have the latest version available: `ansible-galaxy install -f ansible-network.network-engine`. +To use this role, follow the [User Guide](https://github.com/ansible-network/network-engine/blob/devel/docs/user_guide/main.md). +To find other roles maintained by the Ansible Network team, see our [Galaxy Profile](https://galaxy.ansible.com/ansible-network/). Any open bugs and/or feature requests are tracked in [GitHub issues](https://github.com/ansible-network/network-engine/issues). @@ -15,7 +19,9 @@ Interested in contributing to this role? Check out [CONTRIBUTING](https://github ## Documentation -* User guide: [Parser Directives](https://github.com/ansible-network/network-engine/blob/devel/docs/directives/parser_directives.md) +* User guide: + - [How to use](https://github.com/ansible-network/network-engine/blob/devel/docs/user_guide/main.md) + - [Parser Directives](https://github.com/ansible-network/network-engine/blob/devel/docs/directives/parser_directives.md) * Development guide: [How to test](https://github.com/ansible-network/network-engine/blob/devel/docs/tests/test_guide.md) For module documentation see the [modules](#modules) section below. @@ -42,8 +48,8 @@ None The following is a list of modules that are provided by this role, which include documentation & examples: * `cli` [source](https://github.com/ansible-network/network-engine/blob/devel/action_plugins/cli.py) -* `text_parser` [source](https://github.com/ansible-network/network-engine/blob/devel/library/text_parser.py) -* `textfsm` [source](https://github.com/ansible-network/network-engine/blob/devel/library/textfsm.py) +* `command_parser` [source](https://github.com/ansible-network/network-engine/blob/devel/library/command_parser.py) +* `textfsm_parser` [source](https://github.com/ansible-network/network-engine/blob/devel/library/textfsm_parser.py) ## Plugins diff --git a/changelogs/fragments/v251-terminology-changes.yaml b/changelogs/fragments/v251-terminology-changes.yaml new file mode 100644 index 0000000..3faef0d --- /dev/null +++ b/changelogs/fragments/v251-terminology-changes.yaml @@ -0,0 +1,7 @@ +new_modules: +- New module ``command_parser`` (renamed from ``text_parser``) +- New module ``textfsm_parser`` (renamed from ``textfsm``) + +deprecated_features: +- Module ``text_parser`` renamed to ``command_parser``; original name deprecated; legacy use supported; will be removed in 2.6.0. +- Module ``textfsm`` renamed to ``textfsm_parser``; original name deprecated; legacy use supported; will be removed in 2.6.0. diff --git a/docs/directives/parser_directives.md b/docs/directives/parser_directives.md index 633db35..5442188 100644 --- a/docs/directives/parser_directives.md +++ b/docs/directives/parser_directives.md @@ -3,15 +3,15 @@ The `command_parser` module is a module that can be used to parse the results of text strings into Ansible facts. The primary motivation for developing the `command_parser` module is to convert structured ASCII text output (such as -those returned from network devices) into JSON data structures suitable to be +the stdout returned from network devices) into JSON data structures suitable to be used as host facts. -The parser file format is loosely based on the Ansible playbook directives +The parser template file format is loosely based on the Ansible playbook directives language. It uses the Ansible directive language to ease the transition from -writing playbooks to writing parsers. However, parsers developed using this +writing playbooks to writing parser templates. However, parser templates developed using this module are not written directly into the playbook, but are a separate file called from playbooks. This is done for a variety of reasons but most notably -to keep separation from the parsing logical and playbook execution. +to keep separation between the parsing logic and playbook execution. The `command_parser` works based on a set of directives that perform actions on structured data with the end result being a valid JSON structure that can be @@ -19,7 +19,7 @@ returned to the Ansible facts system. ## Parser language -The parser format uses YAML formatting, providing an ordered list of directives +The parser template format uses YAML formatting, providing an ordered list of directives to be performed on the content (provided by the module argument). The overall general structure of a directive is as follows: @@ -70,7 +70,7 @@ configured on any directive. ### `name` -All entries in the parser file many contain a `name` directive. The +All entries in the parser template many contain a `name` directive. The `name` directive can be used to provide an arbitrary description as to the purpose of the parser items. The use of `name` is optional for all directives. @@ -79,14 +79,18 @@ The default value for `name` is `null`. ### `register` -The `register` directive option can be used same as would be used in an -Ansible playbook. It will register the results of the directive operation into -the named variable such that it can be retrieved later. +Use the `register` option to register the results of a directive operation +temporarily into the variable name you specify +so you can retrieve it later in your parser template. You use `register` in +a parser template just as you would in an Ansible playbook. -However, be sure to make note that registered variables are not available -outside of the parser context. Any values registered are only available within -the scope of the parser activities. If you want to provide values back to the -playbook, you will have to define the [export](#export) option. +Variables created with `register` alone are not available outside of the parser context. +Any values registered are only available within the scope of the parser activities. +If you want to provide values back to the playbook, you must also define the [export](#export) option. + +Typically you will use `register` alone for parsing each individual part of the +command output, then amalgamate them into a single variable at the end of the parser template, +register that variable and set `export: yes` on it. The default value for `register` is `null`. @@ -94,7 +98,7 @@ The default value for `register` is `null`. ### `export` -This option will allow any value to be exported back the calling task as an +Use the `export` option to export any value back to the calling task as an Ansible fact. The `export` option accepts a boolean value that defines if the registered fact should be exported to the calling task in the playbook (or role) scope. To export the value, simply set `export` to True. @@ -106,8 +110,9 @@ The default value for `export` is `False`. ### `export_as` -This option will allow value to be exported back in the calling task as an -Ansible fact in mentioned format. The `export_as` option can be used to define the structure of the exported data, which accepts: +Use the `export_as` option to export a value back to the calling task as an +Ansible fact in a specific format. The `export_as` option defines the structure of the exported data. +Accepted values for `export_as`: * `dict` * `hash` @@ -119,8 +124,8 @@ Ansible fact in mentioned format. The `export_as` option can be used to define t ### loop -Sometimes it is necessary to loop over a directive in order to process values. -Using the `loop` option, the parser will iterate over the directive and +Use the `loop` option to loop over a directive in order to process values. +With the `loop` option, the parser will iterate over the directive and provide each of the values provided by the loop content to the directive for processing. @@ -131,11 +136,11 @@ access `{{ item.key }}` and `{{ item.value }}`. ### `loop_control` -`loop_control` option can be used to specify the name of the variable to be +Use the `loop_control` option to specify the name of the variable to be used for the loop instead of default loop variable `item`. When looping over a hash, you can access `{{ foo.key }}` and `{{ foo.value }}` where `foo` is `loop_var`. -The general structure of `loop_control` is as follow: +The general structure of `loop_control` is as follows: ```yaml - name: User defined variable @@ -150,13 +155,13 @@ The general structure of `loop_control` is as follow: ### `when` -The `when` option allows for a conditional to be placed on the directive to +Use the `when` option to place a condition on the directive to decided if it is executed or not. The `when` option operates the same as it would in an Ansible playbook. -For example, let's assume we only want to match perform the match statement -when the value of `ansible_network_os` is set to `ios`. We would apply -the `when` conditional as such: +For example, if you only want to perform the match statement +when the value of `ansible_network_os` is set to `ios`, you can apply +the `when` conditional like this: ```yaml - name: conditionally matched var @@ -173,7 +178,7 @@ can be used to perform its operation. ### `pattern_match` -The `pattern_match` directive is used to extract one or more values from +Use the `pattern_match` directive to extract one or more values from the structured ASCII text based on regular expressions. The following arguments are supported for this directive: @@ -185,7 +190,7 @@ The following arguments are supported for this directive: ### `pattern_group` -The `pattern_group` directive can be used to group multiple +Use the `pattern_group` directive to group multiple `pattern_match` results together. The following arguments are supported for this directive: @@ -196,7 +201,7 @@ The following arguments are supported for this directive: ### `json_template` -The `json_template` directive will create a JSON data structure based on a +Use the `json_template` directive to create a JSON data structure based on a template. This directive will allow you to template out a multi-level JSON blob. @@ -206,12 +211,11 @@ The following arguments are supported for this directive: ### `set_vars` -The `set_vars` directive will set variables to the values like key / value pairs -and returns dictionary. +Use the `set_vars` directive to set variables to the values like key / value pairs +and return a dictionary. ### `export_facts` -The `export_facts` directive takes an arbitrary set of key / value pairs -and exposes (returns) them back to the playbook global namespace. Any key / +Use the `export_facts` directive to take an arbitrary set of key / value pairs +and expose (return) them back to the playbook global namespace. Any key / value pairs that are provided in this directive become available on the host. - diff --git a/docs/tests/test_guide.md b/docs/tests/test_guide.md index de637dc..90685ff 100644 --- a/docs/tests/test_guide.md +++ b/docs/tests/test_guide.md @@ -43,7 +43,7 @@ If you add any new Role for test, make sure to include the role in `test.yml`: ## Add new platforms tests to an existing roles -Create directory with the `platform_name` in `output` and `parsers` directories +Create directory with the `platform_name` in `output` and `parser_templates` directories which will contain output and parser files of the platform. Add corresponding playbook with the `platform_name` in `tasks/$platform_name.yaml` diff --git a/docs/user_guide/command_parser.md b/docs/user_guide/command_parser.md new file mode 100644 index 0000000..03670fc --- /dev/null +++ b/docs/user_guide/command_parser.md @@ -0,0 +1,223 @@ +# command_parser + +The [command_parser](https://github.com/ansible-network/network-engine/blob/devel/library/command_parser.py) +module is closely modeled after the Ansible playbook language. +This module iterates over matching rules defined in YAML format, extracts data from structured ASCII text based on those rules, +and returns Ansible facts in a JSON data structure that can be added to the inventory host facts and/or consumed by Ansible tasks and templates. + +The `command_parser` module requires two inputs: + - the output of commands run on the network device, passed to the `content` parameter + - the parser template that defines the rules for parsing the output, passed to either the `file` or the `dir` parameter + +## Parameters + +### content + +The `content` parameter for `command_parser` must point to the ASCII text output of commands run on network devices. The text output can be in a variable or in a file. + + +### file + +The `file` parameter for `command_parser` must point to a parser template that contains a rule for each data field you want to extract from your network devices. + +Parser templates for the `command_parser` module in the Network Engine role use YAML notation. + + +### dir + +Points to a directory containing parser templates. Use this parameter instead of `file` if your playbook uses multiple parser templates. + +## Sample Parser Templates + +Parser templates for the `command_parser` module in the Network Engine role use YAML syntax. +To write a parser template, follow the [parser_directives documentation](docs/directives/parser_directives.md). + +Here are two sample YAML parser templates: + +`parser_templates/ios/show_interfaces.yaml` +```yaml + +--- +- name: parser meta data + parser_metadata: + version: 1.0 + command: show interface + network_os: ios + +- name: match sections + pattern_match: + regex: "^(\\S+) is up," + match_all: yes + match_greedy: yes + register: section + +- name: match interface values + pattern_group: + - name: match name + pattern_match: + regex: "^(\\S+)" + content: "{{ item }}" + register: name + + - name: match hardware + pattern_match: + regex: "Hardware is (\\S+)," + content: "{{ item }}" + register: type + + - name: match mtu + pattern_match: + regex: "MTU (\\d+)" + content: "{{ item }}" + register: mtu + + - name: match description + pattern_match: + regex: "Description: (.*)" + content: "{{ item }}" + register: description + loop: "{{ section }}" + register: interfaces + +- name: generate json data structure + json_template: + template: + - key: "{{ item.name.matches.0 }}" + object: + - key: config + object: + - key: name + value: "{{ item.name.matches.0 }}" + - key: type + value: "{{ item.type.matches.0 }}" + - key: mtu + value: "{{ item.mtu.matches.0 }}" + - key: description + value: "{{ item.description.matches.0 }}" + loop: "{{ interfaces }}" + export: yes + register: interface_facts + +``` + +`parser_templates/ios/show_version.yaml` + +```yaml + +--- +- name: parser meta data + parser_metadata: + version: 1.0 + command: show version + network_os: ios + +- name: match version + pattern_match: + regex: "Version (\\S+)," + register: version + +- name: match model + pattern_match: + regex: "^Cisco (.+) \\(revision" + register: model + +- name: match image + pattern_match: + regex: "^System image file is (\\S+)" + register: image + +- name: match uptime + pattern_match: + regex: "uptime is (.+)" + register: uptime + +- name: match total memory + pattern_match: + regex: "with (\\S+)/(\\w*) bytes of memory" + register: total_mem + +- name: match free memory + pattern_match: + regex: "with \\w*/(\\S+) bytes of memory" + register: free_mem + +- name: export system facts to playbook + set_vars: + model: "{{ model.matches.0 }}" + image_file: "{{ image.matches.0 }}" + uptime: "{{ uptime.matches.0 }}" + version: "{{ version.matches.0 }}" + memory: + total: "{{ total_mem.matches.0 }}" + free: "{{ free_mem.matches.0 }}" + export: yes + register: system_facts + +``` + +## Sample Playbooks + +To extract the data defined in your parser template, create a playbook that includes the Network Engine role and references the `content` and `file` (or `dir`) parameters of the `command_parser` module. + +Each example playbook below runs a show command, imports the Network Engine role, extracts data from the text output of the command by matching it against the rules defined +in your parser template, and stores the results in a variable. To view the content of that final variable, make sure `export: yes` is set in your parser template, and run your playbook in `verbose` mode: `ansible-playbook -vvv`. + +Make sure the `hosts` definition in the playbook matches a host group in your inventory - in these examples, the playbook expects a group called `ios`. + +The first example parses the output of the `show interfaces` command on IOS and creates facts from that output: + +```yaml + +--- + +# ~/my-playbooks/gather-interface-info.yml + +- hosts: ios + connection: network_cli + + tasks: + - name: Collect interface information from device + ios_command: + commands: + - show interfaces + register: ios_interface_output + + - name: import the network-engine role + import_role: + name: ansible-network.network-engine + + - name: Generate interface facts as JSON + command_parser: + file: "parser_templates/ios/show_interfaces.yaml" + content: "{{ ios_interface_output.stdout.0 }}" + +``` + +The second example parses the output of the `show version` command on IOS and creates facts from that output: + +```yaml + +--- + +# ~/my-playbooks/gather-version-info.yml + +- hosts: ios + connection: network_cli + + tasks: + - name: Collect version information from device + ios_command: + commands: + - show version + register: ios_version_output + + - name: import the network-engine role + import_role: + name: ansible-network.network-engine + + - name: Generate version facts as JSON + command_parser: + file: "parser_templates/ios/show_version.yaml" + content: "{{ ios_version_output.stdout.0 }}" + +``` diff --git a/docs/user_guide/main.md b/docs/user_guide/main.md new file mode 100644 index 0000000..76c30b0 --- /dev/null +++ b/docs/user_guide/main.md @@ -0,0 +1,37 @@ +Using the Network Engine Role +---------------------------------- + +The Network Engine role is supported as a dependency of other Roles. The Network Engine Role extracts data about your network devices as Ansible facts in a JSON data structure, ready to be added to your inventory host facts and/or consumed by Ansible tasks and templates. You define the data elements you want to extract from each network OS command in parser templates, using either YAML or Google TextFSM syntax. The matching rules may be different on each network platform, but by defining the same variable names for the output on all platforms, you can normalize similar data across platforms. That's how the Network Engine Role supports truly platform-agnostic network automation. + +The Network Engine role can also be used directly, though direct usage is not supported with your Red Hat subscription. + +The initial release of the Network Engine role includes two parser modules: +* [command_parser](https://github.com/ansible-network/network-engine/blob/devel/docs/user_guide/command_parser.md) accepts YAML input, uses an internally maintained, loosely defined parsing language based on Ansible playbook directives +* [textfsm_parser](https://github.com/ansible-network/network-engine/blob/devel/docs/user_guide/textfsm_parser.md) accepts Google TextFSM input, uses Google TextFSM parsing language + +Both modules iterate over the data definitions in your parser templates, parse command output from your network devices (structured ASCII text) to find matches, and then convert the matches into Ansible facts in a JSON data structure. + +To use the Network Engine Role: +---------------------------------------- +1. Install the role from Ansible Galaxy +`ansible-galaxy install ansible-network.network-engine` will copy the Network Engine role to `~/.ansible/roles/`. +1. Select the parser engine you prefer +For YAML formatting, use `command_parser`; for TextFSM formatting, use `textfsm_parser`. The parser docs include +examples of how to define your data and create your tasks. +1. Define the data you want to extract (or use a pre-existing parser template) +See the parser_template sections of the command_parser and textfsm_parser docs for examples. +1. Create a playbook to extract the data you've defined +See the Playbook sections of the command_parser and textfsm_parser docs for examples. +1. Run the playbook with `ansible-playbook -i /path/to/your/inventory -u my_user -k /path/to/your/playbook` +1. Consume the JSON-formatted Ansible facts about your device(s) in inventory, templates, and tasks. + +Additional Resources +------------------------------------- +[README](https://galaxy.ansible.com/ansible-network/network-engine/#readme) +[command_parser tests](https://github.com/ansible-network/network-engine/tree/devel/tests/command_parser) +[textfsm_parser tests](https://github.com/ansible-network/network-engine/tree/devel/tests/textfsm_parser) +[Full changelog diff](https://github.com/ansible-network/network-engine/blob/devel/CHANGELOG.rst) + +Contributing and Reporting Feedback +------------------------------------- +[Review issues](https://github.com/ansible-network/network-engine/issues) diff --git a/docs/user_guide/textfsm_parser.md b/docs/user_guide/textfsm_parser.md new file mode 100644 index 0000000..8d0fce2 --- /dev/null +++ b/docs/user_guide/textfsm_parser.md @@ -0,0 +1,82 @@ +# textfsm_parser + +The [textfsm_parser](https://github.com/ansible-network/network-engine/blob/devel/library/textfsm_parser.py) +module is based on [Google TextFSM](https://github.com/google/textfsm/wiki/TextFSM) definitions. +This module iterates over matching rules defined in TextFSM format, extracts data from structured ASCII text based on those rules, +and returns Ansible facts in a JSON data structure that can be added to inventory host facts and/or consumed by Ansible tasks and templates. + +The `textfsm_parser` module requires two inputs: +- the output of commands run on the network device, passed to the `content` parameter +- the parser template that defines the rules for parsing the output, passed to either the `file` or the `src` parameter + +## content + +The `content` parameter for `textfsm_parser` must point to the ASCII text output of commands run on network devices. The text output can be in a variable or in a file. + +## file + +The `file` parameter for `textfsm_parser` must point to a parser template that contains a TextFSM rule for each data field you want to extract from your network devices. + +Parser templates for the `textfsm_parser` module in the Network Engine role use TextFSM notation. + +### name + +The `name` parameter for `textfsm_parser` names the variable in which Ansible will store the JSON data structure. If name is not set, the JSON facts from parsing will not be displayed/exported. + +### src + +The `src` parameter for `textfsm_parser` loads your parser template from an external source, usually a URL. + +## Sample Parser Templates + +Here is a sample TextFSM parser template: + +`parser_templates/ios/show_interfaces` +``` + +Value Required name (\S+) +Value type ([\w ]+) +Value description (.*) +Value mtu (\d+) + +Start + ^${name} is up + ^\s+Hardware is ${type} -> Continue + ^\s+Description: ${description} + ^\s+MTU ${mtu} bytes, -> Record + +``` + +## Sample Playbooks + +To extract the data defined in your parser template, create a playbook that includes the Network Engine role and references the `content` and `file` parameters of the `command_parser` module. + +The example playbook below runs a show command, imports the Network Engine role, extracts data from the text output of the command by matching it against the rules defined +in your parser template, and stores the results in a variable. To view the content of that final variable, add it to the `name` parameter as shown in the example and run the playbook in `verbose` mode: `ansible-playbook -v`. + +Make sure the `hosts` definition in the playbook matches a host group in your inventory - in these examples, the playbook expects a group called `ios`. + +The example below parses the output of the `show interfaces` command on IOS and creates facts from that output: + +```yaml + +--- + +# ~/my-playbooks/textfsm-gather-interface-info.yml + +- hosts: ios + connection: network_cli + + tasks: + - name: Collect interface information from device + ios_command: + commands: "show interfaces" + register: ios_interface_output + + - name: Generate interface facts as JSON + textfsm_parser: + file: "parser_templates/ios/show_interfaces" + content: "{{ ios_interface_output.stdout.0 }}" + name: interface_facts + +```