Merge pull request #298 from Cumulocity-IoT/develop

Develop

.github/workflows/build-release.yml

@@ -23,10 +23,10 @@ jobs:
         id: buildx
         uses: docker/setup-buildx-action@v3
       - uses: actions/checkout@v4
-      - name: Set up JDK 11
+      - name: Set up JDK 21
         uses: actions/setup-java@v4
         with:
-          java-version: 11
+          java-version: 21
           distribution: zulu
       - name: Build MQTT Service Backend
         run: mvn -B package --file pom.xml

.gitignore

@@ -52,3 +52,4 @@ JSONata4Java/dependency-reduced-pom.xml
 dynamic-mapping-ui/node_modules_tmp/
 .vscode/launch.json
 .env
+setEnv.sh

ARCHITECTURE.md

@@ -37,3 +37,10 @@ The mapper processes messages in both directions:
 The Dynamic Data Mapper can be deployed as a **multi tenant microservice** which means you can deploy it once in your enterprise tenant and subscribe additional tenants using the same hardware resources.
 It is also implemented to support **multiple broker connections** at the same time. So you can combine multiple message brokers sharing the same mappings.
 This implies of course that all of them use the same topic structure and payload otherwise the mappings will fail.
+
+Incoming messages are processed by the steps in the following diagram. It shows the involved classes:
+
+<p align="center">
+<img src="resources/image/Dynamic_Mapper_Diagram_Dispatcher.png"  style="width: 100%;" />
+</p>
+<br/>

EXTENSIONS.md

@@ -9,27 +9,34 @@ In addition a Callback must be implemented handling the message broker typical m
 Check out the [MQTTCallback](./dynamic-mapping-service/src/main/java/dynamic/mapping/connector/mqtt/MQTTCallback.java) as an example implementation.
 
 ## Mapper Extensions
-In the folder [dynamic.mapping.processor.extension](./dynamic-mapping-service/src/main/java/dynamic/mapping/processor/extension) you can implement  the Interface `ProcessorExtensionInbound<O>` to implement the processing of your own messages. Together with the Java representation of your message you can build your own processor extension.
+In the folder [dynamic.mapping.processor.extension](./dynamic-mapping-service/src/main/java/dynamic/mapping/processor/extension) you can implement  the Interface `ProcessorExtension<O>` to implement the processing of your own messages. Together with the Java representation of your message you can build your own processor extension.
 This needs to be packages in a ```jar``` file. The extension packaged as a ```jar``` you can upload this extension using the tab ```Processor Extension```, see [Processing Extensions (Protobuf, ...)](#processing-extensions-protobuf) for details.
 In order for the mapper backend (```dynamic-mapping-service```) to find your extension you need to add the properties file ```extension-external.properties```. The content could be as follows:
 ```
-CustomEvent=external.extension.processor.dynamic.mapping.ProcessorExtensionInboundCustomEvent
-CustomMeasurement=external.extension.processor.dynamic.mapping.ProcessorExtensionInboundCustomMeasurement
+CustomEvent=external.extension.processor.dynamic.mapping.ProcessorExtensionCustomEvent
+CustomMeasurement=external.extension.processor.dynamic.mapping.ProcessorExtensionCustomMeasurement
 ```
 
 The steps required for an external extension are as follows. The extension:
-1. has to implement the interface <code>ProcessorExtensionInbound<O></code>
+1. has to implement the interface <code>ProcessorExtension<O></code>
 2. be registered in the properties file <code>dynamic-mapping-extension/src/main/resources/extension-external.properties</code>
 3. be developed/packed in the maven module <code>dynamic-mapping-extension</code>. **Not** in the maven module <code>dynamic-mapping-service</code>. This is reserved for internal extensions.
 4. be uploaded through the Web UI.
 
-> **_NOTE:_** When you implement <code>ProcessorExtensionInbound<O></code> an additional <code>RepairStrategy.CREATE_IF_MISSING</code> can be used. This helps to address mapping cases, where you want to create a mapping that adapts to different structures of source payloads. It is used to create a node in the target if it doesn't exist and allows for using mapping with dynamic content. See [sample 25](./resources/script/mapping/sampleMapping/SampleMappings_06.pdf).
+> **_NOTE:_** When you implement <code>ProcessorExtension<O></code> an additional <code>RepairStrategy.CREATE_IF_MISSING</code> can be used. This helps to address mapping cases, where you want to create a mapping that adapts to different structures of source payloads. It is used to create a node in the target if it doesn't exist and allows for using mapping with dynamic content. See [sample 25](./resources/script/mapping/sampleMapping/SampleMappings_06.pdf).
 
 A sample how to build an extension is contained in the maven module [dynamic-mapping-extension](./dynamic-mapping-extension).
 The following diagram shows how the dispatcher handles messages with different format:
 
-TODO Replace picture
+
 <p align="center">
 <img src="resources/image/Dynamic_Mapper_Diagram_Dispatcher.png"  style="width: 70%;" />
 </p>
+<br/>
+
+The following diagram gives an overview on the step to build and use your own extension:
+
+<p align="center">
+<img src="resources/image/Dynamic_Mapper_Diagram_ProcessorExtensionSource_Guide.png"  style="width: 70%;" />
+</p>
 <br/>

INSTALLATION.md

@@ -17,6 +17,9 @@ You need to install two components to your Cumulocity IoT Tenant:
 Both are provided as binaries in [releases](https://github.com/SoftwareAG/cumulocity-dynamic-mapper/releases).
 Download the binaries from the latest release and upload them to your Cumulocity IoT Tenant.
 
+> **_NOTE:_**
+> The screenshots show the dynamic mapper installed as a plugin in the Cumulocity Administration app. The dynamic mapper can also be installed in other Cumulocity apps, including Cockpit or Device Management, or as a standalone application.
+
 ## Permissions
 
 The solution defines one role:`ROLE_MAPPING_ADMIN` that must be assigned to the user accessing the Dynamic Mapping app.
@@ -61,7 +64,7 @@ Now select the cloned Administration App and go to the "Plugin" Tab. Click on "I
 After successfully adding the plugin you need to refresh the Administration App by pressing F5 and you should see a new navigation entry "Dynamic Mapping"
 
 <p align="center">
- <img src="resources/image/Dynamic_Mapper_WebAppPlugin.png" style="width: 60%;" />
+ <img src="resources/image/Dynamic_Mapper_WebAppPlugin.png" style="width: 40%;" />
 </p>
 
 ##### Blueprint
@@ -77,7 +80,7 @@ Enter an application key and path and click on "Deploy".
 After successful deployment you can find the Dynamic-mapping App in the Application Switcher
 
 <p align="center">
-<img src="resources/image/Dynamic_Mapper_BlueprintApp.png" style="width: 60%;" />
+<img src="resources/image/Dynamic_Mapper_BlueprintApp.png" style="width: 40%;" />
 </p>
 #### Manual
 

README.md

@@ -26,7 +26,7 @@ Here are the **core features** summarized:
 
 <br/>
 <p align="center">
-<img src="resources/image/Dynamic_Mapper_Mapping_Stepper_Substitution.png"  style="width: 80%;" />
+<img src="resources/image/Dynamic_Mapper_Mapping_Stepper_Substitution_Basic.png"  style="width: 80%;" />
 </p>
 <br/>
 

USERGUIDE.md

@@ -119,8 +119,8 @@ Creation of the new mapping starts by pressing `Add Mapping`. On the next modal
 
 1. `JSON`: if your payload is in JSON format
 1. `FLAT_FILE`: if your payload is in a CSV format
-1. `GENERIC_BINARY`: if your payload is in HEX format
-1. `PROTOBUF_STATIC`: if your payload is a serialized protobuf message
+1. `BINARY`: if your payload is in HEX format
+1. `PROTOBUF_INTERNAL`: if your payload is a serialized protobuf message
 1. `PROCESSOR_EXTENSION`: if you want to process the message yourself, by registering a processor extension
 
 <p align="center">
@@ -134,14 +134,14 @@ The wizard to define a mapping consists of the steps:
 
 - `JSON`
 - `FLAT_FILE`
-- `GENERIC_BINARY`
-- `PROTOBUF_STATIC`
+- `BINARY`
+- `PROTOBUF_INTERNAL`
 - `PROCESSOR_EXTENSION`
 
 ---
 
 **NOTE:**
-Payload for `FLAT_FILE` and `GENERIC_BINARY` are wrapped.
+Payload for `FLAT_FILE` and `BINARY` are wrapped.
 For example for a flat file messages:
 
 ```
@@ -204,7 +204,7 @@ For the mappings we differentiate between a **subscription topic** and a **templ
 
 For outbound mappings the properties are slightly different. Most important are the properties:
 
-1. `filterOutbound`: The Filter Outbound can contain one fragment name to associate a
+1. `filterMapping`: The Filter Outbound can contain one fragment name to associate a
    mapping to a Cumulocity MEAO. If the Cumulocity MEAO contains this fragment, the mapping is
    applied.
 2. `publishTopic`: MQTT topic to publish outbound messages to.
@@ -216,7 +216,7 @@ For outbound mappings the properties are slightly different. Most important are
 
 For an outbound mapping to be applied two conditions have to be fulfilled:
 
-1. the Cumulocity MEAO message has to have a fragment that is defined in property `filterOutbound`
+1. the Cumulocity MEAO message has to have a fragment that is defined in property `filterMapping`
 2. for the device a Notification 2.0 subscription has to be created. This is done using the following dialog:
 <p align="center">
 <img src="resources/image/Dynamic_Mapper_Mapping_Stepper_Outbound_subscription.png"  style="width: 70%;" />
@@ -298,7 +298,7 @@ In order to define a substitution (a substitution substitutes values in the targ
 3. Delete an existing substitution, by pressing the button with the red minus
 
 <p align="center">
-<img src="resources/image/Dynamic_Mapper_Mapping_Stepper_Substitution_Detail_Annnotated.png"  style="width: 70%;" />
+<img src="resources/image/Dynamic_Mapper_Mapping_Stepper_Substitution_ExpertMode_Annotated.png"  style="width: 70%;" />
 </p>
 <br/>
 
@@ -313,15 +313,18 @@ To define a new substitution the following steps have to be performed:
    > required since this can overwrite mandatory Cumulocity attributes, e.g. <code>source.id</code>. This can result in API calls that are rejected by the Cumulocity backend!
 
 1. Press the button "Add substitution". In the next modal dialog the following details can be specified: 1. Select option `Expand Array` if the result of the source expression is an array and you want to generate any of the following substitutions:
-_ `multi-device-single-value`
-_ `multi-device-multi-value`
-_ `single-device-multi-value`\
- Otherwise an extracted array is treated as a single value, see [Different type of substitutions](#different-type-of-substitutions). 1. Select option `Resolve to externalId` if you want to resolve system Cumulocity Id to externalId using externalIdType. This can only be used for OUTBOUND mappings. 1. Select a `Reapir Strategy` that determines how the mapping is applied:
-_ `DEFAULT`: Map the extracted values to the attribute addressed on right side
-_ `USE_FIRST_VALUE_OF_ARRAY`: When the left side of the mapping returns an array, only use the 1. item in the array and map this to the right side
-_ `USE_LAST_VALUE_OF_ARRAY`: When the left side of the mapping returns an array, only use the last item in the array and map this to the right side
-_ `REMOVE_IF_MISSING`: When the left side of the mapping returns no result (not NULL), then delete the attribute (that is addressed in mapping) in the target on the right side. This avoids empty attribute, e.d. `airsensor: undefined`
-_ `REMOVE_IF_NULL`: When the left side of the mapping returns `null`, then delete the attribute (that is addressed in mapping) in the target on the right side. This avoids empty attribute, e.d. `airsensor: undefined`
+    * `multi-device-single-value`
+    * `multi-device-multi-value`
+    * `single-device-multi-value`\
+\
+Otherwise an extracted array is treated as a single value, see [Different type of substitutions](#different-type-of-substitutions). 
+4. Select option `Resolve to externalId` if you want to resolve system Cumulocity Id to externalId using externalIdType. This can only be used for OUTBOUND mappings. 
+5. Select a `Repair Strategy` that determines how the mapping is applied:
+    * `DEFAULT`: Map the extracted values to the attribute addressed on right side
+    * `USE_FIRST_VALUE_OF_ARRAY`: When the left side of the mapping returns an array, only use the 1. item in the array and map this to the right side
+    * `USE_LAST_VALUE_OF_ARRAY`: When the left side of the mapping returns an array, only use the last item in the array and map this to the right side
+    * `REMOVE_IF_MISSING_OR_NULL`: When the left side of the mapping returns no result (not NULL), then delete the attribute (that is addressed in mapping) in the target on the right side. This avoids empty attribute, e.g. `airsensor: undefined`
+
 <p align="center">
 <img src="resources/image/Dynamic_Mapper_Mapping_Stepper_Edit_Modal.png"  style="width: 70%;" />
 </p>
@@ -446,7 +449,7 @@ In order to use a previously snooped payload click the button
 `Snooped templates`. Multiples activation of this button iterates over all the recorded templates.
 
 <p align="center">
-<img src="resources/image/Dynamic_Mapper_Mapping_Table_Add_Modal_Snooping_Annnotated.png"  style="width: 70%;" />
+<img src="resources/image/Dynamic_Mapper_Mapping_Table_Add_Modal_Snooping.png"  style="width: 70%;" />
 </p>
 <br/>
 
@@ -501,14 +504,14 @@ When you choose the mapping type `PROCESSOR_EXTENSION` the wizard for defining y
 Using the tab `Processor Extension` you can upload your own processor extension. After the upload the mircroservice load the extensions dynamically.
 
 <p align="center">
-<img src="resources/image/Dynamic_Mapper_Configuration_ProcessorExtensionInbound.png"  style="width: 70%;" />
+<img src="resources/image/Dynamic_Mapper_Configuration_ProcessorExtensionSource.png"  style="width: 70%;" />
 </p>
 <br/>
 
 The following guide lays out the steps to create and use a processor extension:
 
 <p align="center">
-<img src="resources/image/Dynamic_Mapper_Diagram_ProcessorExtensionInbound_Guide.png"  style="width: 70%;" />
+<img src="resources/image/Dynamic_Mapper_Diagram_ProcessorExtensionSource_Guide.png"  style="width: 70%;" />
 </p>
 <br/>
 

dynamic-mapping-extension/pom.xml

@@ -22,28 +22,27 @@
   -->
 
 <project xmlns="http://maven.apache.org/POM/4.0.0"
-    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
-    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
     <modelVersion>4.0.0</modelVersion>
     <packaging>jar</packaging>
-    <groupId>com.softwareag.c8y.mapping</groupId>
+    <groupId>com.cumulocity.mapping</groupId>
     <artifactId>dynamic-mapping-extension</artifactId>
 
     <parent>
-        <groupId>com.softwareag.c8y.mapping</groupId>
+        <groupId>com.cumulocity.mapping</groupId>
         <artifactId>dynamic-mapping-solution</artifactId>
         <version>${revision}</version>
     </parent>
 
     <dependencyManagement>
         <dependencies>
             <dependency>
-                <groupId>com.softwareag.c8y.mapping</groupId>
+                <groupId>com.cumulocity.mapping</groupId>
                 <artifactId>dynamic-mapping-interface</artifactId>
                 <version>${revision}</version>
             </dependency>
             <dependency>
-                <groupId>com.softwareag.c8y.mapping</groupId>
+                <groupId>com.cumulocity.mapping</groupId>
                 <artifactId>dynamic-mapping-service</artifactId>
                 <version>${revision}</version>
             </dependency>
@@ -52,13 +51,13 @@
 
     <dependencies>
         <dependency>
-            <groupId>com.softwareag.c8y.mapping</groupId>
-            <artifactId>dynamic-mapping-service</artifactId>
+            <groupId>com.cumulocity.mapping</groupId>
+            <artifactId>dynamic-mapping-interface</artifactId>
             <scope>compile</scope>
         </dependency>
         <dependency>
-            <groupId>com.softwareag.c8y.mapping</groupId>
-            <artifactId>dynamic-mapping-interface</artifactId>
+            <groupId>com.cumulocity.mapping</groupId>
+            <artifactId>dynamic-mapping-service</artifactId>
             <scope>compile</scope>
         </dependency>
         <dependency>
@@ -71,11 +70,6 @@
             <artifactId>lombok</artifactId>
             <scope>compile</scope>
         </dependency>
-        <!-- <dependency>
-                <groupId>org.eclipse.paho</groupId>
-                <artifactId>org.eclipse.paho.client.mqttv3</artifactId>
-                <scope>test</scope>
-            </dependency> -->
         <dependency>
             <groupId>com.hivemq</groupId>
             <artifactId>hivemq-mqtt-client</artifactId>