Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: upgrade to TX EDC 0.7 #92

Merged
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
4739a4d
chore(deps): bump docker/login-action from 2 to 3
dependabot[bot] Oct 6, 2023
56d0661
feat: first try to upgrade to current versions of EDC and Jena.
drcgjung Oct 19, 2023
fb587a3
feat: Adapt location of shaded google collections. Adapt to changes i…
drcgjung Oct 20, 2023
97d4452
feat: upgrade okio to non-vulnerable version. remove jakarata validat…
drcgjung Oct 20, 2023
15a4912
Merge branch 'feature/KA-409-upgrade-edcasset-v3' into feature/upgrad…
drcgjung Oct 20, 2023
d355e6a
chore(deps): bump helm/chart-testing-action from 2.4.0 to 2.6.1
dependabot[bot] Nov 3, 2023
0204a1c
chore(deps): bump org.junit:junit-bom from 5.9.3 to 5.10.1
dependabot[bot] Nov 6, 2023
e8b99da
fix: adapt aws netty lib to match the EDC awssdk dependency.
drcgjung Nov 9, 2023
114684a
Merge remote-tracking branch 'eclipse/dependabot/maven/main/org.junit…
drcgjung Nov 9, 2023
5ddf5d9
Merge remote-tracking branch 'eclipse/dependabot/github_actions/main/…
drcgjung Nov 9, 2023
fd5d47e
Merge remote-tracking branch 'eclipse/dependabot/github_actions/main/…
drcgjung Nov 9, 2023
f76ab27
fix: upgrade docker image and opentelemetry versions. flexibilize rem…
drcgjung Nov 9, 2023
01903c1
Merge remote-tracking branch 'eclipse/main' into feature/upgrade-tx-e…
drcgjung Feb 13, 2024
7f541e6
feat: upgrade version and get tests to run
drcgjung Feb 13, 2024
1353c7e
Merge remote-tracking branch 'eclipse/main' into feature/upgrade-tx-e…
drcgjung Mar 4, 2024
a195e61
feat: refactor full connector chart into agent plane only chart
drcgjung Mar 4, 2024
d5e2cae
feat: upgradable and testable agent-plane-only charts.
drcgjung Mar 9, 2024
95adb67
chore: add chart test and upgradability check.
drcgjung Mar 9, 2024
841e41c
chore: upgrade version and add s3 runtime dependency.
drcgjung Mar 9, 2024
838c9d6
Merge remote-tracking branch 'eclipse/main' into feature/upgrade-tx-e…
drcgjung Apr 17, 2024
fcd00d1
chore: double step in helm-chart-lint workflow
drcgjung Apr 17, 2024
7f2eb48
chore: make sure the right java version is used for linting build
drcgjung Apr 17, 2024
459ee81
chore: wrong extra args
drcgjung Apr 17, 2024
5400599
chore: ct --helm-extra-args and --helm-extra-set-args are bevaving di…
drcgjung Apr 17, 2024
fca0d00
chore: upgrade version because of helm/chart changes.
drcgjung Apr 17, 2024
6987e8d
chore: choose the correct chart value for the image regisry.
drcgjung Apr 17, 2024
8425a14
feat: align with matchmaking implementation.
drcgjung Apr 23, 2024
955ded3
feat: upgrade to tx-edc 0.7.0 / edc 0.6.1
drcgjung Apr 23, 2024
d3b53ab
fix: trailing spaces in chart.
drcgjung Apr 23, 2024
9055b18
fix: http client configguration override adopted to EDC 0.6.1 upstream
drcgjung Apr 25, 2024
7120f2e
fix: jackson dependencies completed and homogenized.
drcgjung Apr 25, 2024
3290298
fix: mitigate some CVE's in dependencies.
drcgjung Apr 25, 2024
ee9ccc1
chore: refresh dependencies.
drcgjung Apr 25, 2024
ae73c1b
feat: upgrade to EDC 0.7.0 asset definitions and transfer callback in…
drcgjung Apr 29, 2024
20b9167
fix: cleanup code aligned with matchmaking agent.
drcgjung Apr 30, 2024
e6d9746
fix: debug the transfer process/callback mechanism.
drcgjung May 6, 2024
7e64c0a
fix: correct logging rendering of the callback endpoint.
drcgjung May 6, 2024
619c69d
fix: skill creation/update needs to escape regexes.
drcgjung May 6, 2024
419eaa5
fix: add default runMode=ALL for skill registration. remove redundant…
drcgjung May 7, 2024
1cc7858
feat: upgrade charts to EDC 0.7 and 2405 ssi/iatp functionality
drcgjung May 7, 2024
959fb32
fix: do not use temporary image names and tags in normal workflows.
drcgjung May 7, 2024
ae6b3da
fix: correct callback endpoint name.
drcgjung May 7, 2024
0aa1310
fix: helm lint errors.
drcgjung May 7, 2024
93f43b9
fix: get the charts to run sideways with the tx-connector chart in um…
drcgjung May 10, 2024
dcee419
fix: helm linting
drcgjung May 10, 2024
ecbb2b9
fix: helm linting
drcgjung May 10, 2024
83c564c
fix: chart testing values were inconsistent
drcgjung May 10, 2024
ff06a36
fix: address umbrella deployments with connector-side-by-side.
drcgjung May 10, 2024
949beb3
fix: debug control plane references.
drcgjung May 10, 2024
29d51ae
fix: control and management templates.
drcgjung May 10, 2024
a23d00c
fix: lint errors.
drcgjung May 10, 2024
6dd0bf9
fix: chart integration test values.
drcgjung May 10, 2024
3d2ad72
fix: linting issues.
drcgjung May 10, 2024
cc1ce3b
chore: update DEPENDENCIES.
drcgjung May 10, 2024
6551459
fix: trailing quotes.
drcgjung May 13, 2024
03ffb12
chore: make sure the dependent repositories are available when chart …
drcgjung May 13, 2024
baa7de1
chore: update DEPENDENCIES after successful IP checks.
drcgjung May 15, 2024
642ad6e
fix: add resources to agent plane as borrowed from the tractusx charts
drcgjung May 15, 2024
94abcf5
docs: broken link because of style change.
drcgjung May 15, 2024
d157a90
fix: agent plane name was fixed in chart helpers.
drcgjung May 15, 2024
d49adbd
style: copyright license headers should use fixed period formatting.
drcgjung May 16, 2024
9c0c4eb
fix: important code smells/forgery threats as of codeql results 2024-…
drcgjung May 16, 2024
d417eb4
fix: important code smells/forgery threats as of codeql results 2024-…
drcgjung May 16, 2024
5dd2869
fix: incorrect signaling target port in agent service.
drcgjung May 17, 2024
f682324
fix: wrong azure environment vars.
drcgjung May 17, 2024
b9fb296
docs: changelog for the upcoming release.
drcgjung May 17, 2024
b89f17b
docs|fix: chart test for azure-vault plane fixed. Document the charts.
drcgjung May 18, 2024
2a3b7ac
fix: make ct install use the default values
drcgjung May 18, 2024
b94e1d5
fix: yaml lint
drcgjung May 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -90,12 +90,12 @@
submodules: recursive

# Setup build environment
- uses: ./.github/actions/setup-java

Check warning on line 93 in .github/workflows/build.yml

View workflow job for this annotation

GitHub Actions / Analyze

[MEDIUM] Unpinned Actions Full Length Commit SHA

Pinning an action to a full length commit SHA is currently the only way to use an action as an immutable release. Pinning to a particular SHA helps mitigate the risk of a bad actor adding a backdoor to the action's repository, as they would need to generate a SHA-1 collision for a valid Git object payload. When selecting a SHA, you should verify it is from the action's repository and not a repository fork.

# Enable deployment access (on main branch and version tags only)
- name: Login to GitHub Container Registry
if: ${{ ( github.event.inputs.deploy_docker == 'true' || github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/v')) }}
uses: docker/login-action@v2
uses: docker/login-action@v3
Fixed Show fixed Hide fixed
with:
registry: ${{ steps.set-docker-repo.outputs.REGISTRY }}
# Use existing DockerHub credentials present as secrets
Expand Down Expand Up @@ -139,7 +139,7 @@
type=raw,value=latest,enable=${{ github.ref == format('refs/heads/{0}', 'main') }}

- name: Agent Plane Hashicorp Container Build and push
uses: docker/build-push-action@v5

Check warning on line 142 in .github/workflows/build.yml

View workflow job for this annotation

GitHub Actions / Analyze

[MEDIUM] Unpinned Actions Full Length Commit SHA

Pinning an action to a full length commit SHA is currently the only way to use an action as an immutable release. Pinning to a particular SHA helps mitigate the risk of a bad actor adding a backdoor to the action's repository, as they would need to generate a SHA-1 collision for a valid Git object payload. When selecting a SHA, you should verify it is from the action's repository and not a repository fork.
with:
context: agent-plane/agentplane-hashicorp
file: agent-plane/agentplane-hashicorp/src/main/docker/Dockerfile
Expand All @@ -151,7 +151,7 @@
# Important step to push image description to DockerHub - since this is version independent, we always take it from main
- name: Update Docker Hub description for Agent Plane Hashicorp
if: ${{ steps.set-docker-repo.outputs.REPO == 'docker.io' && github.ref == 'refs/heads/main' }}
uses: peter-evans/dockerhub-description@v3

Check warning on line 154 in .github/workflows/build.yml

View workflow job for this annotation

GitHub Actions / Analyze

[MEDIUM] Unpinned Actions Full Length Commit SHA

Pinning an action to a full length commit SHA is currently the only way to use an action as an immutable release. Pinning to a particular SHA helps mitigate the risk of a bad actor adding a backdoor to the action's repository, as they would need to generate a SHA-1 collision for a valid Git object payload. When selecting a SHA, you should verify it is from the action's repository and not a repository fork.
with:
readme-filepath: agent-plane/agentplane-hashicorp/README.md
username: ${{ secrets.DOCKER_HUB_USER }}
Expand All @@ -161,7 +161,7 @@
# Create SemVer or ref tags dependent of trigger event
- name: Docker Meta Agent Plane Azure Vault
id: meta-azr
uses: docker/metadata-action@v5

Check warning on line 164 in .github/workflows/build.yml

View workflow job for this annotation

GitHub Actions / Analyze

[MEDIUM] Unpinned Actions Full Length Commit SHA

Pinning an action to a full length commit SHA is currently the only way to use an action as an immutable release. Pinning to a particular SHA helps mitigate the risk of a bad actor adding a backdoor to the action's repository, as they would need to generate a SHA-1 collision for a valid Git object payload. When selecting a SHA, you should verify it is from the action's repository and not a repository fork.
with:
images: |
${{ steps.set-docker-repo.outputs.REPO }}/agentplane-azure-vault
Expand All @@ -177,7 +177,7 @@
type=raw,value=latest,enable=${{ github.ref == format('refs/heads/{0}', 'main') }}

- name: Agent Plane Azure Vault Container Build and push
uses: docker/build-push-action@v5

Check warning on line 180 in .github/workflows/build.yml

View workflow job for this annotation

GitHub Actions / Analyze

[MEDIUM] Unpinned Actions Full Length Commit SHA

Pinning an action to a full length commit SHA is currently the only way to use an action as an immutable release. Pinning to a particular SHA helps mitigate the risk of a bad actor adding a backdoor to the action's repository, as they would need to generate a SHA-1 collision for a valid Git object payload. When selecting a SHA, you should verify it is from the action's repository and not a repository fork.
with:
context: agent-plane/agentplane-azure-vault/.
file: agent-plane/agentplane-azure-vault/src/main/docker/Dockerfile
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/helm-chart-lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@
fetch-depth: 0

- name: Set up Helm
uses: azure/setup-helm@v3

Check warning on line 65 in .github/workflows/helm-chart-lint.yml

View workflow job for this annotation

GitHub Actions / Analyze

[MEDIUM] Unpinned Actions Full Length Commit SHA

Pinning an action to a full length commit SHA is currently the only way to use an action as an immutable release. Pinning to a particular SHA helps mitigate the risk of a bad actor adding a backdoor to the action's repository, as they would need to generate a SHA-1 collision for a valid Git object payload. When selecting a SHA, you should verify it is from the action's repository and not a repository fork.
with:
version: v3.10.3

Expand All @@ -71,7 +71,7 @@
python-version: 3.9

- name: Set up chart-testing
uses: helm/chart-testing-action@v2.4.0
uses: helm/chart-testing-action@v2.6.1
Fixed Show fixed Hide fixed

- name: Run chart-testing (list-changed)
id: list-changed
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/trivy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@
# Enable repository access (on main branch and version tags only)
- name: Login to GitHub Container Registry
if: ${{ ( github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/v')) }}
uses: docker/login-action@v2
uses: docker/login-action@v3

Check warning on line 107 in .github/workflows/trivy.yml

View workflow job for this annotation

GitHub Actions / Analyze

[MEDIUM] Unpinned Actions Full Length Commit SHA

Pinning an action to a full length commit SHA is currently the only way to use an action as an immutable release. Pinning to a particular SHA helps mitigate the risk of a bad actor adding a backdoor to the action's repository, as they would need to generate a SHA-1 collision for a valid Git object payload. When selecting a SHA, you should verify it is from the action's repository and not a repository fork.
Fixed Show fixed Hide fixed
with:
registry: ${{ steps.set-docker-repo.outputs.REGISTRY }}
# Use existing DockerHub credentials present as secrets
Expand Down Expand Up @@ -132,6 +132,6 @@

- name: Upload Trivy scan results to GitHub Security tab
if: success() && steps.imageCheck.outcome != 'failure'
uses: github/codeql-action/upload-sarif@v2

Check warning on line 135 in .github/workflows/trivy.yml

View workflow job for this annotation

GitHub Actions / Analyze

[MEDIUM] Unpinned Actions Full Length Commit SHA

Pinning an action to a full length commit SHA is currently the only way to use an action as an immutable release. Pinning to a particular SHA helps mitigate the risk of a bad actor adding a backdoor to the action's repository, as they would need to generate a SHA-1 collision for a valid Git object payload. When selecting a SHA, you should verify it is from the action's repository and not a repository fork.
with:
sarif_file: "trivy-results-${{ matrix.image }}.sarif"
8 changes: 1 addition & 7 deletions agent-plane/agent-plane-protocol/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -304,7 +304,7 @@
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>netty-nio-client</artifactId>
<version>2.20.94</version>
<version>2.20.153</version>
<scope>runtime</scope>
</dependency>

Expand Down Expand Up @@ -343,12 +343,6 @@
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>jakarta.validation</groupId>
<artifactId>jakarta.validation-api</artifactId>
<version>${jakarta.validation.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.eclipse.edc</groupId>
<artifactId>jersey-core</artifactId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
import java.util.List;
import java.util.Set;

import org.apache.jena.ext.com.google.common.collect.ArrayListMultimap;
import com.github.jsonldjava.shaded.com.google.common.collect.ArrayListMultimap;

/**
* Implementation of a compact representation of
Expand All @@ -33,7 +33,7 @@
*/
public class TupleSet {

ArrayListMultimap<String,String> bindings=ArrayListMultimap.create();
ArrayListMultimap<String,String> bindings= ArrayListMultimap.create();
List<TupleSet> explodes=new ArrayList<>();

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ protected StreamResult<Stream<Part>> openMatchmaking() {
// Agent call, we translate from KA-MATCH to KA-TRANSFER
String skill=null;
String graph=null;
String asset= request.getSourceDataAddress().getProperties().get(AgentSourceHttpParamsDecorator.ASSET_PROP_ID);
String asset= String.valueOf(request.getSourceDataAddress().getProperties().get(AgentSourceHttpParamsDecorator.ASSET_PROP_ID));
if(asset!=null && asset.length() > 0) {
Matcher graphMatcher= AgentExtension.GRAPH_PATTERN.matcher(asset);
if(graphMatcher.matches()) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -141,13 +141,13 @@ public HttpRequestParams.Builder decorate(DataFlowRequest request, HttpDataAddre
Map<String,List<String>> queryParams=parseParams("?"+getRequestQueryParams(address,request));

if(isTransferRequest(request)) {
if(!address.getProperty(BASE_URL).endsWith(SLASH)) {
params.baseUrl(address.getProperty(BASE_URL)+SLASH);
if(!address.getStringProperty(BASE_URL).endsWith(SLASH)) {
params.baseUrl(address.getStringProperty(BASE_URL)+SLASH);
}
} else {
// we need to annotate the base url "pure" because we do not directly hit the endpoint
params.baseUrl("https://w3id.org/catenax");
params.header(DataspaceServiceExecutor.TARGET_URL_SYMBOL.getSymbol(), address.getProperty(BASE_URL));
params.header(DataspaceServiceExecutor.TARGET_URL_SYMBOL.getSymbol(), address.getStringProperty(BASE_URL));

// there is the case where a KA-BIND protocol call is
// one-to-one routed through the transfer plane ... in which case
Expand All @@ -165,7 +165,7 @@ public HttpRequestParams.Builder decorate(DataFlowRequest request, HttpDataAddre
queryParams.remove(QUERY_PARAM);
mergeParams(queryParams,bodyParams);
}
String accept=address.getProperty(ACCEPT_HEADER,null);
String accept=address.getStringProperty(ACCEPT_HEADER,null);
List<String> cxAccepts=queryParams.getOrDefault(CX_ACCEPT_PARAM,List.of());
queryParams.remove(CX_ACCEPT_PARAM);
if(accept==null) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -323,7 +323,7 @@ public QueryIterator createExecution(OpService opOriginal, String serviceURL, Se
}
}
// the asset type should be annotated in the rdf type property
assetType=endpoint.getProperties().getOrDefault("http://www.w3.org/1999/02/22-rdf-syntax-ns#type",assetType);
assetType=String.valueOf(endpoint.getProperties().getOrDefault("http://www.w3.org/1999/02/22-rdf-syntax-ns#type",assetType));

// put the endpoint information into a new service operator
// and cater for the EDC public api slash problem
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ protected void executeAction(AgentHttpAction action) {
* @param targetProperties a set of address properties of the asset to invoke
* @return simulated ok response
*/
public Response execute(Request request, String skill, String graph, Map<String,String> targetProperties) {
public Response execute(Request request, String skill, String graph, Map<String,Object> targetProperties) {

// wrap jakarta into java.servlet
HttpServletContextAdapter contextAdapter=new HttpServletContextAdapter(request);
Expand All @@ -208,12 +208,14 @@ public Response execute(Request request, String skill, String graph, Map<String,
action.getContext().set(DataspaceServiceExecutor.AUTH_CODE_SYMBOL,targetProperties.getOrDefault(DataspaceServiceExecutor.AUTH_CODE_SYMBOL.getSymbol(),null));
action.getContext().set(ARQConstants.sysOptimizerFactory,optimizerFactory);
if(targetProperties.containsKey(DataspaceServiceExecutor.ALLOW_SYMBOL.getSymbol())) {
action.getContext().set(DataspaceServiceExecutor.ALLOW_SYMBOL,Pattern.compile(targetProperties.get(DataspaceServiceExecutor.ALLOW_SYMBOL.getSymbol())));
action.getContext().set(DataspaceServiceExecutor.ALLOW_SYMBOL,
Pattern.compile(String.valueOf(targetProperties.get(DataspaceServiceExecutor.ALLOW_SYMBOL.getSymbol()))));
} else {
action.getContext().set(DataspaceServiceExecutor.ALLOW_SYMBOL,config.getServiceAssetAllowPattern());
}
if(targetProperties.containsKey(DataspaceServiceExecutor.DENY_SYMBOL.getSymbol())) {
action.getContext().set(DataspaceServiceExecutor.DENY_SYMBOL,Pattern.compile(targetProperties.get(DataspaceServiceExecutor.DENY_SYMBOL.getSymbol())));
action.getContext().set(DataspaceServiceExecutor.DENY_SYMBOL,
Pattern.compile(String.valueOf(targetProperties.get(DataspaceServiceExecutor.DENY_SYMBOL.getSymbol()))));
} else {
action.getContext().set(DataspaceServiceExecutor.DENY_SYMBOL,config.getServiceAssetDenyPattern());
}
Expand Down
23 changes: 11 additions & 12 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -37,24 +37,23 @@
<maven.compiler.target>11</maven.compiler.target>
<skipTests>false</skipTests>

<junit.version>5.9.3</junit.version>
<junit.version>5.10.1</junit.version>
<mockito.version>5.4.0</mockito.version>
<tx.edc.version>0.5.0</tx.edc.version>
<edc.version>0.1.3</edc.version>
<tx.edc.version>0.6.0-rc1</tx.edc.version>
<edc.version>0.3.1</edc.version>
<failsafe.version>3.3.2</failsafe.version>
<okhttp.version>4.11.0</okhttp.version>
<okio.version>3.4.0</okio.version>
<slf4j.version>2.0.7</slf4j.version>
<okio.version>3.6.0</okio.version>
<slf4j.version>2.0.9</slf4j.version>
<jakarta.ws.rs-api.version>3.1.0</jakarta.ws.rs-api.version>
<jakarta.validation.version>3.0.2</jakarta.validation.version>
<com.nimbusds.version>9.31</com.nimbusds.version>
<com.nimbusds.version>9.35</com.nimbusds.version>
<javax.servlet-api.version>4.0.1</javax.servlet-api.version>
<jetty-jakarta-servlet-api.version>5.0.2</jetty-jakarta-servlet-api.version>
<org.apache.jena.version>4.8.0</org.apache.jena.version>
<com.azure.azure-identity.version>1.9.1</com.azure.azure-identity.version>
<com.azure.azure-security-keyvault-secrets.version>4.6.2</com.azure.azure-security-keyvault-secrets.version>
<org.yaml.snakeyaml.version>2.0</org.yaml.snakeyaml.version>
<net.minidev.jsonsmart.version>2.4.11</net.minidev.jsonsmart.version>
<org.apache.jena.version>4.9.0</org.apache.jena.version>
<com.azure.azure-identity.version>1.10.1</com.azure.azure-identity.version>
<com.azure.azure-security-keyvault-secrets.version>4.6.5</com.azure.azure-security-keyvault-secrets.version>
<org.yaml.snakeyaml.version>2.2</org.yaml.snakeyaml.version>
<net.minidev.jsonsmart.version>2.5.0</net.minidev.jsonsmart.version>

<!-- Source characteristics -->
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
Expand Down
Loading