[Doc][Task] Could not add step to AWS EMR

[xudalei1977]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

[INFO] 2025-01-15 09:17:39.974 +0000 - emr task params:{"localParams":[],"resourceList":[],"programType":"ADD_JOB_FLOW_STEPS","stepsDefineJson":"{\n "JobFlowId": "j-2CY7LGFW4NG9H",\n "Steps": [\n {\n "Name": "calculate_pi",\n "ActionOnFailure": "CONTINUE",\n "HadoopJarStep": {\n "Jar": "command-runner.jar",\n "Args": [\n "/usr/lib/spark/bin/run-example",\n "SparkPi",\n "15"\n ]\n }\n }\n ]\n}\n"}

[INFO] 2025-01-15 09:17:39.977 +0000 - Success initialized task plugin instance success

[INFO] 2025-01-15 09:17:39.977 +0000 - Success set taskVarPool: null

[ERROR] 2025-01-15 09:17:43.931 +0000 - emr task submit failed with error

com.amazonaws.services.elasticmapreduce.model.AmazonElasticMapReduceException: The security token included in the request is invalid. (Service: AmazonElasticMapReduce; Status Code: 400; Error Code: UnrecognizedClientException; Request ID: ed7bb6e1-4be4-4b3f-8778-4b78dbdf5277; Proxy: null)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1862)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1415)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1384)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1154)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:811)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)

at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)

at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)

at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)

at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.doInvoke(AmazonElasticMapReduceClient.java:3669)

at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.invoke(AmazonElasticMapReduceClient.java:3636)

at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.invoke(AmazonElasticMapReduceClient.java:3625)

at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.executeAddJobFlowSteps(AmazonElasticMapReduceClient.java:491)

at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.addJobFlowSteps(AmazonElasticMapReduceClient.java:460)

at org.apache.dolphinscheduler.plugin.task.emr.EmrAddStepsTask.submitApplication(EmrAddStepsTask.java:78)

at org.apache.dolphinscheduler.plugin.task.api.AbstractRemoteTask.handle(AbstractRemoteTask.java:62)

at org.apache.dolphinscheduler.server.worker.runner.DefaultWorkerDelayTaskExecuteRunnable.executeTask(DefaultWorkerDelayTaskExecuteRunnable.java:49)

at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecuteRunnable.run(WorkerTaskExecuteRunnable.java:174)

at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)

at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)

at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)

at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:834)

[INFO] 2025-01-15 09:17:43.931 +0000 - emr task finished with step status : null

[ERROR] 2025-01-15 09:17:43.931 +0000 - Task execute failed, due to meet an exception

org.apache.dolphinscheduler.plugin.task.api.TaskException: emr task submit fail

at org.apache.dolphinscheduler.plugin.task.emr.EmrAddStepsTask.submitApplication(EmrAddStepsTask.java:89)

at org.apache.dolphinscheduler.plugin.task.api.AbstractRemoteTask.handle(AbstractRemoteTask.java:62)

at org.apache.dolphinscheduler.server.worker.runner.DefaultWorkerDelayTaskExecuteRunnable.executeTask(DefaultWorkerDelayTaskExecuteRunnable.java:49)

at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecuteRunnable.run(WorkerTaskExecuteRunnable.java:174)

at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)

at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)

at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)

at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:834)

Caused by: com.amazonaws.services.elasticmapreduce.model.AmazonElasticMapReduceException: The security token included in the request is invalid. (Service: AmazonElasticMapReduce; Status Code: 400; Error Code: UnrecognizedClientException; Request ID: ed7bb6e1-4be4-4b3f-8778-4b78dbdf5277; Proxy: null)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1862)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1415)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1384)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1154)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:811)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)

at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)

at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)

at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)

at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.doInvoke(AmazonElasticMapReduceClient.java:3669)

at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.invoke(AmazonElasticMapReduceClient.java:3636)

at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.invoke(AmazonElasticMapReduceClient.java:3625)

at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.executeAddJobFlowSteps(AmazonElasticMapReduceClient.java:491)

at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.addJobFlowSteps(AmazonElasticMapReduceClient.java:460)

at org.apache.dolphinscheduler.plugin.task.emr.EmrAddStepsTask.submitApplication(EmrAddStepsTask.java:78)

... 10 common frames omitted

[INFO] 2025-01-15 09:17:43.931 +0000 - trying cancel emr step, taskId:7, clusterId:null, stepId:null

[ERROR] 2025-01-15 09:17:45.178 +0000 - Task execute failed and cancel the application failed, this will not affect the taskInstance status, but you need to check manual

com.amazonaws.services.elasticmapreduce.model.AmazonElasticMapReduceException: The security token included in the request is invalid. (Service: AmazonElasticMapReduce; Status Code: 400; Error Code: UnrecognizedClientException; Request ID: 821fd80e-5c60-4f77-ba63-3878c0037ce4; Proxy: null)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1862)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1415)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1384)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1154)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:811)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)

at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)

at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)

at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)

at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.doInvoke(AmazonElasticMapReduceClient.java:3669)

at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.invoke(AmazonElasticMapReduceClient.java:3636)

at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.invoke(AmazonElasticMapReduceClient.java:3625)

at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.executeCancelSteps(AmazonElasticMapReduceClient.java:619)

at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.cancelSteps(AmazonElasticMapReduceClient.java:588)

at org.apache.dolphinscheduler.plugin.task.emr.EmrAddStepsTask.cancelApplication(EmrAddStepsTask.java:185)

at org.apache.dolphinscheduler.plugin.task.api.AbstractRemoteTask.cancel(AbstractRemoteTask.java:39)

at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecuteRunnable.cancelTask(WorkerTaskExecuteRunnable.java:133)

at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecuteRunnable.afterThrowing(WorkerTaskExecuteRunnable.java:119)

at org.apache.dolphinscheduler.server.worker.runner.DefaultWorkerDelayTaskExecuteRunnable.afterThrowing(DefaultWorkerDelayTaskExecuteRunnable.java:59)

at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecuteRunnable.run(WorkerTaskExecuteRunnable.java:180)

at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)

at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)

at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)

at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:834)

[INFO] 2025-01-15 09:17:45.179 +0000 - Get a exception when execute the task, will send the task execute result to master, the current task execute result is TaskExecutionStatus{code=6, desc='failure'}

Documentation Links

https://dolphinscheduler.apache.org/en-us/docs/3.2.2/guide/task/emr

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct