I'm trying to copy 20TB of HDFS files from one cluster to another, using this single distcp command:
hadoop distcp
-Dmapreduce.map.memory.mb=16834
-Dmapreduce.reduce.memory.mb=32768
-Dyarn.app.mapreduce.am.resource.mb=32768
-direct -update -delete -m 100
webhdfs://source hdfs://target throws error
But it throws the IOException below. What does this mean?
java.io.IOException: DistCp failure: Job job_1711122443047_0032 has failed: Application application_1711122443047_0032 failed2 times due to ApplicationMaster for attempt appattempt_1711122443047_0032_000002 timed out. Failing the application.