Home > Storage > PowerScale (Isilon) > Industry Solutions and Verticals > Analytics > Multi-Cloud Data Services for Dell PowerScale in AWS: Amazon EMR for Data Analytics Solutions > HDFS, YARN, MapReduce, and Distcp
Script started on 2021-06-03 18:21:17+0000
EEEEEEEEEEEEEEEEEEEE MMMMMMMM MMMMMMMM RRRRRRRRRRRRRRR
E::::::::::::::::::E M:::::::M M:::::::M R::::::::::::::R
EE:::::EEEEEEEEE:::E M::::::::M M::::::::M R:::::RRRRRR:::::R
E::::E EEEEE M:::::::::M M:::::::::M RR::::R R::::R
E::::E M::::::M:::M M:::M::::::M R:::R R::::R
E:::::EEEEEEEEEE M:::::M M:::M M:::M M:::::M R:::RRRRRR:::::R
E::::::::::::::E M:::::M M:::M:::M M:::::M R:::::::::::RR
E:::::EEEEEEEEEE M:::::M M:::::M M:::::M R:::RRRRRR::::R
E::::E M:::::M M:::M M:::::M R:::R R::::R
E::::E EEEEE M:::::M MMM M:::::M R:::R R::::R
EE:::::EEEEEEEE::::E M:::::M M:::::M R:::R R::::R
E::::::::::::::::::E M:::::M M:::::M RR::::R R::::R
EEEEEEEEEEEEEEEEEEEE MMMMMMM MMMMMMM RRRRRRR RRRRRR
]0;hadoop@ip-10-15-1-140:~/scripts
[?1034h[hadoop@ip-10-15-1-140 scripts]$ ./hdfs_mr.sh
++ clear
[3J[H[2J++ POWERSCALE_HDFS=hdfs://10.1.1.15:8020
++ sudo -u hdfs hdfs dfs -rm -skipTrash -r hdfs://10.1.1.15:8020/user/hdfs/mr
Deleted hdfs://10.1.1.15:8020/user/hdfs/mr
++ sudo -u hdfs hdfs dfs -mkdir -p hdfs://10.1.1.15:8020/user/hdfs/mr
++ sudo -u hdfs hdfs dfs -put /etc/os-release hdfs://10.1.1.15:8020/user/hdfs/mr/
++ sudo -u hdfs hdfs dfs -ls hdfs://10.1.1.15:8020/user/hdfs/mr/
Found 1 items
-rw-r--r-- 3 root Administrators 212 2021-06-03 18:21 hdfs://10.1.1.15:8020/user/hdfs/mr/os-release
++ sudo -u hdfs yarn jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount /user/hdfs/mr/os-release hdfs://10.1.1.15:8020/user/hdfs/mr/output
21/06/03 18:21:31 INFO client.RMProxy: Connecting to ResourceManager at ip-10-15-1-140.ec2.internal/10.15.1.140:8032
21/06/03 18:21:31 INFO client.AHSProxy: Connecting to Application History server at ip-10-15-1-140.ec2.internal/10.15.1.140:10200
21/06/03 18:21:32 INFO input.FileInputFormat: Total input files to process : 1
21/06/03 18:21:32 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
21/06/03 18:21:32 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev e24bb348396ead67774fc8eb51646b5aaeae53a1]
21/06/03 18:21:32 INFO mapreduce.JobSubmitter: number of splits:1
21/06/03 18:21:32 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1622731208564_0010
21/06/03 18:21:32 INFO conf.Configuration: resource-types.xml not found
21/06/03 18:21:32 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
21/06/03 18:21:32 INFO resource.ResourceUtils: Adding resource type - name = memory-mb, units = Mi, type = COUNTABLE
21/06/03 18:21:32 INFO resource.ResourceUtils: Adding resource type - name = vcores, units = , type = COUNTABLE
21/06/03 18:21:32 INFO impl.YarnClientImpl: Submitted application application_1622731208564_0010
21/06/03 18:21:32 INFO mapreduce.Job: The url to track the job: http://ip-10-15-1-140.ec2.internal:20888/proxy/application_1622731208564_0010/
21/06/03 18:21:32 INFO mapreduce.Job: Running job: job_1622731208564_0010
21/06/03 18:21:38 INFO mapreduce.Job: Job job_1622731208564_0010 running in uber mode : false
21/06/03 18:21:38 INFO mapreduce.Job: map 0% reduce 0%
21/06/03 18:21:43 INFO mapreduce.Job: map 100% reduce 0%
21/06/03 18:21:49 INFO mapreduce.Job: map 100% reduce 67%
21/06/03 18:21:50 INFO mapreduce.Job: map 100% reduce 100%
21/06/03 18:21:50 INFO mapreduce.Job: Job job_1622731208564_0010 completed successfully
21/06/03 18:21:51 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=328
FILE: Number of bytes written=878643
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=322
HDFS: Number of bytes written=240
HDFS: Number of read operations=12
HDFS: Number of large read operations=0
HDFS: Number of write operations=6
Job Counters
Launched map tasks=1
Launched reduce tasks=3
Rack-local map tasks=1
Total time spent by all maps in occupied slots (ms)=123075
Total time spent by all reduces in occupied slots (ms)=1146150
Total time spent by all map tasks (ms)=2735
Total time spent by all reduce tasks (ms)=12735
Total vcore-milliseconds taken by all map tasks=2735
Total vcore-milliseconds taken by all reduce tasks=12735
Total megabyte-milliseconds taken by all map tasks=3938400
Total megabyte-milliseconds taken by all reduce tasks=36676800
Map-Reduce Framework
Map input records=9
Map output records=14
Map output bytes=268
Map output materialized bytes=316
Input split bytes=110
Combine input records=14
Combine output records=14
Reduce input groups=14
Reduce shuffle bytes=316
Reduce input records=14
Reduce output records=14
Spilled Records=28
Shuffled Maps =3
Failed Shuffles=0
Merged Map outputs=3
GC time elapsed (ms)=353
CPU time spent (ms)=3690
Physical memory (bytes) snapshot=1286746112
Virtual memory (bytes) snapshot=16800141312
Total committed heap usage (bytes)=1018691584
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=212
File Output Format Counters
Bytes Written=240
++ sudo -u hdfs hdfs dfs -ls -R hdfs://10.1.1.15:8020/user/hdfs/mr/
-rw-r--r-- 3 root Administrators 212 2021-06-03 18:21 hdfs://10.1.1.15:8020/user/hdfs/mr/os-release
drwxr-xr-x - root Administrators 0 2021-06-03 18:21 hdfs://10.1.1.15:8020/user/hdfs/mr/output
-rw-r--r-- 3 root Administrators 0 2021-06-03 18:21 hdfs://10.1.1.15:8020/user/hdfs/mr/output/_SUCCESS
-rw-r--r-- 3 root Administrators 100 2021-06-03 18:21 hdfs://10.1.1.15:8020/user/hdfs/mr/output/part-r-00000
-rw-r--r-- 3 root Administrators 85 2021-06-03 18:21 hdfs://10.1.1.15:8020/user/hdfs/mr/output/part-r-00001
-rw-r--r-- 3 root Administrators 55 2021-06-03 18:21 hdfs://10.1.1.15:8020/user/hdfs/mr/output/part-r-00002
++ sudo -u hdfs hdfs dfs -cat 'hdfs://10.1.1.15:8020/user/hdfs/mr/output/part*'
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2" 1
Linux" 1
NAME="Amazon 1
VERSION="2" 1
VERSION_ID="2" 1
2" 1
HOME_URL="https://amazonlinux.com/" 1
ID="amzn" 1
Linux 1
PRETTY_NAME="Amazon 1
ANSI_COLOR="0;33" 1
ID_LIKE="centos 1
fedora" 1
rhel 1
++ true
]0;hadoop@ip-10-15-1-140:~/scripts
[hadoop@ip-10-15-1-140 scripts]$ exit
exit
Script done on 2021-06-03 18:22:00+0000