-
Notifications
You must be signed in to change notification settings - Fork 73
Description
Hello I could help with this error does not take me the data of the mapreduce and I mark error some idea of which may be the fault
This is the code I'm using
Sys.setenv(HADOOP_HOME="/usr/local/hadoop")
Sys.setenv(HADOOP_CMD="/usr/local/hadoop/bin/hadoop")
Sys.setenv(JAVA_HOME="/usr/lib/jvm/java-7-openjdk-i386")
Sys.setenv(HADOOP_STREAMING="/usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.6.0.jar")
library('rmr2')
data=to.dfs(1:10)
res = mapreduce(input = data, map = function(k, v) cbind(v, 2*v))
from.dfs(res)
This is what appears to me on the console
Sys.setenv(HADOOP_CMD="/usr/local/hadoop/bin/hadoop")
Sys.setenv(JAVA_HOME="/usr/lib/jvm/java-7-openjdk-i386")
Sys.setenv(HADOOP_STREAMING="/usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.6.0.jar")
library('rmr2')
data=to.dfs(1:10)
OpenJDK Server VM warning: You have loaded library /usr/local/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c ', or link it with '-z noexecstack'.
17/04/10 18:12:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/04/10 18:12:46 INFO compress.CodecPool: Got brand-new compressor [.deflate]
res = mapreduce(input = data, map = function(k, v) cbind(v, 2*v))
OpenJDK Server VM warning: You have loaded library /usr/local/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c ', or link it with '-z noexecstack'.
17/04/10 18:12:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/04/10 18:12:50 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
packageJobJar: [/tmp/hadoop-unjar5133688999707817678/] [] /tmp/streamjob6112579330814301418.jar tmpDir=null
17/04/10 18:12:51 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.0.24:8050
17/04/10 18:12:52 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.0.24:8050
17/04/10 18:12:53 INFO mapred.FileInputFormat: Total input paths to process : 1
17/04/10 18:12:53 INFO mapreduce.JobSubmitter: number of splits:2
17/04/10 18:12:54 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1491851865431_0014
17/04/10 18:12:54 INFO impl.YarnClientImpl: Submitted application application_1491851865431_0014
17/04/10 18:12:54 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1491851865431_0014/
17/04/10 18:12:54 INFO mapreduce.Job: Running job: job_1491851865431_0014
17/04/10 18:13:02 INFO mapreduce.Job: Job job_1491851865431_0014 running in uber mode : false
17/04/10 18:13:02 INFO mapreduce.Job: map 0% reduce 0%
17/04/10 18:13:10 INFO mapreduce.Job: map 50% reduce 0%
17/04/10 18:13:11 INFO mapreduce.Job: map 100% reduce 0%
17/04/10 18:13:12 INFO mapreduce.Job: Job job_1491851865431_0014 completed successfully
17/04/10 18:13:13 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=220440
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=973
HDFS: Number of bytes written=244
HDFS: Number of read operations=14
HDFS: Number of large read operations=0
HDFS: Number of write operations=4
Job Counters
Launched map tasks=2
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=13405
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=13405
Total vcore-seconds taken by all map tasks=13405
Total megabyte-seconds taken by all map tasks=13726720
Map-Reduce Framework
Map input records=3
Map output records=0
Input split bytes=180
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=124
CPU time spent (ms)=2450
Physical memory (bytes) snapshot=296972288
Virtual memory (bytes) snapshot=1424506880
Total committed heap usage (bytes)=217579520
File Input Format Counters
Bytes Read=793
File Output Format Counters
Bytes Written=244
17/04/10 18:13:13 INFO streaming.StreamJob: Output directory: /tmp/file649a194b8a0e
from.dfs(res)
OpenJDK Server VM warning: You have loaded library /usr/local/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c ', or link it with '-z noexecstack'.
17/04/10 18:13:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/04/10 18:13:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Error in scan(file = file, what = what, sep = sep, quote = quote, dec = dec, :
Line 1 does not have 8 elements