Skip to content

Problems indexing MOD09GQ.005 #32

@mrlathy

Description

@mrlathy

Trying to import MOD09GQ.005 file in Spatial Hadoop 2.3 (installation shipped with apache hadoop).
File was taken from here:
http://e4ftl01.cr.usgs.gov/MOLT/MOD09GQ.005/2016.02.17/MOD09GQ.A2016048.h35v09.005.2016050064358.hdf

File was copied to HDFS as 1.hdf. Index command:
bin/shadoop index 1.hdf 1.grid mbr:0,0,100,100 sindex:rtree shape:nasapoint
returns the following error:

 bin/shadoop index 1.hdf 1.grid mbr:0,0,100,100 sindex:rtree shape:nasapoint
16/03/18 19:13:09 INFO util.NativeCodeLoader: Loaded the native-hadoop library
16/03/18 19:13:09 WARN snappy.LoadSnappy: Snappy native library not loaded
16/03/18 19:13:09 INFO mapred.FileInputFormat: No block filter specified
16/03/18 19:13:09 INFO mapred.FileInputFormat: Total input paths to process : 1
16/03/18 19:13:09 INFO operations.Repartition: Reading a sample of 10%
16/03/18 19:13:09 INFO mapred.FileInputFormat: No block filter specified
16/03/18 19:13:09 INFO mapred.FileInputFormat: Total input paths to process : 1
16/03/18 19:13:10 INFO mapred.JobClient: Running job: job_local161844959_0001
16/03/18 19:13:10 INFO mapred.LocalJobRunner: Waiting for map tasks
16/03/18 19:13:10 INFO mapred.LocalJobRunner: Starting task: attempt_local161844959_0001_m_000000_0
16/03/18 19:13:10 INFO util.ProcessTree: setsid exited with exit code 0
16/03/18 19:13:10 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@5a4ca8a8
16/03/18 19:13:10 INFO mapred.MapTask: Processing split: file:/root/spatialhadoop-2.3-hadoop-1.2.1/1.hdf:0+71498628
16/03/18 19:13:10 INFO mapred.SpatialRecordReader: Open a SpatialRecordReader to file: file:/root/spatialhadoop-2.3-hadoop-1.2.1/1.hdf
16/03/18 19:13:10 INFO mapred.MapTask: numReduceTasks: 1
16/03/18 19:13:10 INFO mapred.MapTask: io.sort.mb = 100
16/03/18 19:13:10 INFO mapred.MapTask: data buffer = 79691776/99614720
16/03/18 19:13:10 INFO mapred.MapTask: record buffer = 262144/327680
16/03/18 19:13:10 INFO mapred.LocalJobRunner: Map task executor complete.
16/03/18 19:13:10 WARN mapred.LocalJobRunner: job_local161844959_0001
java.lang.Exception: java.lang.NumberFormatException: empty String
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:354)
Caused by: java.lang.NumberFormatException: empty String
        at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1011)
        at java.lang.Double.parseDouble(Double.java:540)
        at edu.umn.cs.spatialHadoop.io.TextSerializerHelper.consumeDouble(TextSerializerHelper.java:182)
        at edu.umn.cs.spatialHadoop.core.Point.fromText(Point.java:136)
        at edu.umn.cs.spatialHadoop.nasa.NASAPoint.fromText(NASAPoint.java:77)
        at edu.umn.cs.spatialHadoop.operations.Sampler$Map.map(Sampler.java:122)
        at edu.umn.cs.spatialHadoop.operations.Sampler$Map.map(Sampler.java:69)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
        at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:223)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
16/03/18 19:13:11 INFO mapred.JobClient:  map 0% reduce 0%
16/03/18 19:13:11 INFO mapred.JobClient: Job complete: job_local161844959_0001
16/03/18 19:13:11 INFO mapred.JobClient: Counters: 0
16/03/18 19:13:11 INFO mapred.JobClient: Job Failed: NA
java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1357)
        at edu.umn.cs.spatialHadoop.operations.Sampler.sampleMapReduceWithRatio(Sampler.java:211)
        at edu.umn.cs.spatialHadoop.operations.Sampler.sample(Sampler.java:543)
        at edu.umn.cs.spatialHadoop.operations.Repartition.packInRectangles(Repartition.java:494)
        at edu.umn.cs.spatialHadoop.operations.Repartition.packInRectangles(Repartition.java:463)
        at edu.umn.cs.spatialHadoop.operations.Repartition.repartitionLocal(Repartition.java:590)
        at edu.umn.cs.spatialHadoop.operations.Repartition.repartition(Repartition.java:676)
        at edu.umn.cs.spatialHadoop.operations.Repartition.main(Repartition.java:721)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
        at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
        at edu.umn.cs.spatialHadoop.operations.Main.main(Main.java:115)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions