bootRun: Could not locate executable null\bin\winutils.exe in the Hadoop binaries And throwing NullPointerException when initLocalMetastore() #545
Description
I encountered some errors getting started with Coral. I'm using jdk8 and Python3.8.
Under coral-service, I executed: ../gardlew bootRun --args='--spring.profiles.active=localMetastore'
then follows the logs:
Task :coral-service:bootRun
. ____ _ __ _ _
/\ / ' __ _ () __ __ _ \ \ \
( ( )__ | '_ | '| | ' / ` | \ \ \
\/ )| |)| | | | | || (| | ) ) ) )
' || .__|| ||| |_, | / / / /
=========||==============|/=////
:: Spring Boot :: (v2.4.0)
2025-01-09 18:34:04.871 INFO 3216 --- [ main] c.l.c.c.CoralServiceApplication : Starting CoralServiceApplication using Java 1.8.0_291 on LAPTOP-2S6ONLIK with PID 3216 (E:\project\coral\coral-service\build\classes\java\main started by xh254 in E:\project\coral\coral-service)
2025-01-09 18:34:04.873 INFO 3216 --- [ main] c.l.c.c.CoralServiceApplication : The following profiles are active: localMetastore
2025-01-09 18:34:06.194 INFO 3216 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2025-01-09 18:34:06.204 INFO 3216 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2025-01-09 18:34:06.204 INFO 3216 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.39]
2025-01-09 18:34:06.337 INFO 3216 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2025-01-09 18:34:06.337 INFO 3216 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1330 ms
2025-01-09 18:34:06.474 INFO 3216 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor'
2025-01-09 18:34:06.557 WARN 3216 --- [ main] ion$DefaultTemplateResolverConfiguration : Cannot find template location: classpath:/templates/ (please add some templates or check your Thymeleaf configuration)
2025-01-09 18:34:06.614 INFO 3216 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2025-01-09 18:34:06.631 ERROR 3216 --- [ main] org.apache.hadoop.util.Shell : Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:356) [hadoop-common-2.7.0.jar:na]
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:371) [hadoop-common-2.7.0.jar:na]
at org.apache.hadoop.util.Shell.(Shell.java:364) [hadoop-common-2.7.0.jar:na]
at org.apache.hadoop.hive.conf.HiveConf$ConfVars.findHadoopBinary(HiveConf.java:2333) [hive-common-1.2.2.jar:1.2.2]
at org.apache.hadoop.hive.conf.HiveConf$ConfVars.(HiveConf.java:368) [hive-common-1.2.2.jar:1.2.2]
at org.apache.hadoop.hive.conf.HiveConf.(HiveConf.java:108) [hive-common-1.2.2.jar:1.2.2]
at com.linkedin.coral.coralservice.utils.CoralProvider.loadResourceHiveConf(CoralProvider.java:86) [main/:na]
at com.linkedin.coral.coralservice.utils.CoralProvider.initLocalMetastore(CoralProvider.java:53) [main/:na]
at com.linkedin.coral.coralservice.controller.TranslationControllerLocal.onApplicationEvent(TranslationControllerLocal.java:36) [main/:na]
at com.linkedin.coral.coralservice.controller.TranslationControllerLocal.onApplicationEvent(TranslationControllerLocal.java:27) [main/:na]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:203) [spring-context-5.3.1.jar:5.3.1]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:196) [spring-context-5.3.1.jar:5.3.1]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:161) [spring-context-5.3.1.jar:5.3.1]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:426) [spring-context-5.3.1.jar:5.3.1]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:383) [spring-context-5.3.1.jar:5.3.1]
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:945) [spring-context-5.3.1.jar:5.3.1]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:591) [spring-context-5.3.1.jar:5.3.1]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:144) [spring-boot-2.4.0.jar:2.4.0]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:767) [spring-boot-2.4.0.jar:2.4.0]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759) [spring-boot-2.4.0.jar:2.4.0]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:426) [spring-boot-2.4.0.jar:2.4.0]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:326) [spring-boot-2.4.0.jar:2.4.0]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1309) [spring-boot-2.4.0.jar:2.4.0]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1298) [spring-boot-2.4.0.jar:2.4.0]
at com.linkedin.coral.coralservice.CoralServiceApplication.main(CoralServiceApplication.java:19) [main/:na]
2025-01-09 18:34:06.798 INFO 3216 --- [ main] c.l.c.c.CoralServiceApplication : Temp Workspace: C:\Users\xh254\AppData\Local\Temp/coral/service/d1ac1ebb-91cc-40b4-8614-928923d6c76a
2025-01-09 18:34:06.871 WARN 3216 --- [ main] org.apache.hadoop.util.NativeCodeLoader : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2025-01-09 18:34:06.897 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.HiveMetaStore : 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
2025-01-09 18:34:06.941 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.ObjectStore : ObjectStore, initialize called
log4j:WARN No appenders could be found for logger (DataNucleus.General).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2025-01-09 18:34:07.814 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.ObjectStore : Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
2025-01-09 18:34:08.896 INFO 3216 --- [ main] o.a.h.hive.metastore.MetaStoreDirectSql : Using direct SQL, underlying DB is DERBY
2025-01-09 18:34:08.897 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.ObjectStore : Initialized ObjectStore
2025-01-09 18:34:08.953 WARN 3216 --- [ main] o.a.hadoop.hive.metastore.ObjectStore : Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
2025-01-09 18:34:09.044 WARN 3216 --- [ main] o.a.hadoop.hive.metastore.ObjectStore : Failed to get database default, returning NoSuchObjectException
2025-01-09 18:34:09.138 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.HiveMetaStore : Added admin role in metastore
2025-01-09 18:34:09.141 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.HiveMetaStore : Added public role in metastore
2025-01-09 18:34:09.189 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.HiveMetaStore : No user is added in admin role, since config is empty
2025-01-09 18:34:09.247 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.HiveMetaStore : 0: get_all_databases
2025-01-09 18:34:09.247 INFO 3216 --- [ main] o.a.h.h.metastore.HiveMetaStore.audit : ugi=xh254 ip=unknown-ip-addr cmd=get_all_databases
2025-01-09 18:34:09.258 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.HiveMetaStore : 0: get_functions: db=default pat=*
2025-01-09 18:34:09.259 INFO 3216 --- [ main] o.a.h.h.metastore.HiveMetaStore.audit : ugi=xh254 ip=unknown-ip-addr cmd=get_functions: db=default pat=*
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:466)
at com.linkedin.coral.coralservice.utils.CoralProvider.initLocalMetastore(CoralProvider.java:64)
at com.linkedin.coral.coralservice.controller.TranslationControllerLocal.onApplicationEvent(TranslationControllerLocal.java:36)
at com.linkedin.coral.coralservice.controller.TranslationControllerLocal.onApplicationEvent(TranslationControllerLocal.java:27)
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:203)
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:196)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:161)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:426)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:383)
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:945)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:591)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:144)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:767)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:426)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:326)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1309)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1298)
at com.linkedin.coral.coralservice.CoralServiceApplication.main(CoralServiceApplication.java:19)
Caused by: java.lang.NullPointerException
Caused by: java.lang.NullPointerException
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:483)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:815)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:798)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:728)
at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:486)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:527)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:504)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:305)
at org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3678)
at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:597)
at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
... 19 more
2025-01-09 18:34:09.341 INFO 3216 --- [ main] c.l.c.c.CoralServiceApplication : Started CoralServiceApplication in 4.765 seconds (JVM running for 5.318)