Skip to content
This repository has been archived by the owner on Jun 26, 2020. It is now read-only.

YARN memory default vritual memory settings limits #56

Open
Hokan-Ashir opened this issue Jul 22, 2016 · 0 comments
Open

YARN memory default vritual memory settings limits #56

Hokan-Ashir opened this issue Jul 22, 2016 · 0 comments

Comments

@Hokan-Ashir
Copy link

If you would like to write custom YARN application, i.e. via Spring Boot, using this manual (https://spring.io/guides/gs/yarn-basic/) you will face couple of errors:

  1. One issue about guide.
    Make sure you have corresponding versions of "spring-yarn-boot", "spring-boot-starter-parent" and "spring-boot-maven-plugin" in pom-files. Simpliest solution go to maven central and use latest versions.
    Otherwise you may face "cannot instantiate 'yarnClient' bean" error, which tell you almost nothing why it failed
  2. Another about this docker image.
    Make sure you set limitless virtual memory for YARN.
    For example, you may set in yarn-site.xml this properties:
<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
    <description>Whether virtual memory limits will be enforced for containers</description>
  </property>
  <property>
    <name>yarn.nodemanager.vmem-pmem-ratio</name>
    <value>4</value>
    <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
  </property>

Cause you use it in Docker, you have to change it somehow after hadoop cluster initiation, or before, mapping volumes with "-v" option
Another solution is to use fork of this docker image, which has this settings by default, i.e. - https://github.com/sfedyakov/hadoop-271-cluster

@Hokan-Ashir Hokan-Ashir changed the title YARN memory defaults vritual memory settings limits YARN memory default vritual memory settings limits Jul 22, 2016
Hokan-Ashir added a commit to Hokan-Ashir/HadoopTest that referenced this issue Jul 22, 2016
…ation

 to make it work pay attention to sequenceiq/hadoop-docker#56 issue description
 also minor pom-files fixes
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant