Skip to content

build and deploy issue #21

@aymec

Description

@aymec

Hi,

I greatly enjoyed your article Lessons Learned from Running Apache Kafka at Scale at Pinterest. Thank you very much.
That is where I discovered this project, looking for an upgrade over CMAK.
Orion looks very promising, though I met a few issues when building and starting the components. Please let me know if I should split this or report it differently.

I have used the master branch, commit f94201ff9e106eff38c14e6e20c0674fafade584. It builds Orion version 0.0.33.

  1. The build-deployment.sh expects to be run from an Ubuntu 18.04 LTS, from the node debian repo addition for bionic. Only Ubuntu is mentioned in the QuickStart, not the version.
    Replacing bionic with $(grep -oP -m1 "CODENAME=\K(\w+)" /etc/os-release) in the script solves this, and works for debian distros as well.
    With this change, I have successfully built Orion on Ubuntu 20.04 (Focal Fossa), debian stretch and debian buster.
    For buster, a hack is necessary to install openjdk-8-jdk, since it is not available in the official repos for that version.

  2. The script needs to be run as sudo, to be able to add the node repo key, configure the apt sources, etc. but then it also builds the project as sudo, when the call to mvn clean Package is made.
    I am not sure how to solve that cleanly. Maybe adding the sudo where needed in the script ?

  3. The orion-agent debian package built has a dependency to openjdk-8-jdk instead of openjdk-8-jre
    I have not looked in the sources where to change that.

  4. To run the orion server, I have had to modify the run_orion.sh with a different JAVA_HOME value, which for openjdk-8-jre seems incorrect.
    On Ubuntu 20.04, debian stretch and debian buster, the JAVA_HOME is /usr/lib/jvm/java-8-openjdk-amd64.
    Also, the script could check if JAVA_HOME is already set.

  5. The run_orion.sh script needs to be run from /opt/orion-server/ otherwise it reports an error in /var/log/orion/orion-stdout.log: Error: Could not find or load main class com.pinterest.orion.server.OrionServer
    I believe it comes from settings the ORION_HOME parameter as follow ORION_HOME=$(pwd $DIR/..)

  6. After solving that, a new issue comes up when starting the server, the server outputs an exception in /var/log/orion/orion-stdout.log
    Commenting all endpointEnabled in the config file /opt/orion-server/conf/kafka-server.yaml allowed to start the server, but I guess I just lost some functions.

Exception details

io.dropwizard.configuration.ConfigurationParsingException: /opt/orion-server/conf/kafka-server.yaml has an error:
  * Unrecognized field at: plugins.actionConfigs.[8].endpointEnabled
    Did you mean?:
      - endpoint
      - enabled
      - configuration
      - class
      - key

	at io.dropwizard.configuration.ConfigurationParsingException$Builder.build(ConfigurationParsingException.java:279)
	at io.dropwizard.configuration.BaseConfigurationFactory.build(BaseConfigurationFactory.java:140)
	at io.dropwizard.configuration.BaseConfigurationFactory.build(BaseConfigurationFactory.java:89)
	at io.dropwizard.cli.ConfiguredCommand.parseConfiguration(ConfiguredCommand.java:126)
	at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:74)
	at io.dropwizard.cli.Cli.run(Cli.java:78)
	at io.dropwizard.Application.run(Application.java:93)
	at com.pinterest.orion.server.OrionServer.main(OrionServer.java:298)
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "endpointEnabled" (class com.pinterest.orion.core.configs.ActionConfig), not marked as ignorable (5 known properties: "class", "enabled", "endpoint", "configuration", "key"])
 at [Source: UNKNOWN; line: -1, column: -1] (through reference chain: com.pinterest.orion.server.config.OrionConf["plugins"]->com.pinterest.orion.server.config.OrionPluginConfig["actionConfigs"]->java.util.ArrayList[8]->com.pinterest.orion.core.configs.ActionConfig["endpointEnabled"])
	at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)
	at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:823)
	at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:1153)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1589)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1567)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:258)
	at com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize(SuperSonicBeanDeserializer.java:155)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:286)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:245)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:27)
	at com.fasterxml.jackson.module.afterburner.deser.SettableObjectMethodProperty.deserializeAndSet(SettableObjectMethodProperty.java:47)
	at com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize(SuperSonicBeanDeserializer.java:159)
	at com.fasterxml.jackson.module.afterburner.deser.SettableObjectMethodProperty.deserializeAndSet(SettableObjectMethodProperty.java:47)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:252)
	at com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize(SuperSonicBeanDeserializer.java:155)
	at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3985)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2277)
	at io.dropwizard.configuration.BaseConfigurationFactory.build(BaseConfigurationFactory.java:127)
	... 6 more

  1. Finally, a suggestion about the log folders. Since orion-server is installed in /opt/orion-server and orion-agent is installed in /opt/orion-agent, what about using /var/log/orion-server as the default path for the orion-server instead of /var/log/orion/ ?

This is actually where I stopped for now.

I have a question though about where the server and agents should be installed.
Is it recommended, or not, to install both the server and agent on the same machine ?
Should it be installed on one of the Kafka brokers or should it be installed separately ?

Thanks a lot for the great work.
Please let me know if I should submit pull requests for any of the points mentioned above.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions