Adding autocompletion + scripts + standard deviation in 'mapper time' heuristic#275

Open

alexandre32 wants to merge 288 commits intolinkedin:masterfrom

alexandre32:master

This pull request is big! We're only showing the most recent 250 commits

Commits on Dec 13, 2014

HADOOP-8096: Property 'username' is converted to lowercase before supplying it to the query instead of asking the query to perform the case-insensitive comparison.
Akshay Rai
committed

Commits on Dec 15, 2014

[HADOOP-7814] Prepare to add metrics library to dr-elephant
mcvsubbu
committed

Commits on Dec 23, 2014

[HADOOP-7814] Publish counters via dr-elephant
mcvsubbu
committed

Commits on Jan 16, 2015

[HADOOP-8640] Fix hang in dr-elephant due to h1 and h2 incompatibility
mcvsubbu
committed

Commits on Jan 21, 2015

Commits on Jan 29, 2015

HADOOP-8294: Improved slow performance of join queries
Akshay Rai
committed

Commits on Feb 2, 2015

HADOOP-5369: Paginate search results of Dr. Elephant
Akshay Rai
committed

Commits on Feb 3, 2015

HADOOP-7948: Make Dr. Elephant's Reducer Time Moderate Severity cut-off 51 reducers instead of 50
Fangshi Li
committed
HADOOP-8320: Add detailed suggestions in Dr. Elephant help page
Fangshi Li
committed
HADOOP-8320: Add detailed suggestions in Dr. Elephant help page
Fangshi Li
committed
HADOOP-8856: Fix a broken Dr.Elephant test case for reducer time heuristic
Fangshi Li
committed

Commits on Feb 4, 2015

HADOOP-8859: Add ideal task time suggestion in Dr. Elephant 's reducer time help page
Fangshi Li
committed
HADOOP-8846: Release Dr. Elephant v0.6.5
Fangshi Li
committed

Commits on Feb 9, 2015

HADOOP-8929: Update Dr.Elephant version number to 0.6.6-SNAPSHOT
Fangshi Li
committed

Commits on Mar 30, 2015

HADOOP-9685: Dr. Elephant fails to fetch task counter data on hadoop2
Fangshi Li
committed

Commits on Mar 31, 2015

HADOOP-8941: Further improve search performance for Dr. Elephant
Akshay Rai
committed

Commits on Apr 2, 2015

HADOOP-9714: Add Pluggable jobtype to Dr. Elephant
Fangshi Li
committed
HADOOP-9716: Dr. Elephant rest interface needs search endpoint
Fangshi Li
committed
HADOOP-9853: Release Dr. Elephant v0.6.6
Fangshi Li
committed

Commits on Apr 6, 2015

HADOOP-9886: Fix for making Dr. Elephant's pagination thread safe
Akshay Rai
committed

Commits on Apr 20, 2015

HADOOP-10289: Update Dr. Elephant version number to 0.6.7-SNAPSHOT
Fangshi Li
committed
HADOOP-10089: Dr. Elephant misses on type of Voldemort bnp job
Fangshi Li
committed
HADOOP-10301: Dr. Elephant occasionally misses a job
Fangshi Li
committed

Commits on Apr 21, 2015

Commits on Apr 25, 2015

[HADOOP-10405] Fix one-off error in getting cluster name from NN hostname
mcvsubbu
committed

Commits on Jun 12, 2015

HADOOP-11768: Update Dr. Elephant version to v1.0.0-SNAPSHOT
Fangshi Li
committed

Commits on Jun 15, 2015

Commits on Jun 17, 2015

HADOOP-11876: Separate Dr. Elephant configs from deployable zip
Fangshi Li
committed

Commits on Jun 18, 2015

HDOOP-11456: Change Dr. Elephant's heuristic from MapperInputSize to MapperTime
Fangshi Li
committed
HADOOP-11922: Fix Dr. Elephant search by heuristic field
Fangshi Li
committed
HADOOP-11912: Release Dr. Elephant v1.0.0
Fangshi Li
committed

Commits on Jun 22, 2015

HADOOP-12013: Dr. Elephant has a corner case to analyze duplicate entries
Fangshi Li
committed

Commits on Jun 29, 2015

HADOOP-12247 - Update Dr. Elephant version to v1.0.1-SNAPSHOT
Fangshi Li
committed

Commits on Jul 9, 2015

HADOOP-11913: Fix Dr. Elephant broken tests
Fangshi Li
committed
HADOOP-12067: Dr. Elephant heuristic is too lenient to jobs with long running tasks
Fangshi Li
committed

Commits on Jul 10, 2015

HADOOP-12023: Dr. Elephagent new heuristics: memory util ratio
Fangshi Li
committed
HADOOP-12593: Release Dr. Elephant v1.0.1
Fangshi Li
committed

Commits on Jul 13, 2015

HADOOP-12632: Enable Dr. Elephant 's mapper spill heuristics
Fangshi Li
committed

Commits on Jul 14, 2015

HADOOP-12669: Dr. Elephant's mapper spill heuristic is too harsh
Fangshi Li
committed
HADOOP-12674: Release Dr. Elephant v1.0.2
Fangshi Li
committed
HADOOP-12603: Dr. Elephant has a failed test in Mapper Speed Heuristic
Fangshi Li
committed
HADOOP-12603: Update Mapper Spill Heuristic help page
Fangshi Li
committed

Commits on Jul 15, 2015

Commits on Jul 27, 2015

LIHADOOP-13019, LIHADOOP-13020: Upgrade to Spark 1.4, and add event log limit for Spark log fetcher
timyitong
committed

Commits on Jul 28, 2015

LIHADOOP-13018: Dr. Elephant 's Spill heuristic should be lenient to small number of tasks
Fangshi Li
committed

Commits on Jul 29, 2015

LIHADOOP-13094: Fix Dr. Elephant build on Jenkins after spark1.4 upgrade
Fangshi Li
committed
LIHADOOP-13016: Release Dr. Elephant v1.0.3
Fangshi Li
committed

Commits on Aug 21, 2015

Adjust StageRuntime heuristics so that it's the real duration time instead of total aggregated executor time
timyitong
committed

Commits on Aug 24, 2015

LIHADOOP-13747 Update Spark driver name to be consistent with Spark 1.4.
Victsm
committed

Commits on Sep 1, 2015

LIHADOOP-13953 The start script should take arguments to enable evolutions
Akshay Rai
committed
LIHADOOP-5177 Update documentation of Dr. Elephant
Akshay Rai
committed

Commits on Sep 9, 2015

LIHADOOP-12802 Occasional Concurrent Modification Exception during Dr. Elephant start
Akshay Rai
committed

Commits on Sep 11, 2015

LIHADOOP-8114 [Opensource Dr. E] Apache License and LinkedIn Copyright Notices

akshayrai
authored and
Akshay Rai
committed
Minor formatting change in Readme License
Akshay Rai
committed

Commits on Sep 14, 2015

Commits on Sep 29, 2015

LIHADOOP-14322 Dr. Elephant feature request. The ability to compare result for two flow executions
Akshay Rai
committed

Commits on Sep 30, 2015

LIHADOOP-14722: Adjust some Spark heuristic parameters of Dr Elephant
timyitong
committed

Commits on Oct 8, 2015

LIHADOOP-14688 Adding a timestamp to Dr. Elephant search result page
Akshay Rai
committed

Commits on Oct 14, 2015

LIHADOOP-13819: Dr. Elephant heuristic request. GC efficiency
Fangshi Li
committed
LIHADOOP-13711: Dr. Elephant should add virtual memory info in its memory heuristics
Fangshi Li
committed
LIHADOOP-15134: Dr. Elephant should provide exception stacktrace from failed jobs
Fangshi Li
committed
LIHADOOP-14342: Dr. Elephant's memory heuristic has a corner case to throw NumberFormatException
Fangshi Li
committed
LIHADOOP-15136: Release Dr. Elephant v1.0.5
Fangshi Li
committed

Commits on Oct 22, 2015

LIHADOOP-15080 Clean up and organize the views in Dr. Elephant
Akshay Rai
committed
LIHADOOP-15252: Update Dr. Elephant version to v1.0.6-SNAPSHOT
Fangshi Li
committed

Commits on Oct 26, 2015

LIHADOOP-15372 Update the Heuristic Conf for helpGC with the proper package name
akshayrai
committed

Commits on Oct 28, 2015

LIHADOOP-15377 Job execution URLs on Dr. Elephant's Flow Execution page is incorrect
akshayrai
committed

Commits on Dec 18, 2015

LIHADOOP-16125 Other executions of a job should list the recent jobs from history firBt
Akshay Rai
committed

Commits on Dec 23, 2015

LIHADOOP-15217 Implement comparison of historical flows/jobs in Dr. Elephant
Akshay Rai
committed
LIHADOOP-16102 Discuss and remove Hadoop 1 support from Dr. Elephant
Akshay Rai
committed

Commits on Jan 6, 2016

LIHADOOP-9004 Support Hadoop 2 properties in Dr. Elephant
Akshay Rai
committed

Commits on Jan 14, 2016

LIHADOOP-16687 Update Dr. Elephant to use the new Hadoop API
Akshay Rai
committed

Commits on Jan 29, 2016

LIHADOOP-16576 Make severity threshold values configurable for the heuristics.

Akshay Rai
authored and
akshayrai
committed

Commits on Feb 1, 2016

LIHADOOP-17131 Prepare Readme for Dr. Elephant
akshayrai
committed

Commits on Feb 12, 2016

LIHADOOP-16587 Dr. Elephant should provide search by start time
akshayrai
committed

Commits on Feb 15, 2016

Commits on Feb 22, 2016

LIHADOOP-16100: Dr. Elephant should show timestamp specific to timezones or at least UTC.
akshayrai
committed

Commits on Feb 25, 2016

LIHADOOP-15268: Task sampling in Dr. Elephant
Fangshi Li
committed

Commits on Mar 1, 2016

Commits on Mar 7, 2016

Commits on Mar 8, 2016

Commits on Mar 14, 2016

Commits on Apr 7, 2016

Commits on Apr 11, 2016

Commits on Apr 19, 2016

Get the active resource manager URL when HA is enabled
Krishna Puttaswamy
committed
fix start script to redirect output to a file instead of /dev/null
Krishna Puttaswamy
committed
addressed code review comments on RM HA support PR
Krishna Puttaswamy
committed
Merge pull request #15 from krishnap/kp_fix_start_script
akshayrai
committed
Merge pull request #14 from krishnap/kp_fix_rm_issue
akshayrai
committed

Commits on Apr 22, 2016

LIHADOOP-19325: [DR-Elephant] Add integration tests for the rest API's in Dr. Elephant

rajagopr
authored and
akshayrai
committed

Commits on Apr 26, 2016

Added a link to the mailing list.
akshayrai
committed

Commits on Apr 27, 2016

Commits on Apr 28, 2016

Commits on May 2, 2016

LIHADOOP-19610: Metrics should not be shown if runtime and resourceused are 0
nntnag17
committed

Commits on May 4, 2016

LIHADOOP-19609: Make Spark Event Log Directory configurable
akshayrai
committed

Commits on May 5, 2016

Fix an cast error in SparkDataCollection.scala (#37)

stiga-huang
authored and
akshayrai
committed

Commits on May 6, 2016

Separate configuration logic from RM address update logic (#38)

tglstory
authored and
akshayrai
committed

Commits on May 10, 2016

LIHADOOP-19850: Test for AnalyticJob (#42)

rajagopr
authored and
akshayrai
committed

Commits on May 18, 2016

LIHADOOP-20088: Add organization name to Dr. Elephant to generate ivy file (#46)

rajagopr
authored and
akshayrai
committed

Commits on May 20, 2016

LIHADOOP-18924: Improve Job History to show runtime, waittime, resource used, resource wasted.
nntnag17
committed

Commits on May 23, 2016

LIHADOOP-20170: Write help pages for new metrics in Dr. Elephant
nntnag17
committed

Commits on May 28, 2016

Added engineering blog link
akshayrai
committed

Commits on Jun 1, 2016

LIHADOOP-20476: Generating code coverage report in xml format too. This will be consumed by the product dashboard. (#65)

rajagopr
authored and
akshayrai
committed

Commits on Jun 6, 2016

Commits on Jun 8, 2016

Removing some un-necessary logic. (#70)

rajagopr
authored and
akshayrai
committed

Commits on Jun 9, 2016

Added support to search by queue-name (#40)

chetnachaudhari
authored and
akshayrai
committed

Commits on Jun 18, 2016

Try alternate resource manager if one is down (#78)

paulbramsen
authored and
akshayrai
committed
Ignore and remove .DS_Store files (#80)

paulbramsen
authored and
akshayrai
committed

Commits on Jun 22, 2016

Add GeneralConf.xml to make number of executors configurable (#75)

stiga-huang
authored and
akshayrai
committed

Commits on Jun 23, 2016

Add spark log ext conf param (#79)

paulbramsen
authored and
akshayrai
committed
Fix fallback to default `SPARK_LOG_EXT` value (#82)

ljank
authored and
akshayrai
committed

Commits on Jun 25, 2016

Add Travis CI support (#86)

paulbramsen
authored and
akshayrai
committed

Commits on Jul 1, 2016

Add Airflow scheduler support to Dr. Elephant. Includes Tests. (#83)

paulbramsen
authored and
akshayrai
committed
Make default container mem configurable (#90)

paulbramsen
authored and
akshayrai
committed

Commits on Jul 5, 2016

Web analytics for Dr. Elephant. (#94)

rajagopr
authored and
akshayrai
committed

Commits on Jul 7, 2016

Fix job history page tab highlighting (#98)

paulbramsen
authored and
akshayrai
committed
dr-elephant#95: copy conf files to zip package automatically (#96)

YannByron
authored and
akshayrai
committed

Commits on Jul 14, 2016

Add Gitter badge (#102)

gitter-badger
authored and
akshayrai
committed
Put build and gitter badges on the same line (#105)

paulbramsen
authored and
akshayrai
committed

Commits on Jul 15, 2016

Commits on Jul 18, 2016

Add 3.sql and revert modifications in 1.sql
nntnag17
committed

Commits on Jul 20, 2016

Merge pull request #109 from nntnag17/master
nntnag17
authored

Commits on Jul 21, 2016

Make flow search more powerful (#99)

paulbramsen
authored and
akshayrai
committed
Add thread to all log output for better debugging (#108)

paulbramsen
authored and
akshayrai
committed

Commits on Jul 22, 2016

Fix Flow/Job ID/Url mixups (#100)

paulbramsen
authored and
akshayrai
committed

Commits on Jul 25, 2016

fix a log error when throttling spark event log (#114)

stiga-huang
authored and
akshayrai
committed

Commits on Jul 27, 2016

LIHADOOP-21568: Add Dropwizard metrics to Dr. Elephant. This change enables metrics by default. (#115)

rajagopr
authored and
akshayrai
committed

Commits on Jul 28, 2016

Add default for boolean unboxing (#119)

paulbramsen
authored and
akshayrai
committed

Commits on Aug 1, 2016

Fix sync issue in info extractor (#118)

paulbramsen
authored and
akshayrai
committed
Modify main run loop to be inline with thread pool executor paradigms (#121)

paulbramsen
authored and
akshayrai
committed
Faster fetcher for MapReduce apps (#67)

stiga-huang
authored and
akshayrai
committed

Commits on Aug 2, 2016

Show result count in search (#120)

paulbramsen
authored and
akshayrai
committed
LIHADOOP-21657: Release Dr. Elephant v. 2.0.5 (#117)

rajagopr
authored and
akshayrai
committed

Commits on Aug 8, 2016

Add rest interface for getting resource usage by user (#125)

shankar37
authored and
akshayrai
committed
Improved metrics (#126)

paulbramsen
authored and
akshayrai
committed

Commits on Aug 29, 2016

Add Metrics feature for Spark Jobs (#135)

shankar37
authored and
akshayrai
committed

Commits on Sep 14, 2016

Changes to load an agent jar along with Dr. Elephant. Current use-case is to publish the metrics
rajagopr
committed

Commits on Oct 3, 2016

Add rest end point for new user interface
nntnag17
committed

Commits on Oct 11, 2016

LIHADOOP-23923: Add rest end points for search results and search options
nntnag17
committed

Commits on Oct 19, 2016

Added License to all the new files
nntnag17
committed

Commits on Oct 24, 2016

Integration of the ember app with play framework
nntnag17
committed

Commits on Oct 26, 2016

Commits on Nov 7, 2016

Commits on Nov 9, 2016

Multiple fixes: (#161)
nntnag17
authored

Commits on Nov 17, 2016

Oozie scheduler support (#152)

NoamShaish
authored and
akshayrai
committed

Commits on Nov 23, 2016

Add null checks around update methods (#164)

babak-altiscale
authored and
akshayrai
committed

Commits on Dec 12, 2016

logger actually returned number of application types, not job types (#169)

Clemens Valiente
authored and
akshayrai
committed
add exclude_jobs_filter to heuristics that is checked in AnalyticJob before running heuristic (#168)

Clemens Valiente
authored and
akshayrai
committed
fix superParentId - now also returns the Coordinator id for subworkflows (#167)

Clemens Valiente
authored and
akshayrai
committed
use an ordered map to preserve order of jobs (#170)

Clemens Valiente
authored and
akshayrai
committed

Commits on Dec 13, 2016

Commits on Jan 5, 2017

Commits on Jan 6, 2017

LIHADOOP-25142: Build the user-summary page for Dr. Elephant (#174)
nntnag17
authored

Commits on Jan 8, 2017

Removes play dependencies to read config file and read system properties. (#183)

shkhrgpt
authored and
akshayrai
committed

Commits on Jan 13, 2017

LIHADOOP-25809: Update Dr. Elephant README to point to Quick Setup and How to Contribute page (#185)
akshayrai
authored

Commits on Jan 15, 2017

Scheduler-less support for flow and job history (#181)

vaxvms
authored and
akshayrai
committed

Commits on Jan 24, 2017

Show exceptions for failed workflows (#188)
nntnag17
authored

Commits on Feb 6, 2017

Added new heuristic DistributedCacheLimit heuristic. (#187)

rajagopr
authored and
akshayrai
committed

Commits on Feb 7, 2017

Cleanes up MapReduceTaskData class by removing unnecessary constructors (#202)

shkhrgpt
authored and
akshayrai
committed
Fixes Spark REST fetcher for client mode applications (#193)

shkhrgpt
authored and
akshayrai
committed

Commits on Feb 9, 2017

Fix for null pointers in TaskList returned by MapReduceFSFetcherHadoop2 (#203)

stiga-huang
authored and
akshayrai
committed

Commits on Feb 22, 2017

Fix #162 with the right calculation for resourceswasted and add missing workflow links (#207)

shankar37
authored and
akshayrai
committed

Commits on Feb 23, 2017

Fix Exception thrown when JAVA_EXTRA_OPTIONS is not present (#210)

shankar37
authored and
akshayrai
committed

Commits on Feb 28, 2017

Adds an option to fetch recently finished apps from RM (#212)

shkhrgpt
authored and
akshayrai
committed
Fixes issue caused by http in history server config property (#217)

shkhrgpt
authored and
akshayrai
committed
add config for timezone of job history server (#214)

stiga-huang
authored and
akshayrai
committed

Commits on Mar 4, 2017

Include reference to the weekly meeting
akshayrai
authored

Commits on Mar 28, 2017

Fixes MapReduce aggregator and heuristic to correctly handle task data when sampling is enabled (#222)

shkhrgpt
authored and
akshayrai
committed

Commits on Apr 12, 2017

Spark fetcher is now able to fetch event logs via REST API (#225)

superbobry
authored and
akshayrai
committed

Commits on Apr 14, 2017

Update #224 (credits: rayortigas) to add FSFetcher as a standalone fetcher (#232)

shankar37
authored and
akshayrai
committed

Commits on Apr 18, 2017

Spark metrics aggregator fix (#237)

shankar37
authored and
akshayrai
committed

Commits on Apr 19, 2017

Minor bug fixes in exception and UI (#238)

nntnag17
authored and
akshayrai
committed

Commits on Apr 20, 2017

Updates Spark configuration heuristic severity calculations (#229)

shkhrgpt
authored and
akshayrai
committed

Commits on May 3, 2017

Enables SparkFetcher to only get eventLog via rest and process it locally. (#243)

shkhrgpt
authored and
akshayrai
committed

Commits on May 23, 2017

Refactor statusapiv1 to trait and implement for ease of creation of these objects when we implement our own parser (#248)

shankar37
authored and
akshayrai
committed

Commits on May 30, 2017

MRfetcher ignores failed tasks (#249)

shankar37
authored and
akshayrai
committed

Commits on Jun 7, 2017

Add index on severity, finish_time to speed up welcome page display (#250)

ashangit
authored and
akshayrai
committed

Commits on Jun 12, 2017

Add pinball scheduler to dr-elephant (#253)

georgewu2
authored and
akshayrai
committed

Commits on Jun 21, 2017

add s3, s3a, s3n bytes read and bytes written, and update heuristics to use them (#254)

georgewu2
authored and
akshayrai
committed

Commits on Jul 28, 2017

Add custom flowtime per scheduler (#268)

Maxime Kestemont
authored and
akshayrai
committed
Add filtering on Job Definition Id in the Search view (#269)

Maxime Kestemont
authored and
akshayrai
committed
added logic for map reduce time-skew heuristic (#267)

skakker
authored and
akshayrai
committed

Commits on Aug 10, 2017

update.sh and update.py scripts for automatic adding new heuristic + autocompletion search fild in tab job history
saga_install
committed
Standard deviation in mapper time heuristic
saga_install
committed