[Proposal] Move Notebook functionality from dashboards-observability to dashboards-notebooks

[SuZhou-Joe]

Background

Due to the growing needs of the Notebook plugin, developing notebook features within the dashboards-observability repository has become challenging:

1. The dashboards-observability repository primarily focuses on OpenTelemetry data analysis (logs, metrics, and traces), containing thousands of files and 36 CI workflows.
2. The combination of different plugins makes it difficult for maintainers to focus effectively and creates a high barrier for new maintainers, who must be knowledgeable in both observability and notebook functionality.

Motivation

1. Reduced Codebase Complexity: The dashboards-observability repository currently contains thousands of files, while notebook functionality only requires tens of files. Moving notebook code to a dedicated repository will significantly reduce maintenance overhead and make the codebase more focused.

2. Faster Development Cycle: As a standalone plugin, notebook feature development can proceed independently without being coupled to the larger observability codebase. This separation will enable:

   • Faster iterations on notebook-specific features
   • Simplified testing and validation
   • Independent release cycles when needed
   • Reduced risk of conflicts with other observability features

3. Better Resource Allocation: A dedicated repository allows for:

   • Clear ownership and responsibility
   • More focused code reviews
   • Easier onboarding for new contributors
   • Simplified dependency management

4. Enhanced Feature Development: With notebooks becoming a standalone plugin:

   • New features can be developed and tested in isolation
   • Integration testing becomes more straightforward
   • Performance optimizations can be implemented without affecting other components

Challenge

1. Code Migration Effort:

   • Large number of files and dependencies to move
   • Need to carefully identify and handle all notebook-related code
   • Risk of missing dependencies or breaking existing functionality
   • Significant testing effort required to ensure everything works after migration
   • Time investment needed for setup and configuration of new repository

2. Backward Compatibility:

   • Must maintain existing APIs and interfaces
   • Need to ensure no disruption to current users
   • Challenge in managing version compatibility between repositories
   • Migration path must be smooth for existing installations

3. Maintainance effort:

   • New plugins requires build / deployment / release / test into official opensearch plugin release cycle.
   • New maintainers required for the new repo to triage issues, bump versions.

Mitigation

1. Code Migration Effort:

   • We've created a branch for PoC and it works fine locally:
   • dashboards-notebook: https://github.com/SuZhou-Joe/dashboards-observability/tree/feature/standalone-notebook

2. Backward Compatibility:

   • We will add notebook plugin detection logic inside dashboards-observability so that once notebook plugin is installed, observability wont register notebook related features into system.
   • branch: https://github.com/SuZhou-Joe/dashboards-observability/tree/notebook_compatible

3. Maintainance effort

   • A clear ownership could help to accelerate feature requirement / bug-fix and more community maintainers shall be attracted for the tiny powerful plugin.

[ps48]

Issue is being discussed here: opensearch-project/.github#339