Documentation Refactoring Plan: MkDocs + GitHub Pages

[emmanuelmathot]

Overview

We're planning to refactor our documentation to improve organization, readability, and maintainability. The new documentation will use MkDocs with the Material theme and will be published to GitHub Pages.

Goals

• Organize documentation in a logical, user-friendly structure
• Implement MkDocs with Material theme for a modern look and feel
• Document deployment with multiple cloud providers (GKE, EKS, AKS, generic K8s)
• Document various PostgreSQL options (CrunchyData, RDS, Azure, GCP, self-managed)
• Automate documentation deployment with GitHub Actions
• Improve content with diagrams, examples, and troubleshooting guides

Proposed Structure

docs/
├── index.md                           # Main landing page
├── what-is-eoapi.md                   # Introduction to eoAPI
├── setup-kubernetes/                  # Kubernetes setup guides
│   ├── index.md                       # Overview of Kubernetes setup options
│   ├── minikube.md                    # Local setup with Minikube
│   ├── aws-eks.md                     # AWS EKS setup
│   ├── gcp-gke.md                     # GCP GKE setup
│   ├── azure-aks.md                   # Azure AKS setup (new)
│   ├── cloudferro.md                  # Cloudferro setup (new - European cloud providers)
│   ├── generic-k8s.md                 # Generic Kubernetes setup (new)
│   └── terraform.md                   # Terraform-based setup
├── setup-eoapi/                       # eoAPI installation guides
│   ├── index.md                       # Overview of installation process
│   ├── helm-install.md                # Basic Helm installation
│   ├── configuration.md               # Configuration options
│   ├── authentication/                # Authentication documentation (new)
│   │   ├── index.md                   # Overview of authentication options
│   │   ├── stac-auth-proxy.md         # STAC Auth Proxy integration
│   │   └── custom-auth.md             # Custom authentication implementation
│   ├── secret-management/             # Secret management documentation (new)
│   │   ├── index.md                   # Overview of secret management
│   │   ├── kubernetes-secrets.md      # Using Kubernetes secrets
│   │   ├── aws-secrets-manager.md     # AWS Secrets Manager integration
│   │   ├── azure-key-vault.md         # Azure Key Vault integration
│   │   └── hashicorp-vault.md         # HashiCorp Vault integration
│   ├── database-options/              # Database setup options
│   │   ├── index.md                   # Overview of database options
│   │   ├── crunchydata-operator.md    # CrunchyData PostgreSQL Operator
│   │   ├── aws-rds.md                 # AWS RDS (new)
│   │   ├── azure-postgres.md          # Azure Database for PostgreSQL (new)
│   │   ├── gcp-cloudsql.md            # GCP Cloud SQL (new)
│   │   └── self-managed.md            # Self-managed PostgreSQL (new)
│   └── uninstalling.md                # Uninstallation guide
├── administrator-guide/               # Administration documentation
│   ├── index.md                       # Overview for administrators
│   ├── architecture.md                # System architecture
│   ├── monitoring.md                  # Monitoring and observability
│   ├── autoscaling.md                 # Autoscaling configuration
│   ├── health-checks.md               # Health checks and liveness probes
│   ├── load-testing.md                # Load testing guidance
│   ├── security.md                    # Security considerations (new)
│   ├── upgrading.md                   # Upgrade procedures (new)
│   └── troubleshooting.md             # Troubleshooting guide (new)
├── user-guide/                        # User documentation
│   ├── index.md                       # Overview for users
│   ├── manage-data.md                 # Data management
│   ├── data-ingestion.md              # Data ingestion guide (new)
│   ├── configure-queryables.md        # Configure STAC queryables (new)
│   ├── api-usage.md                   # API usage examples (new)
│   └── client-libraries.md            # Client libraries (new)
├── developer-guide/                   # Developer documentation
│   ├── index.md                       # Overview for developers
│   ├── contributing.md                # Contribution guidelines (new)
│   └── release-process.md             # Release workflow
└── resources/                         # Additional resources
    ├── index.md                       # Overview of resources
    ├── faq.md                         # Frequently asked questions (new)
    ├── glossary.md                    # Glossary of terms (new)
    ├── related-projects.md            # Related projects (new)
    └── case-studies.md                # User testimonials and case studies (new)

Implementation Plan

• Phase 1: Initial Structure and Migration #218

  • Set up MkDocs with Material theme
  • Create directory structure
  • Migrate existing content
  • Implement GitHub Actions workflow

• Phase 2: Content Enhancement

  • Improve existing documentation
  • Add diagrams and visual aids
  • Create new content for missing sections
  • Develop comprehensive database options documentation
  • Add troubleshooting guides and FAQs

• Phase 3: Cloud Provider Documentation

  • Enhance AWS EKS documentation
  • Enhance GCP GKE documentation
  • Create Azure AKS documentation
  • Create Cloudferro documentation (European cloud provider)
  • Create generic Kubernetes documentation
  • Ensure consistent structure across all cloud provider guides

• Phase 4: Database Options Documentation

  • Document CrunchyData PostgreSQL Operator in detail (priority)
  • Create AWS RDS integration guide
  • Create Azure Database for PostgreSQL integration guide
  • Create GCP Cloud SQL integration guide
  • Document self-managed PostgreSQL setup

• Phase 5: Authentication and Security Documentation

  • Document STAC Auth Proxy integration
  • Document secret management approaches
  • Document security best practices
  • Create examples for external secret stores

• Phase 6: Review and Polish

  • Review all documentation for accuracy
  • Ensure consistent style and terminology
  • Add cross-references between related sections
  • Optimize for search engine discoverability
  • Gather feedback from team members and users

MkDocs Configuration

Proposed mkdocs.yml configuration:

site_name: eoapi-k8s
site_url: https://developmentseed.org/eoapi-k8s/
repo_url: https://github.com/developmentseed/eoapi-k8s
repo_name: developmentseed/eoapi-k8s
edit_uri: edit/main/docs/

theme:
  name: material
  logo: assets/eoapi-k8s.svg
  favicon: assets/favicon.ico
  palette:
    primary: indigo
    accent: indigo
  features:
    - navigation.tabs
    - navigation.sections
    - navigation.expand
    - navigation.indexes
    - content.code.copy
    - content.tabs.link

markdown_extensions:
  - admonition
  - pymdownx.details
  - pymdownx.superfences
  - pymdownx.tabbed
  - pymdownx.emoji
  - pymdownx.tasklist
  - pymdownx.highlight
  - toc:
      permalink: true

plugins:
  - search
  - mermaid2
  - git-revision-date-localized

GitHub Actions Workflow

We'll implement a GitHub Actions workflow to automatically build and deploy the documentation to GitHub Pages:

name: Deploy MkDocs to GitHub Pages

on:
  push:
    branches:
      - main
    paths:
      - 'docs/**'
      - 'mkdocs.yml'
      - '.github/workflows/docs.yml'

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.10'
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install mkdocs-material mkdocs-git-revision-date-localized-plugin mkdocs-mermaid2-plugin
      - name: Deploy documentation
        run: mkdocs gh-deploy --force

Integration with Main eoAPI Documentation

Throughout the documentation, we will link to the main eoAPI documentation at https://eoapi.dev/ for API-specific details and usage. This will ensure that users have access to the most up-to-date API documentation while keeping the eoapi-k8s documentation focused on deployment and infrastructure concerns.

Priority Areas Based on Team Feedback

Based on team feedback, we will prioritize the following areas:

1. Authentication documentation: Add comprehensive documentation on authentication options, with a focus on the STAC Auth Proxy integration.

2. Secret management: Include examples and tutorials for using external secret stores, particularly for cloud providers.

3. Queryables configuration: Add a dedicated guide on configuring queryables in the STAC API.

4. European cloud providers: Prioritize documentation for European cloud providers like Cloudferro alongside the major cloud providers.

5. CrunchyData PostgreSQL Operator: Provide detailed documentation on the CrunchyData PostgreSQL Operator as the preferred database option.

Next Steps

1. Review and discuss this updated plan
2. Assign responsibilities for different sections
3. Set up the initial MkDocs structure
4. Begin content migration and creation

Approach to Case Studies and Testimonials

We'll start by creating a "Related Projects" page that will include:

• Links to projects using eoapi-k8s
• Brief descriptions of how they're using it
• Technical highlights and lessons learned

As more content becomes available, we can expand this into a dedicated case studies section with more detailed testimonials and implementation stories.

[emmanuelmathot]

@pantierra @j08lue @sunu @ividito @batpad please let me know

[sunu]

@emmanuelmathot This is super impressive and really thorough! 🙌 A few ideas that might be cool to add:

• Maybe a section on authentication? Even just an overview of the current state of things would be helpful -- could link to the stac-auth-proxy project.
• A bit on secret management could be useful too. Especially some examples on how to use an external secret store.
• And for the data management user guide, adding a quick walkthrough on how to configure queryables and such would be great I think

[j08lue]

Great plan - I think the structure by role will work well (setup / sys admin, administrator, user / data manager, developer).

Questions for the Team

1. Are there any additional sections or topics we should include?

What @sunu said!

2. Should we prioritize certain cloud providers or database options?

The EOEPCA defaults - European cloud providers like Cloudferro, crunchy postgres

3. Do we want to include user testimonials or case studies?

Let's see what content we have for that and then decide. To start, we can perhaps keep a loose link reel on the "related projects" page you propose to keep under /resources, or so.

[emmanuelmathot]

Thank you @j08lue and @sunu for the feedback. I have updated the issue with your input.
I will convert soon each phase to specific tickets to start working on the doc.

[pantierra]

Looks really great. Another topic we might want to add: ingestion of data.

[emmanuelmathot]

@ciaransweet Please fell free to review and comment

[ciaransweet]

Looks good to me - Do we/should we have a (as one click as we can) one-click get started/demo option?

[emmanuelmathot]

Looks good to me - Do we/should we have a (as one click as we can) one-click get started/demo option?

I will integrate that option. Do you have a framework to do that using temporary and small resources? (e.g. binder)

[ciaransweet]

Looks good to me - Do we/should we have a (as one click as we can) one-click get started/demo option?

I will integrate that option. Do you have a framework to do that using temporary and small resources? (e.g. binder)

I do not.

I guess the suggestion would be minikube? Just abstract it away with an easy to run script or something