Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feature: add file chunker microservice #2701

Closed
wants to merge 10 commits into from
Closed

feature: add file chunker microservice #2701

wants to merge 10 commits into from
+6,843 −5

Commits on Nov 15, 2024

  1. feature: add file chunker microservice

    @densumesh @skeptrunedev
    densumesh authored and skeptrunedev committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    094c8df View commit details
    Browse the repository at this point in the history

  2. feature: add logging and write to clickhouse

    @densumesh @skeptrunedev
    densumesh authored and skeptrunedev committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    cca1f68 View commit details
    Browse the repository at this point in the history

  3. feature: added api key and file chunking status

    @densumesh @skeptrunedev
    densumesh authored and skeptrunedev committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    f0f9299 View commit details
    Browse the repository at this point in the history

  4. feature: add CLI to test pdf chunking

    @densumesh @skeptrunedev
    densumesh authored and skeptrunedev committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    23ddaf7 View commit details
    Browse the repository at this point in the history

  5. cleanup: add .env.dist + use env's for model in pdf_chunk.rs

    @skeptrunedev
    skeptrunedev committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    e93788b View commit details
    Browse the repository at this point in the history

  6. feat: use pdla boxes for chunking 

    cleanup: move test program

feat: different prompt

feat: build metadata strings

fix: pdla server returns floats

feat: fix image cropping

feat: move pdla types

feat: new system prompt

feat: include tables

cleanup: remove test binary

fix: handle empty text better

fix: issues after rebase

fix: keep llm model in env
    @drew-harris @skeptrunedev
    drew-harris authored and skeptrunedev committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    8686c2e View commit details
    Browse the repository at this point in the history

  7. cleanup: clippy

    @drew-harris @skeptrunedev
    drew-harris authored and skeptrunedev committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    c8136f5 View commit details
    Browse the repository at this point in the history

  8. feat: add docker-compose.yml for local testing

    @drew-harris @skeptrunedev
    drew-harris authored and skeptrunedev committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    dd43a63 View commit details
    Browse the repository at this point in the history

  9. feature: pdf to md :)

    @densumesh @skeptrunedev
    densumesh authored and skeptrunedev committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    f58a161 View commit details
    Browse the repository at this point in the history

  10. feature: add page right after its been processed

    @densumesh @skeptrunedev
    densumesh authored and skeptrunedev committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    3e7758c View commit details
    Browse the repository at this point in the history