Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pipeline interfaces #186

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
+882 −669
Open

Pipeline interfaces #186

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
bcf4577
Pipeline interfaces and compatibility
hejny Nov 28, 2024
42bf43e
Pipeline interfaces and compatibility
hejny Nov 28, 2024
2e34023
TODOs
hejny Nov 28, 2024
fe37980
Merge branch 'main' into feature/pipeline-interfaces
hejny Nov 28, 2024
2ef583b
Draft of changelog
hejny Nov 28, 2024
6d1cce3
Merge branch 'main' into feature/pipeline-interfaces
hejny Dec 27, 2024
d941784
Merge branch 'main' into feature/pipeline-interfaces
hejny Dec 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -630,6 +630,30 @@ Implicit formfactors
- You don't need to specify the formfactor or input+output params explicitly. Implementing the formfactor interface is sufficient.
- Fix in deep cloning of arrays

## In Pre-release

### `0.!!!!!!.0` _(202--)_

compatibility

[🐠] - to PROGRAMMING
isXxx
validateXxx
checkXxx

is
check
validate

ACRY DoesNotMakeSenseError search for sense

'IDENTICAL'
'IDENTICAL_UNPREPARED'
'IDENTICAL_INTERFACE'
'IDENTICAL_NAMES'
'INTERSECTING'
'NON_INTERSECTING'

## Drafts

### `0..0` _(2024--)_
296 changes: 143 additions & 153 deletions src/_packages/core.index.ts
View file
Original file line number Diff line number Diff line change
@@ -1,49 +1,47 @@
// ⚠️ WARNING: This code has been generated so that any manual changes will be overwritten
// `@promptbook/core`

import { BOOK_LANGUAGE_VERSION, PROMPTBOOK_ENGINE_VERSION } from '../version';
import { collectionToJson } from '../collection/collectionToJson';
import { createCollectionFromJson } from '../collection/constructors/createCollectionFromJson';
import { createCollectionFromPromise } from '../collection/constructors/createCollectionFromPromise';
import { createCollectionFromUrl } from '../collection/constructors/createCollectionFromUrl';
import { createSubcollection } from '../collection/constructors/createSubcollection';
import { NAME } from '../config';
import { ADMIN_EMAIL } from '../config';
import { ADMIN_GITHUB_NAME } from '../config';
import { CLAIM } from '../config';
import { LOGO_LIGHT_SRC } from '../config';
import { LOGO_DARK_SRC } from '../config';
import { DEFAULT_TITLE } from '../config';
import { MAX_FILENAME_LENGTH } from '../config';
import { DEFAULT_INTERMEDIATE_FILES_STRATEGY } from '../config';
import { DEFAULT_MAX_PARALLEL_COUNT } from '../config';
import { DEFAULT_MAX_EXECUTION_ATTEMPTS } from '../config';
import { DEFAULT_MAX_KNOWLEDGE_SOURCES_SCRAPING_DEPTH } from '../config';
import { DEFAULT_MAX_KNOWLEDGE_SOURCES_SCRAPING_TOTAL } from '../config';
import { DEFAULT_BOOKS_DIRNAME } from '../config';
import { DEFAULT_EXECUTIONS_CACHE_DIRNAME } from '../config';
import { DEFAULT_SCRAPE_CACHE_DIRNAME } from '../config';
import { DEFAULT_PIPELINE_COLLECTION_BASE_FILENAME } from '../config';
import { RESERVED_PARAMETER_NAMES } from '../config';
import { DEFAULT_REMOTE_URL } from '../config';
import { DEFAULT_REMOTE_URL_PATH } from '../config';
import { DEFAULT_CSV_SETTINGS } from '../config';
import { DEFAULT_IS_VERBOSE } from '../config';
import { SET_IS_VERBOSE } from '../config';
import { DEFAULT_IS_AUTO_INSTALLED } from '../config';
import {
CLAIM,
DEFAULT_BOOKS_DIRNAME,
DEFAULT_CSV_SETTINGS,
DEFAULT_EXECUTIONS_CACHE_DIRNAME,
DEFAULT_INTERMEDIATE_FILES_STRATEGY,
DEFAULT_IS_AUTO_INSTALLED,
DEFAULT_IS_VERBOSE,
DEFAULT_MAX_EXECUTION_ATTEMPTS,
DEFAULT_MAX_KNOWLEDGE_SOURCES_SCRAPING_DEPTH,
DEFAULT_MAX_KNOWLEDGE_SOURCES_SCRAPING_TOTAL,
DEFAULT_MAX_PARALLEL_COUNT,
DEFAULT_PIPELINE_COLLECTION_BASE_FILENAME,
DEFAULT_REMOTE_URL,
DEFAULT_REMOTE_URL_PATH,
DEFAULT_SCRAPE_CACHE_DIRNAME,
DEFAULT_TITLE,
LOGO_DARK_SRC,
LOGO_LIGHT_SRC,
MAX_FILENAME_LENGTH,
NAME,
RESERVED_PARAMETER_NAMES,
} from '../config';
import { pipelineJsonToString } from '../conversion/pipelineJsonToString';
import { pipelineStringToJson } from '../conversion/pipelineStringToJson';
import { pipelineStringToJsonSync } from '../conversion/pipelineStringToJsonSync';
import { prettifyPipelineString } from '../conversion/prettify/prettifyPipelineString';
import { extractParameterNamesFromTask } from '../conversion/utils/extractParameterNamesFromTask';

Check failure on line 36 in src/_packages/core.index.ts

GitHub Actions / 🧪 Lint 

'extractParameterNamesFromTask' is defined but never used
import { removePipelineCommand } from '../conversion/utils/removePipelineCommand';

Check failure on line 37 in src/_packages/core.index.ts

GitHub Actions / 🧪 Lint 

'removePipelineCommand' is defined but never used
import { renamePipelineParameter } from '../conversion/utils/renamePipelineParameter';

Check failure on line 38 in src/_packages/core.index.ts

GitHub Actions / 🧪 Lint 

'renamePipelineParameter' is defined but never used
import { stringifyPipelineJson } from '../conversion/utils/stringifyPipelineJson';
import { validatePipeline } from '../conversion/validation/validatePipeline';
import { CallbackInterfaceTools } from '../dialogs/callback/CallbackInterfaceTools';
import type { CallbackInterfaceToolsOptions } from '../dialogs/callback/CallbackInterfaceToolsOptions';
import { BoilerplateError } from '../errors/0-BoilerplateError';

Check failure on line 43 in src/_packages/core.index.ts

GitHub Actions / 🧪 Lint 

'BoilerplateError' is defined but never used
import { PROMPTBOOK_ERRORS } from '../errors/0-index';

Check failure on line 44 in src/_packages/core.index.ts

GitHub Actions / 🧪 Lint 

'PROMPTBOOK_ERRORS' is defined but never used
import { AbstractFormatError } from '../errors/AbstractFormatError';
import { CollectionError } from '../errors/CollectionError';
import { EnvironmentMismatchError } from '../errors/EnvironmentMismatchError';
@@ -66,8 +64,7 @@
import { ExecutionReportStringOptionsDefaults } from '../execution/execution-report/ExecutionReportStringOptions';
import { addUsage } from '../execution/utils/addUsage';
import { isPassingExpectations } from '../execution/utils/checkExpectations';
import { ZERO_USAGE } from '../execution/utils/usage-constants';
import { UNCERTAIN_USAGE } from '../execution/utils/usage-constants';
import { UNCERTAIN_USAGE, ZERO_USAGE } from '../execution/utils/usage-constants';
import { usageToHuman } from '../execution/utils/usageToHuman';
import { usageToWorktime } from '../execution/utils/usageToWorktime';
import { CsvFormatDefinition } from '../formats/csv/CsvFormatDefinition';
@@ -90,15 +87,17 @@
import { limitTotalUsage } from '../llm-providers/_common/utils/count-total-usage/limitTotalUsage';
import { _AnthropicClaudeMetadataRegistration } from '../llm-providers/anthropic-claude/register-configuration';
import { _AzureOpenAiMetadataRegistration } from '../llm-providers/azure-openai/register-configuration';
import { _GoogleMetadataRegistration } from '../llm-providers/google/register-configuration';

Check failure on line 90 in src/_packages/core.index.ts

GitHub Actions / 🧪 Lint 

'_GoogleMetadataRegistration' is defined but never used
import { joinLlmExecutionTools } from '../llm-providers/multiple/joinLlmExecutionTools';
import { MultipleLlmExecutionTools } from '../llm-providers/multiple/MultipleLlmExecutionTools';
import { _OpenAiMetadataRegistration } from '../llm-providers/openai/register-configuration';
import { _OpenAiAssistantMetadataRegistration } from '../llm-providers/openai/register-configuration';
import {
_OpenAiAssistantMetadataRegistration,
_OpenAiMetadataRegistration,
} from '../llm-providers/openai/register-configuration';
import { preparePersona } from '../personas/preparePersona';
import { GENERIC_PIPELINE_INTERFACE } from '../pipeline/PipelineInterface/constants';
import { getPipelineInterface } from '../pipeline/PipelineInterface/getPipelineInterface';
import { isPipelineImplementingInterface } from '../pipeline/PipelineInterface/isPipelineImplementingInterface';
import { isPipelineImplementingInterface } from '../pipeline/PipelineInterface/isPipelineCompatible';
import { isPipelineInterfacesEqual } from '../pipeline/PipelineInterface/isPipelineInterfacesEqual';
import { EXPECTATION_UNITS } from '../pipeline/PipelineJson/Expectations';
import { isPipelinePrepared } from '../prepare/isPipelinePrepared';
@@ -118,132 +117,123 @@
import { MemoryStorage } from '../storage/memory/MemoryStorage';
import { PrefixStorage } from '../storage/utils/PrefixStorage';
import { MODEL_VARIANTS } from '../types/ModelVariant';
import { NonTaskSectionTypes } from '../types/SectionType';
import { SectionTypes } from '../types/SectionType';
import { NonTaskSectionTypes, SectionTypes } from '../types/SectionType';
import { TaskTypes } from '../types/TaskType';

import { BOOK_LANGUAGE_VERSION, PROMPTBOOK_ENGINE_VERSION } from '../version';

// Note: Exporting version from each package
export { BOOK_LANGUAGE_VERSION, PROMPTBOOK_ENGINE_VERSION };


// Note: Entities of the `@promptbook/core`
export { collectionToJson };
export { createCollectionFromJson };
export { createCollectionFromPromise };
export { createCollectionFromUrl };
export { createSubcollection };
export { NAME };
export { ADMIN_EMAIL };
export { ADMIN_GITHUB_NAME };
export { CLAIM };
export { LOGO_LIGHT_SRC };
export { LOGO_DARK_SRC };
export { DEFAULT_TITLE };
export { MAX_FILENAME_LENGTH };
export { DEFAULT_INTERMEDIATE_FILES_STRATEGY };
export { DEFAULT_MAX_PARALLEL_COUNT };
export { DEFAULT_MAX_EXECUTION_ATTEMPTS };
export { DEFAULT_MAX_KNOWLEDGE_SOURCES_SCRAPING_DEPTH };
export { DEFAULT_MAX_KNOWLEDGE_SOURCES_SCRAPING_TOTAL };
export { DEFAULT_BOOKS_DIRNAME };
export { DEFAULT_EXECUTIONS_CACHE_DIRNAME };
export { DEFAULT_SCRAPE_CACHE_DIRNAME };
export { DEFAULT_PIPELINE_COLLECTION_BASE_FILENAME };
export { RESERVED_PARAMETER_NAMES };
export { DEFAULT_REMOTE_URL };
export { DEFAULT_REMOTE_URL_PATH };
export { DEFAULT_CSV_SETTINGS };
export { DEFAULT_IS_VERBOSE };
export { SET_IS_VERBOSE };
export { DEFAULT_IS_AUTO_INSTALLED };
export { pipelineJsonToString };
export { pipelineStringToJson };
export { pipelineStringToJsonSync };
export { prettifyPipelineString };
export { extractParameterNamesFromTask };
export { removePipelineCommand };
export { renamePipelineParameter };
export { stringifyPipelineJson };
export { validatePipeline };
export { CallbackInterfaceTools };
export type { CallbackInterfaceToolsOptions };
export { BoilerplateError };
export { PROMPTBOOK_ERRORS };
export { AbstractFormatError };
export { CollectionError };
export { EnvironmentMismatchError };
export { ExpectError };
export { KnowledgeScrapeError };
export { LimitReachedError };
export { MissingToolsError };
export { NotFoundError };
export { NotYetImplementedError };
export { ParseError };
export { PipelineExecutionError };
export { PipelineLogicError };
export { PipelineUrlError };
export { UnexpectedError };
export { assertsExecutionSuccessful };
export { createPipelineExecutor };
export { embeddingVectorToString };
export { executionReportJsonToString };
export type { ExecutionReportStringOptions };
export { ExecutionReportStringOptionsDefaults };
export { addUsage };
export { isPassingExpectations };
export { ZERO_USAGE };
export { UNCERTAIN_USAGE };
export { usageToHuman };
export { usageToWorktime };
export { CsvFormatDefinition };
export { CsvFormatError };
export { MANDATORY_CSV_SETTINGS };
export { TextFormatDefinition };
export { BoilerplateFormfactorDefinition };
export { ChatbotFormfactorDefinition };
export { GeneratorFormfactorDefinition };
export { GenericFormfactorDefinition };
export { FORMFACTOR_DEFINITIONS };
export { MatcherFormfactorDefinition };
export { SheetsFormfactorDefinition };
export { TranslatorFormfactorDefinition };
export { $llmToolsMetadataRegister };
export { $llmToolsRegister };
export { createLlmToolsFromConfiguration };
export { cacheLlmTools };
export { countTotalUsage };
export { limitTotalUsage };
export { _AnthropicClaudeMetadataRegistration };
export { _AzureOpenAiMetadataRegistration };
export { _GoogleMetadataRegistration };
export { joinLlmExecutionTools };
export { MultipleLlmExecutionTools };
export { _OpenAiMetadataRegistration };
export { _OpenAiAssistantMetadataRegistration };
export { preparePersona };
export { GENERIC_PIPELINE_INTERFACE };
export { getPipelineInterface };
export { isPipelineImplementingInterface };
export { isPipelineInterfacesEqual };
export { EXPECTATION_UNITS };
export { isPipelinePrepared };
export { preparePipeline };
export { prepareTasks };
export { unpreparePipeline };
export { prepareKnowledgePieces };
export { $scrapersMetadataRegister };
export { $scrapersRegister };
export { makeKnowledgeSourceHandler };
export { _LegacyDocumentScraperMetadataRegistration };
export { _DocumentScraperMetadataRegistration };
export { _MarkdownScraperMetadataRegistration };
export { _PdfScraperMetadataRegistration };
export { _WebsiteScraperMetadataRegistration };
export { BlackholeStorage };
export { MemoryStorage };
export { PrefixStorage };
export { MODEL_VARIANTS };
export { NonTaskSectionTypes };
export { SectionTypes };
export { TaskTypes };
export {
$llmToolsMetadataRegister,
$llmToolsRegister,
$scrapersMetadataRegister,
$scrapersRegister,
_AnthropicClaudeMetadataRegistration,
_AzureOpenAiMetadataRegistration,
_DocumentScraperMetadataRegistration,
_LegacyDocumentScraperMetadataRegistration,
_MarkdownScraperMetadataRegistration,
_OpenAiAssistantMetadataRegistration,
_OpenAiMetadataRegistration,
_PdfScraperMetadataRegistration,
_WebsiteScraperMetadataRegistration,
AbstractFormatError,
addUsage,
assertsExecutionSuccessful,
BlackholeStorage,
BoilerplateFormfactorDefinition,
cacheLlmTools,
CallbackInterfaceTools,
ChatbotFormfactorDefinition,
CLAIM,
CollectionError,
collectionToJson,
countTotalUsage,
createCollectionFromJson,
createCollectionFromPromise,
createCollectionFromUrl,
createLlmToolsFromConfiguration,
createPipelineExecutor,
createSubcollection,
CsvFormatDefinition,
CsvFormatError,
DEFAULT_BOOKS_DIRNAME,
DEFAULT_CSV_SETTINGS,
DEFAULT_EXECUTIONS_CACHE_DIRNAME,
DEFAULT_INTERMEDIATE_FILES_STRATEGY,
DEFAULT_IS_AUTO_INSTALLED,
DEFAULT_IS_VERBOSE,
DEFAULT_MAX_EXECUTION_ATTEMPTS,
DEFAULT_MAX_KNOWLEDGE_SOURCES_SCRAPING_DEPTH,
DEFAULT_MAX_KNOWLEDGE_SOURCES_SCRAPING_TOTAL,
DEFAULT_MAX_PARALLEL_COUNT,
DEFAULT_PIPELINE_COLLECTION_BASE_FILENAME,
DEFAULT_REMOTE_URL,
DEFAULT_REMOTE_URL_PATH,
DEFAULT_SCRAPE_CACHE_DIRNAME,
DEFAULT_TITLE,
embeddingVectorToString,
EnvironmentMismatchError,
ERRORS,
executionReportJsonToString,
ExecutionReportStringOptionsDefaults,
EXPECTATION_UNITS,
ExpectError,
FORMFACTOR_DEFINITIONS,
GeneratorFormfactorDefinition,
GENERIC_PIPELINE_INTERFACE,
GenericFormfactorDefinition,
getPipelineInterface,
isPassingExpectations,
isPipelineImplementingInterface,
isPipelineInterfacesEqual,
isPipelinePrepared,
joinLlmExecutionTools,
KnowledgeScrapeError,
LimitReachedError,
limitTotalUsage,
LOGO_DARK_SRC,
LOGO_LIGHT_SRC,
makeKnowledgeSourceHandler,
MANDATORY_CSV_SETTINGS,
MatcherFormfactorDefinition,
MAX_FILENAME_LENGTH,
MemoryStorage,
MissingToolsError,
MODEL_VARIANTS,
MultipleLlmExecutionTools,
NAME,
NonTaskSectionTypes,
NotFoundError,
NotYetImplementedError,
ParseError,
PipelineExecutionError,
pipelineJsonToString,
PipelineLogicError,
pipelineStringToJson,
pipelineStringToJsonSync,
PipelineUrlError,
PrefixStorage,
prepareKnowledgePieces,
preparePersona,
preparePipeline,
prepareTasks,
prettifyPipelineString,
RESERVED_PARAMETER_NAMES,
SectionTypes,
SheetsFormfactorDefinition,
stringifyPipelineJson,
TaskTypes,
TextFormatDefinition,
TranslatorFormfactorDefinition,
UNCERTAIN_USAGE,
UnexpectedError,
unpreparePipeline,
usageToHuman,
usageToWorktime,
validatePipeline,
ZERO_USAGE,
};
export type { CallbackInterfaceToolsOptions, ExecutionReportStringOptions };
Loading