Building AI infrastructure on Elixir/BEAM. Research focus on LLM reliability, distributed systems, and functional programming.
105 public repos | 323 stars | @North-Shore-AI
The BEAM virtual machine provides unique advantages for AI infrastructure that no other runtime offers: true preemptive concurrency with lightweight processes, battle-tested fault tolerance through supervision trees, and hot code reloading for zero-downtime model updates. These properties make Elixir ideal for building reliable, observable ML systems that can gracefully handle partial failures across distributed deployments.
This ecosystem represents a comprehensive research platform for world-class ML reliability on BEAM. From hexagonal RAG architectures to dialectical reasoning frameworks, from asset-first data orchestration to multi-agent systems, each project contributes to a unified vision: production-grade AI infrastructure built on functional programming principles.
| Tier | Layer | Components |
|---|---|---|
| 1 | Interface | crucible_ui, cns_ui, ingot - Phoenix LiveView dashboards |
| 2 | Gateway | nsai_gateway - Auth, rate limiting, routing, circuit breaker |
| 3 | Processing | forge, anvil, labeling_ir - Data pipelines & HITL workflows |
| 4 | Core Framework | crucible_framework, crucible_ir, crucible_bench - Pipeline orchestration |
| 5 | MLOps | crucible_train, crucible_model_registry, crucible_deployment |
| 6 | Reliability | crucible_ensemble, crucible_hedging, LlmGuard |
| 7 | Utilities | tiktoken_ex, embed_ex, nx_penalties, chz_ex |
Hexagonal architecture for building flexible RAG systems with pluggable adapters.
| Project | Description |
|---|---|
| portfolio_core | Hexagonal architecture core - 16 Port specifications, ETS-backed Registry, manifest engine |
| portfolio_index | Production adapters (Pgvector, Neo4j, Gemini), Broadway pipelines, RAG strategies (Hybrid, Self-RAG, GraphRAG) |
| portfolio_manager | CLI workflows, multi-provider LLM routing, agent framework, pipeline orchestration |
| portfolio_coder | Code intelligence platform - repository indexing, semantic code search, dependency graphs |
Architecture:
portfolio_coder (Code Intelligence)
|
portfolio_manager (CLI, Router, Agent, Pipeline)
|
portfolio_index (Adapters: Pgvector, Neo4j, Gemini, Claude)
|
portfolio_core (Ports, Registry, Manifest, Telemetry)
| Project | Stars | Description |
|---|---|---|
| flowstone | 12 | Asset-first data orchestration - lineage tracking, partitions, scatter/gather, approval gates, Oban persistence |
| flowstone_ai | 1 | AI extensions for FlowStone pipelines |
| synapse | 31 | Headless multi-agent orchestration - signal bus, workflow engine, Postgres persistence, LLM gateway |
FlowStone treats assets (named data artifacts) as the primary abstraction for building reliable, auditable data pipelines. Features include partition-aware execution, scatter fan-out, signal gates for external webhooks, and comprehensive lineage queries.
Synapse provides domain-agnostic multi-agent orchestration with a declarative signal registry, configurable agent runtime, and workflow engine with step-level retry and compensation.
Open research platform for LLM reliability.
| Project | Description |
|---|---|
| crucible_framework | ML experimentation orchestration, pipeline runner, composable stages |
| crucible_ir | Intermediate representation for ML pipelines (experiment, stage_def, reliability configs) |
| crucible_bench | Statistical testing - t-tests, ANOVA, effect sizes, power analysis, normality tests |
| Project | Description |
|---|---|
| crucible_train | Unified ML training (supervised, RL, DPO, distillation), multi-backend logging |
| crucible_model_registry | Model versioning, artifact storage (S3/HF/local), lineage graphs |
| crucible_deployment | Model deployment (vLLM/Ollama/TGI/K8s), canary/blue-green rollouts |
| crucible_feedback | Production feedback, drift detection, active learning, retraining triggers |
| Project | Description |
|---|---|
| crucible_kitchen | Backend-agnostic ML training orchestration engine (the facility) |
| tinkex_cookbook | Training recipes and configurations for Tinker platform (the instructions) |
| Project | Description |
|---|---|
| crucible_ensemble | Multi-model voting (majority, weighted, ranked-choice, semantic similarity) |
| crucible_hedging | Request hedging strategies (fixed, adaptive, percentile, workload-aware) |
| crucible_trace | Causal reasoning chain logging, Mermaid diagram export |
| LlmGuard | AI firewall - prompt injection, jailbreak, data leakage detection |
Chiral Narrative Synthesis - framework for dialectical reasoning with structured argumentation.
| Project | Description |
|---|---|
| cns | Core framework - Proposer, Antagonist, Synthesizer agents; SNO graphs; topology metrics |
| cns_crucible | CNS + Crucible integration, SciFact experiments, topology demos |
| cns_ui | Phoenix LiveView UI - dialectical visualization, SNO explorer |
| tinkerer | Research workspace for CNS experiments |
Dialectical Flow:
Proposer (thesis) -> Antagonist (antithesis) -> Synthesizer (synthesis)
| | |
Extract SNOs Flag contradictions Resolve with evidence
(claims+evidence) (Beta-1 gaps, chirality) (critic-guided)
Critic Ensemble: Grounding, Logic, Novelty, Bias, Causal
Key Concepts:
- SNO (Structured Narrative Objects) - Claims with supporting evidence
- Beta-1 Gaps - Missing supporting evidence
- Chirality - Logical handedness of arguments
Domain-agnostic sample factory for human-in-the-loop workflows.
| Project | Description |
|---|---|
| forge | Raw data processing factory - sample generation, transformation (shaping raw materials) |
| anvil | Labeling queue and governance toolkit for HITL workflows (careful refinement) |
| ingot | Phoenix LiveView labeling UI module (finished product) |
| labeling_ir | Shared IR structs for labeling stack |
Metaphor: Raw materials (Forge) are shaped on the Anvil through careful human refinement, producing polished Ingots ready for consumption.
| Project | Stars | Description |
|---|---|---|
| gemini_ex | 22 | Elixir Interface / Adapter for Google Gemini LLM, for bot... |
| claude_agent_sdk | 13 | An Elixir SDK for Claude Code - provides programmatic acc... |
| codex_sdk | 4 | OpenAI Codex SDK written in Elixir |
| jules_ex | 2 | Elixir client SDK for the Jules API - orchestrate AI codi... |
| altar_ai | 1 | Protocol-based AI adapter foundation for Elixir - unified... |
| mcp_client | 0 | Full-featured Elixir client for the Model Context Protoco... |
| Project | Stars | Description |
|---|---|---|
| json_remedy | 23 | A practical, multi-layered JSON repair library for Elixir... |
| snakepit | 10 | High-performance, generalized process pooler and session ... |
| snakebridge | 2 | Compile-time Elixir code generator for Python library bin... |
| tinkex | 2 | Elixir SDK for the Tinker ML platform—LoRA training, samp... |
| gepa_ex | 1 | Elixir implementation of GEPA: LLM-driven evolutionary op... |
| nsai_gateway | 1 | Unified API gateway for the NSAI ecosystem—authentication... |
| nsai_registry | 1 | Service discovery and registry for the NSAI ecosystem—dis... |
| pilot | 1 | Interactive CLI and REPL for the NSAI ecosystem—unified i... |
| rag_ex | 1 | Elixir RAG library with multi-LLM routing (Gemini, Claude... |
| skill_ex | 1 | Claude Skill Aggregator |
| tiktoken_ex | 1 | Pure Elixir TikToken-style byte-level BPE tokenizer (Kimi... |
| tinkerer | 1 | Chiral Narrative Synthesis workspace for Thinker/Tinker L... |
| work | 1 | NSAI.Work - Unified job scheduler for North-Shore-AI plat... |
| hf_hub_ex | 0 | Elixir client for HuggingFace Hub—dataset/model metadata,... |
| hf_peft_ex | 0 | Elixir port of HuggingFace's PEFT (Parameter-Efficient Fi... |
| portfolio_core | 0 | Hexagonal architecture core for Elixir RAG systems. Port ... |
| portfolio_index | 0 | Production adapters and pipelines for PortfolioCore. Vect... |
| portfolio_manager | 0 | AI-native personal project intelligence system - manage, ... |
| tinkex_cookbook | 0 | Elixir port of tinker-cookbook: training and evaluation r... |
| Project | Stars | Description |
|---|---|---|
| synapse | 31 | Headless, declarative multi-agent orchestration framework... |
| ds_ex | 15 | DSPEx - Declarative Self-improving Elixir |
| flowstone | 12 | Asset-first data orchestration for Elixir/BEAM. Dagster-i... |
| DSPex | 10 | Declarative Self Improving Elixir - DSPy Orchestration in... |
| pipeline_ex | 7 | Claude Code + Gemini AI collaboration orchestration tools |
| ALTAR | 6 | The Agent {{AUTO_GENERATED_CONTENT}} Tool Arbitration Protocol |
| mabeam | 5 | Multi-agent systems framework for the BEAM platform - bui... |
| flowstone_ai | 1 | FlowStone integration for altar_ai - AI-powered data pipe... |
| synapse_ai | 1 | Synapse integration for altar_ai - SDK-backed LLM provide... |
| Project | Stars | Description |
|---|---|---|
| exdantic | 10 | A powerful, flexible schema definition and validation lib... |
| sinter | 10 | Unified schema definition, validation, and JSON generatio... |
| perimeter | 7 | Advanced typing and type validation mechanism for Elixir ... |
| Project | Stars | Description |
|---|---|---|
| ex_dbg | 10 | State-of-the-Art Introspection and Debugging System for E... |
| elixir_scope | 5 | Revolutionary AST-based debugging and code intelligence p... |
| ElixirScope | 4 | AI-Powered Execution Cinema Debugger for Elixir/BEAM |
| elixir_dashboard | 1 | A Phoenix LiveView performance monitoring dashboard for t... |
| elixir_tracer | 1 | Local-first observability for Elixir with New Relic API p... |
| portfolio_coder | 0 | Code Intelligence Platform: Repository analysis, semantic... |
| pristine | 0 | Manifest-driven hexagonal core for generating Elixir SDKs... |
| Project | Stars | Description |
|---|---|---|
| superlearner | 8 | OTP Supervisor Educational Platform |
| apex | 4 | Core Apex framework for OTP supervision and monitoring |
| apex_ui | 4 | Web UI for Apex OTP supervision and monitoring tools |
| arsenal | 4 | Metaprogramming framework for automatic REST API generati... |
| arsenal_plug | 3 | Phoenix/Plug adapter for Apex Arsenal framework |
| Project | Stars | Description |
|---|---|---|
| playwriter | 6 | Elixir WSL-to-Windows browser integration |
| cluster_test | 4 | Distributed Erlang/Elixir test cluster management via Mix... |
| sandbox | 4 | Isolated OTP application management system for Elixir/Erlang |
| supertester | 4 | A battle-hardened testing toolkit for building robust and... |
| Project | Stars | Description |
|---|---|---|
| foundation | 11 | Elixir infrastructure and Observability Library |
| AITrace | 1 | The unified observability layer for the AI Control Plane |
| Citadel | 1 | The command and control layer for the AI-powered enterprise |
| telemetry_reporter | 0 | Pachka-powered telemetry reporter for Elixir that batches... |
| Project | Stars | Description |
|---|---|---|
| duckdb_ex | 1 | DuckDB driver client in Elixir |
| embed_ex | 1 | Vector embeddings service for Elixir—multi-provider suppo... |
| weaviate_ex | 1 | Modern Elixir client for Weaviate vector database with he... |
| ex_topology | 0 | Pure Elixir library for graph topology, TDA, and computat... |
| nx_penalties | 0 | Composable regularization penalties for Elixir Nx. L1/L2/... |
| Project | Stars | Description |
|---|---|---|
| GUARDRAIL | 16 | GUARDRAIL - MCP Security - Gateway for Unified Access, Re... |
| ASKA | 7 | Secure Computing in the AI age |
| pqc-hqc | 5 | Post-quantum cryptographic implementation of HQC (Hamming... |
| Shield | 0 | SHIELD: Secure Hierarchical Inter-agent Layer for Distrib... |
| Project | Stars | Description |
|---|---|---|
| ChronoLedger | 2 | Hardware-Secured Temporal Blockchain |
| cns | 1 | Chiral Narrative Synthesis - Dialectical reasoning framew... |
| EADS | 0 | Evolutionary Autonomous Development System |
| cns_ui | 0 | Phoenix LiveView interface for CNS dialectical reasoning ... |
| Project | Stars | Description |
|---|---|---|
| youtube_audio_dl | 2 | Download high-quality audio from YouTube as MP3 files usi... |
| tools | 1 | Utility library and helper functions for Elixir developme... |
| chz_ex | 0 | Elixir port of OpenAI's chz library - a powerful configur... |
| multipart_ex | 0 | Client-agnostic multipart/form-data builder for Elixir wi... |
| Project | Stars | Description |
|---|---|---|
| nshkrdotcom | 0 | Personal GitHub profile README with Elixir/AI projects an... |
GitHub Organizations:
- @North-Shore-AI - Research organization
- @nshkrdotcom - Personal projects
Websites:
- nsai.online - Main site
- nsai.store - Store
- nsai.space - Space
- nshkr.com - Personal
Updated 2026-01-02