Consider reusing local cached sources

[Karrq]

Cargo does cache the sources locally to compile, couldn't we use the same sources for generating documentation?

[snowmead]

@claude look into this

[github-actions]

Claude finished @snowmead's task —— View job

---

Analysis: Reusing Cargo's Local Cached Sources

Tasks:

• Read project instructions (CLAUDE.md)
• Understand current source handling in rust-docs-mcp
• Research Cargo's source caching mechanism
• Analyze feasibility and benefits
• Provide findings and recommendations

How rust-docs-mcp Currently Works

1. Downloads sources independently from crates.io, GitHub, or local paths
2. Stores in custom location: ~/.rust-docs-mcp/cache/crates/{name}/{version}/source/
3. Runs cargo rustdoc on the downloaded source to generate JSON documentation
4. Manages its own cache with metadata, search indexes, and generated docs

How Cargo's Cache Works

Cargo maintains downloaded sources in:

• Registry cache: ~/.cargo/registry/cache/ - Contains .crate files (compressed archives)
• Registry source: ~/.cargo/registry/src/ - Contains extracted source code
• Git cache: ~/.cargo/git/ - Contains cloned git repositories

Feasibility Analysis

✅ Technically Feasible:

• Cargo's extracted sources in ~/.cargo/registry/src/ could be reused
• The directory structure is predictable: {registry-hash}/{crate-name}-{version}/
• Git sources are also available in ~/.cargo/git/checkouts/

🎯 Benefits:

1. Reduced disk usage - Avoid duplicate downloads and storage
2. Faster initial access - Skip download if already in Cargo cache
3. Bandwidth savings - Reuse existing downloads
4. Better integration - Leverage user's existing Cargo workflow

⚠️ Challenges:

1. Registry hash calculation - Need to determine the correct registry subdirectory
2. Cache invalidation - Cargo may clean its cache independently
3. Permission issues - May need read access to user's Cargo directories
4. Git checkouts complexity - Git sources use commit hashes, not versions
5. Incomplete cache - Not all crates may be present in Cargo's cache

Recommendations

1. Implement as an optional fallback strategy:

   // Pseudocode for the approach
   async fn get_source(name: &str, version: &str) -> Result<PathBuf> {
       // First check our own cache
       if let Ok(path) = check_our_cache(name, version) {
           return Ok(path);
       }
       
       // Then check Cargo's cache
       if let Ok(path) = check_cargo_cache(name, version) {
           // Copy to our cache for consistency
           copy_to_our_cache(&path, name, version)?;
           return Ok(our_cache_path(name, version));
       }
       
       // Finally download from source
       download_and_cache(name, version).await
   }

2. Add configuration option:

   [cache]
   use_cargo_cache = true  # Default: false for backward compatibility

3. Implement robust error handling:

   • Gracefully fall back to downloading if Cargo cache is unavailable
   • Handle permission errors when accessing ~/.cargo/
   • Validate source integrity before using

4. Consider crate-specific benefits:

   • Most beneficial for large, commonly-used crates (serde, tokio, etc.)
   • Less beneficial for rarely-used or private crates

Next Steps

If you'd like to proceed with implementation:

1. Add a CargoCache module to check for existing sources
2. Integrate it as a fallback in the CrateDownloader
3. Add configuration options for users to enable/disable
4. Test with various crate sources (registry, git, local)

This would be a valuable optimization that aligns with Rust ecosystem best practices while maintaining backward compatibility.