feat: Implement git: provider

[tkesgar]

Resolves #84

This PR allow giget to download using git::

$ giget git:[email protected]:unjs/template

Initially we added a new git field in TemplateProvider. Based on the review feedback, we adjusted the tar in TemplateProvider to accept a function that returns Readable/ReadableStream for a tarfile:

export interface TemplateInfo {
  name: string
  tar: string | ((options: { auth?: string }) => TarOutput | Promise<TarOutput>);
}

When giget receives a function instead of a string, giget will pipe the stream to the internal tar path (reuses the code from download()). This way, we can move out most of the git handling to provider and reuse the existing tar handling (cache, extract subdirectory).

Git handling logic is mostly in the git provider now, with the logic to parse input is in parseGitCloneURI utility. It works similarly with package managers like npm/pnpm by calling git command in shell, then pack it back into a tar and pass it as stream.

We accept various kinds of URLs similar to what we can use in package.json:

• [email protected]:unjs/template.git
• [email protected]:unjs/template
• github.com:unjs/template
• github:unjs/template
• gh:unjs/template
• unjs/template (implies github.com)
• gitlab:unjs/template (=> [email protected]:unjs/template)

For custom git remote, use GIGET_GIT_* variables:

GIGET_GIT_USERNAME: nicole
GIGET_GIT_PASSWORD: denny123
GIGET_GIT_HOST: randomplay.com

myorg/my-project => nicole:[email protected]:myorg/my-project

Specific ref is supported via hash: github:unjs/template#main.

I cannot find any existing convention for specific subdirectory, so I implemented it as : after ref: github:unjs/template#main:src. Omitting the branch also works: github:unjs/template#:src.

Local path is supported: giget git:/path/to/local/repo (absolute), giget git:./local/repo (relative). However, this will allow users to clone any local repositories, so I put it behind GIGET_GIT_ALLOW_LOCAL environment variable. I don't think there are a lot of use cases for this (probably useful for testing), but I am planning to explore this for personal purpose (cloning from a local gitea instance).

HTTPS URL is supported: giget git:https://github.com/unjs/template.git (notice the git: prefix; otherwise giget will use its own URL handing). This feature relies on Git protocol itself. This is mainly added for testing purposes, but there may be people who want to use HTTPS URL.

A minor chore change is to exclude ./test/.tmp from Vitest, since Vitest faithfully tries to run every tests from git repos we cloned 😅

I think the implementation is mostly complete for now, but since we call some shell commands we probably want to test it a bit on Windows.

[pi0]

love the new impl!

[pi0]

Any reason closing @tkesgar?

[tkesgar]

Oh sorry, I was a bit emotional due to personal stuff a few days ago.

I will work on the PR again once I have some time. The only thing remaining is to fix the tests and sync with the latest main branch (there are some changes on the tar usage IIRC).

[tkesgar]

Looks like the test job fails because of public key issues when cloning via git.

From running the CI job with act in local:

[ci/ci]   ❗  ::error file=/home/tkesgar/git/tkesgar/giget/src/providers.ts,title=test/getgit.test.ts > downloadTemplate > clone unjs/template using git provider,line=229,column=17::Error: Failed to clone git repository from [email protected]:unjs/template. Make sure the repository exists and the provided version is correct.%0A ❯ Object.tar src/providers.ts:229:17%0A ❯ downloadTemplate src/giget.ts:118:27%0A ❯ test/getgit.test.ts:54:21%0A%0ACaused by: Caused by: Error: Command failed: git clone [email protected]:unjs/template /tmp/giget-uV6uEH --depth=1%0ACloning into '/tmp/giget-uV6uEH'...%[email protected]: Permission denied (publickey).%0D%0Afatal: Could not read from remote repository.%0A%0APlease make sure you have the correct access rights%0Aand the repository exists.%0A%0A⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯%0ASerialized Error: { code: 128, killed: false, signal: null, cmd: 'git clone [email protected]:unjs/template /tmp/giget-uV6uEH --depth=1' }%0A

I think it should be possible to fix this by adding the github public key, but I want to try an alternative approach to use https URL instead.

---

Update: I tried adding known_hosts but it does not seem to work. Therefore, I added some small change so we can use Git over HTTPS, and the tests are now using Git over HTTPS. For some reason it is much faster, I am not sure if GitHub has different behavior when cloning over HTTPS.

[tkesgar]

Hi @pi0, I think the current implementation should be okay from me. Let me know if there are any issues/changes need to be made.

I also created a small change in #220 to remove the duplicate error logs in console.

[tkesgar]

Looks like Github Actions only retrieve a specific commit, so we cannot test clone with specific version reference in CI.

I added .skipIf(process.env.CI) so it still works on local test run but skipped in the actions.