Skip to content

Commit d09851a

Browse files
dfrnochdfrnoch
committed
feat: ✨ production
1 parent edda41f commit d09851a

File tree

12 files changed

+513
-2638
lines changed

12 files changed

+513
-2638
lines changed

‎.DS_Store

0 Bytes
Binary file not shown.

‎.github/.DS_Store

6 KB
Binary file not shown.

‎.github/workflows/publish.yml

+136
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
# CI that:
2+
#
3+
# * checks for a Git Tag that looks like a release
4+
# * creates a Github Release™ and fills in its text
5+
# * builds artifacts with cargo-dist (executable-zips, installers)
6+
# * uploads those artifacts to the Github Release™
7+
#
8+
# Note that the Github Release™ will be created before the artifacts,
9+
# so there will be a few minutes where the release has no artifacts
10+
# and then they will slowly trickle in, possibly failing. To make
11+
# this more pleasant we mark the release as a "draft" until all
12+
# artifacts have been successfully uploaded. This allows you to
13+
# choose what to do with partial successes and avoids spamming
14+
# anyone with notifications before the release is actually ready.
15+
name: Release
16+
17+
permissions:
18+
contents: write
19+
20+
# This task will run whenever you push a git tag that looks like a version
21+
# like "v1", "v1.2.0", "v0.1.0-prerelease01", "my-app-v1.0.0", etc.
22+
# The version will be roughly parsed as ({PACKAGE_NAME}-)?v{VERSION}, where
23+
# PACKAGE_NAME must be the name of a Cargo package in your workspace, and VERSION
24+
# must be a Cargo-style SemVer Version.
25+
#
26+
# If PACKAGE_NAME is specified, then we will create a Github Release™ for that
27+
# package (erroring out if it doesn't have the given version or isn't cargo-dist-able).
28+
#
29+
# If PACKAGE_NAME isn't specified, then we will create a Github Release™ for all
30+
# (cargo-dist-able) packages in the workspace with that version (this is mode is
31+
# intended for workspaces with only one dist-able package, or with all dist-able
32+
# packages versioned/released in lockstep).
33+
#
34+
# If you push multiple tags at once, separate instances of this workflow will
35+
# spin up, creating an independent Github Release™ for each one.
36+
#
37+
# If there's a prerelease-style suffix to the version then the Github Release™
38+
# will be marked as a prerelease.
39+
on:
40+
push:
41+
tags:
42+
- '*-?v[0-9]+*'
43+
44+
jobs:
45+
# Create the Github Release™ so the packages have something to be uploaded to
46+
create-release:
47+
runs-on: ubuntu-latest
48+
outputs:
49+
has-releases: ${{ steps.create-release.outputs.has-releases }}
50+
env:
51+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
52+
steps:
53+
- uses: actions/checkout@v3
54+
- name: Install cargo-dist
55+
run: curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.13.0/cargo-dist-installer.sh | sh
56+
- id: create-release
57+
run: |
58+
cargo dist plan --tag=${{ github.ref_name }} --output-format=json > dist-manifest.json
59+
echo "dist plan ran successfully"
60+
cat dist-manifest.json
61+
62+
# Create the Github Release™ based on what cargo-dist thinks it should be
63+
ANNOUNCEMENT_TITLE=$(jq --raw-output ".announcement_title" dist-manifest.json)
64+
IS_PRERELEASE=$(jq --raw-output ".announcement_is_prerelease" dist-manifest.json)
65+
jq --raw-output ".announcement_github_body" dist-manifest.json > new_dist_announcement.md
66+
gh release create ${{ github.ref_name }} --draft --prerelease="$IS_PRERELEASE" --title="$ANNOUNCEMENT_TITLE" --notes-file=new_dist_announcement.md
67+
echo "created announcement!"
68+
69+
# Upload the manifest to the Github Release™
70+
gh release upload ${{ github.ref_name }} dist-manifest.json
71+
echo "uploaded manifest!"
72+
73+
# Disable all the upload-artifacts tasks if we have no actual releases
74+
HAS_RELEASES=$(jq --raw-output ".releases != null" dist-manifest.json)
75+
echo "has-releases=$HAS_RELEASES" >> "$GITHUB_OUTPUT"
76+
77+
# Build and packages all the things
78+
upload-artifacts:
79+
# Let the initial task tell us to not run (currently very blunt)
80+
needs: create-release
81+
if: ${{ needs.create-release.outputs.has-releases == 'true' }}
82+
strategy:
83+
matrix:
84+
# For these target platforms
85+
include:
86+
- os: ubuntu-20.04
87+
dist-args: --artifacts=global
88+
install-dist: curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.13.0/cargo-dist-installer.sh | sh
89+
- os: macos-11
90+
dist-args: --artifacts=local --target=aarch64-apple-darwin --target=x86_64-apple-darwin
91+
install-dist: curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.13.0/cargo-dist-installer.sh | sh
92+
- os: ubuntu-20.04
93+
dist-args: --artifacts=local --target=x86_64-unknown-linux-gnu
94+
install-dist: curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.13.0/cargo-dist-installer.sh | sh
95+
- os: windows-2019
96+
dist-args: --artifacts=local --target=x86_64-pc-windows-msvc
97+
install-dist: irm https://github.com/axodotdev/cargo-dist/releases/download/v0.13.0/cargo-dist-installer.ps1 | iex
98+
99+
runs-on: ${{ matrix.os }}
100+
env:
101+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
102+
steps:
103+
- uses: actions/checkout@v3
104+
- name: Install cargo-dist
105+
run: ${{ matrix.install-dist }}
106+
- name: Run cargo-dist
107+
# This logic is a bit janky because it's trying to be a polyglot between
108+
# powershell and bash since this will run on windows, macos, and linux!
109+
# The two platforms don't agree on how to talk about env vars but they
110+
# do agree on 'cat' and '$()' so we use that to marshal values between commands.
111+
run: |
112+
# Actually do builds and make zips and whatnot
113+
cargo dist build --tag=${{ github.ref_name }} --output-format=json ${{ matrix.dist-args }} > dist-manifest.json
114+
echo "dist ran successfully"
115+
cat dist-manifest.json
116+
117+
# Parse out what we just built and upload it to the Github Release™
118+
jq --raw-output ".artifacts[]?.path | select( . != null )" dist-manifest.json > uploads.txt
119+
echo "uploading..."
120+
cat uploads.txt
121+
gh release upload ${{ github.ref_name }} $(cat uploads.txt)
122+
echo "uploaded!"
123+
124+
# Mark the Github Release™ as a non-draft now that everything has succeeded!
125+
publish-release:
126+
# Only run after all the other tasks, but it's ok if upload-artifacts was skipped
127+
needs: [create-release, upload-artifacts]
128+
if: ${{ always() && needs.create-release.result == 'success' && (needs.upload-artifacts.result == 'skipped' || needs.upload-artifacts.result == 'success') }}
129+
runs-on: ubuntu-latest
130+
env:
131+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
132+
steps:
133+
- uses: actions/checkout@v3
134+
- name: mark release as non-draft
135+
run: |
136+
gh release edit ${{ github.ref_name }} --draft=false

‎Cargo.lock

+53
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml

+1
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ version = "0.1.0"
44
edition = "2021"
55

66
[dependencies]
7+
clap = { version = "4.5.4", features = ["derive"] }
78
env_logger = "0.11.3"
89
glob = "0.3.1"
910
globset = "0.4"

‎LICENSE

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2024 David Frnoch
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md

+70
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
<div align="center">
2+
3+
<img height="170x" src="./assets/icon.png" />
4+
5+
<h1>Code Scraper for AI</h1>
6+
7+
<p>
8+
<strong>Extract code from a source directory for AI analysis</strong>
9+
</p>
10+
11+
</div>
12+
13+
## Why
14+
15+
- Recursively processes files in a specified source directory
16+
- Respects `.gitignore` file to exclude unwanted files and directories
17+
- Supports custom ignore files for additional exclusion patterns
18+
- Generates a single output file containing the extracted code
19+
- Optimized for code analysis by AI systems
20+
- Provides logging capabilities for better visibility and debugging
21+
22+
## Installation
23+
24+
### Using Cargo
25+
26+
```bash
27+
cargo install --git https://github.com/dfrnoch/code_scraper
28+
```
29+
30+
### Downloading from GitHub
31+
32+
Every new version will be automatically compiled and available for download in the [releases page](https://github.com/dfrnoch/code_scraper/releases).
33+
34+
## Usage
35+
36+
```bash
37+
code_scraper <source_folder> [custom_ignore_file] [--log]
38+
```
39+
40+
- `<source_folder>`: The path to the source directory containing the code files to be extracted.
41+
- `[custom_ignore_file]` (optional): The path to a custom ignore file specifying additional patterns to exclude from extraction. If not provided, only the `.gitignore` file in the source directory will be used.
42+
- `[--log]` (optional): Enable logging for detailed output and debugging information.
43+
44+
## Configuration
45+
46+
Code Scraper for AI provides various configuration options to customize its behavior:
47+
48+
- `source_folder`: Specify the path to the source directory containing the code files to be extracted.
49+
- `custom_ignore_file`: Provide a custom ignore file with additional patterns to exclude from extraction.
50+
- `log`: Enable logging for detailed output and debugging information.
51+
52+
## Output
53+
54+
Code Scraper for AI generates an output file named `output.txt` in the current directory. The file contains the extracted code from the processed files, with each file's content separated by a header indicating the file path.
55+
56+
## Create a Bug Report
57+
58+
If you encounter an error message or run into an issue, please [create an issue](https://github.com/dfrnoch/code_scraper/issues). Your effort is valued and will help improve the tool for other users.
59+
60+
## Submit a Feature Request
61+
62+
If you have an idea for a new feature or enhancement, please [submit a feature request](https://github.com/dfrnoch/code_scraper/issues).
63+
64+
## Contributing
65+
66+
Code Scraper for AI is an open-source project. Whether you are helping to fix bugs, proposing new features, or improving documentation, your contributions are highly appreciated. Please feel free to submit pull requests or open issues to contribute to the project.
67+
68+
## License
69+
70+
Code Scraper for AI is released under the [MIT License](LICENSE).

‎assets/icon.png

751 KB
Loading

0 commit comments

Comments
 (0)