Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
5d37f97
pulling over initial parquet changes
koncheto-broad Mar 12, 2024
f0a3f15
finalizing merge
koncheto-broad Oct 16, 2024
41f3a26
cherry picking in parquet changes
koncheto-broad Mar 12, 2024
3caef40
finishing cherry picking
koncheto-broad Mar 19, 2024
6e4fd32
small update
koncheto-broad Mar 20, 2024
3958d6b
finally getting the parquet copying part sorted out
koncheto-broad Mar 20, 2024
20eb9cb
suffering a file name clash and overriding the same file. Changing t…
koncheto-broad Mar 20, 2024
d90990d
suffering a file name clash and overriding the same file. Changing t…
koncheto-broad Mar 20, 2024
da7254b
no more build issues
koncheto-broad Oct 8, 2025
2d48b4a
Cherry picking in and modifying parquet code for header and ref data
RoriCremer Jun 7, 2024
0969d88
updating header and ref code
koncheto-broad Oct 9, 2025
2022582
Updating vet schema to match current
koncheto-broad Oct 14, 2025
6606187
pushing wdl changes
koncheto-broad Oct 14, 2025
5997158
dockstore
koncheto-broad Oct 14, 2025
7d04210
turning on ref ranges creation
koncheto-broad Oct 15, 2025
fa37d5b
updating stupid issue with changing bash variable name
koncheto-broad Oct 15, 2025
5358f1b
making directory names consistent
koncheto-broad Oct 16, 2025
ca6d213
Implementing the doc, updating tests, and building the new docker
koncheto-broad Oct 28, 2025
09f3891
consolidating changes into standard import genomes workflow
koncheto-broad Oct 28, 2025
b2e6532
Prepping for test run--more work to be done integrating parquet with …
koncheto-broad Oct 28, 2025
d4a6d76
normalizing 'done' types
koncheto-broad Oct 29, 2025
a27a7d7
fixing incorrect docker images
koncheto-broad Oct 29, 2025
1fdf830
getting rid of billing project argument to see if the bucket listing …
koncheto-broad Oct 29, 2025
6ad2531
Getting rid of billing project input instead of commenting it out
koncheto-broad Oct 29, 2025
90d5331
Fixing issue with code creating multiple trailing slashes, thus resul…
koncheto-broad Oct 29, 2025
6f9aad5
Fixing weird ref_ranges -> ref name remapping and fixing WDL error th…
koncheto-broad Oct 29, 2025
4d5ef6b
Updating tests and variants docker
koncheto-broad Oct 29, 2025
17aed13
updating the parquet file loading to better handle interruptions
koncheto-broad Nov 17, 2025
7a26988
bug fix
koncheto-broad Nov 18, 2025
24949b7
updating docker
koncheto-broad Nov 18, 2025
1dd6e2c
making this more idempotent
koncheto-broad Nov 18, 2025
c883dd2
Adding lifecycle rules so the parquet files auto-delete after 14 days
koncheto-broad Nov 21, 2025
934b83a
Adding useful documentation
koncheto-broad Dec 3, 2025
0e1e978
moving docs around
koncheto-broad Dec 8, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .dockstore.yml
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,7 @@ workflows:
branches:
- master
- ah_var_store
- VS-1736
tags:
- /.*/
- name: GvsBulkIngestGenomes
Expand All @@ -220,6 +221,7 @@ workflows:
branches:
- master
- ah_var_store
- VS-1736
tags:
- /.*/
- name: GvsPrepareRangesCallset
Expand Down
7 changes: 7 additions & 0 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -389,6 +389,13 @@ dependencies {
// pgen jni
implementation('org.broadinstitute:pgenjni:1.0.1')

// parquet writing
implementation('org.apache.parquet:parquet-common:1.13.1')
implementation('org.apache.parquet:parquet-encoding:1.13.1')
implementation('org.apache.parquet:parquet-column:1.13.1')
implementation('org.apache.parquet:parquet-hadoop:1.13.1')
implementation 'org.apache.parquet:parquet-avro:1.13.1'

testUtilsImplementation sourceSets.main.output
testUtilsImplementation 'org.testng:testng:' + testNGVersion
testUtilsImplementation 'org.apache.hadoop:hadoop-minicluster:' + hadoopVersion
Expand Down
14 changes: 14 additions & 0 deletions build.log
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
Option -e requires an argument.
Usage: ./build_docker.sh: -e <GITHUB_TAG> [-psl]
where <GITHUB_TAG> is the github tag (or hash when -s is used) to use in building the docker image
(e.g. bash build_docker.sh -e 1.0.0.0-alpha1.2.1)
Optional arguments:
-s The GITHUB_TAG (-e parameter) is actually a github hash, not tag. git hashes cannot be pushed as latest, so -l is implied.
-l Do not also push the image to the 'latest' tag.
-u Do not run the unit tests.
-m Build the lite image (which does not contain the conda environment).
-d <STAGING_DIR> staging directory to grab code from repo and build the docker image. If unspecified, then use whatever is in current dir (do not go to the repo). NEVER SPECIFY YOUR WORKING DIR
-p (GATK4 developers only) push image to docker hub once complete. This will use the GITHUB_TAG in dockerhub as well.
Unless -l is specified, this will also push this image to the 'latest' tag.
-r (GATK4 developers only) Do not remove the unit test docker container. This is useful for debugging failing unit tests.
-t <PULL_REQUEST_NUMBER> (Travis CI only) The pull request number. This is only used during pull request builds on Travis CI.
Loading
Loading