Skip to content

Commit 6c11c46

Browse files
SergiAguiloSergiAguilo
committed
Merge branch 'main' of https://github.com/EGA-archive/beacon2-pi-api into omop
2 parents de9812d + 6bb8796 commit 6c11c46

33 files changed

+352020
-132
lines changed

‎.github/workflows/update_api_version.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ name: update api version
22

33
on:
44
push:
5-
branches: ['main', 'api-version']
5+
branches: ['*', '!main']
66

77
jobs:
88
updateapiversion:

‎README.md

+23-15
Original file line numberDiff line numberDiff line change
@@ -44,28 +44,36 @@ If you are using a build with all the services in the same cluster, you can use:
4444
docker compose up -d --build
4545
```
4646

47+
#### Up the containers (with services in independent servers)
48+
49+
If you wish to have each service (or some of them) in different servers, you will need to use the remote version of the docker compose file, and deploy the remote services you need by selecting them individually in the build. Example:
50+
51+
```bash
52+
docker-compose -f docker-compose.remote.yml up -d --build beaconprod
53+
```
54+
55+
After that, you will need to configure the IPs in the different conf files to make them connect. Remember to bind the IP in mongo to 0.0.0.0 in case you are making an independent deployment of the beacon and the mongodb.
56+
4757
#### Load the data
4858

49-
To load the database we execute the following commands:
59+
To load the database (mongo) just copy your files in the data folder. Then, locate yourself in the mongo folder:
5060

5161
```bash
52-
docker cp /path/to/analyses.json mongoprod:tmp/analyses.json
53-
docker cp /path/to/biosamples.json mongoprod:tmp/biosamples.json
54-
docker cp /path/to/cohorts.json mongoprod:tmp/cohorts.json
55-
docker cp /path/to/datasets.json mongoprod:tmp/datasets.json
56-
docker cp /path/to/genomicVariations.json mongoprod:tmp/genomicVariations.json
57-
docker cp /path/to/individuals.json mongoprod:tmp/individuals.json
58-
docker cp /path/to/runs.json mongoprod:tmp/runs.json
62+
cd beacon/connections/mongo
5963
```
6064

65+
And execute the next commands (only the ones you need):
66+
6167
```bash
62-
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /tmp/datasets.json --collection datasets
63-
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /tmp/analyses.json --collection analyses
64-
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /tmp/biosamples.json --collection biosamples
65-
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /tmp/cohorts.json --collection cohorts
66-
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /tmp/genomicVariations.json --collection genomicVariations
67-
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /tmp/individuals.json --collection individuals
68-
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /tmp/runs.json --collection runs
68+
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /data/datasets.json --collection datasets
69+
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /data/individuals.json --collection individuals
70+
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /data/cohorts.json --collection cohorts
71+
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /data/analyses.json --collection analyses
72+
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /data/biosamples.json --collection biosamples
73+
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /data/genomicVariations.json --collection genomicVariations
74+
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /data/runs.json --collection runs
75+
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /data/targets.json --collection targets
76+
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /data/caseLevelData.json --collection caseLevelData
6977
```
7078

7179
This loads the JSON files inside of the `data` folder into the MongoDB database container. Each time you import data you will have to create indexes for the queries to run smoothly. Please, check the next point about how to Create the indexes.

‎beacon/conf/api_version.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
api_version: v2.0-1c0b896
1+
api_version: v2.0-eeabc63

‎beacon/conf/conf.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@
2323
alternative_url = 'https://beacon.ega-archive.org/api'
2424
create_datetime = '2021-11-29T12:00:00.000000'
2525
update_datetime = ''
26-
default_beacon_granularity = "record" # boolean, count or record
26+
max_beacon_granularity = "record" # boolean, count or record
2727
security_levels = ['PUBLIC', 'REGISTERED', 'CONTROLLED']
2828
documentation_url = 'https://b2ri-documentation-demo.ega-archive.org/'
2929
alphanumeric_terms = ['libraryStrategy', 'molecularAttributes.geneIds', 'diseases.ageOfOnset.iso8601duration', 'molecularAttributes.aminoacidChanges','phenotypicFeatures.onset.iso8601duration']

‎beacon/connections/mongo/Makefile

+9-18
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,10 @@
11
build:
2-
docker cp data/analyses.json mongoprod:tmp/analyses.json
3-
docker cp data/biosamples.json mongoprod:tmp/biosamples.json
4-
docker cp data/cohorts.json mongoprod:tmp/cohorts.json
5-
docker cp data/datasets.json mongoprod:tmp/datasets.json
6-
docker cp data/genomicVariations.json mongoprod:tmp/genomicVariations.json
7-
docker cp data/individuals.json mongoprod:tmp/individuals.json
8-
docker cp data/runs.json mongoprod:tmp/runs.json
9-
docker cp data/targets.json mongoprod:tmp/targets.json
10-
docker cp data/caseLevelData.json mongoprod:tmp/caseLevelData.json
11-
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /tmp/datasets.json --collection datasets
12-
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /tmp/individuals.json --collection individuals
13-
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /tmp/cohorts.json --collection cohorts
14-
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /tmp/analyses.json --collection analyses
15-
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /tmp/biosamples.json --collection biosamples
16-
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /tmp/genomicVariations.json --collection genomicVariations
17-
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /tmp/runs.json --collection runs
18-
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /tmp/targets.json --collection targets
19-
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /tmp/caseLevelData.json --collection caseLevelData
2+
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /data/datasets.json --collection datasets
3+
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /data/individuals.json --collection individuals
4+
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /data/cohorts.json --collection cohorts
5+
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /data/analyses.json --collection analyses
6+
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /data/biosamples.json --collection biosamples
7+
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /data/genomicVariations.json --collection genomicVariations
8+
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /data/runs.json --collection runs
9+
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /data/targets.json --collection targets
10+
docker exec mongoprod mongoimport --jsonArray --uri "mongodb://root:[email protected]:27017/beacon?authSource=admin" --file /data/caseLevelData.json --collection caseLevelData

‎beacon/connections/mongo/analyses.py

+17-9
Original file line numberDiff line numberDiff line change
@@ -67,17 +67,25 @@ def get_variants_of_analysis(self, entry_id: Optional[str], qparams: RequestPara
6767
if bioid == analysis_ids["biosampleId"]:
6868
break
6969
position+=1
70+
if position == len(bioids):
71+
schema = DefaultSchemas.GENOMICVARIATIONS
72+
return schema, 0, -1, None, dataset
7073
position=str(position)
71-
position1="^"+position+","
72-
position2=","+position+","
73-
position3=","+position+"$"
74-
query_cl={ "$or": [
75-
{"biosampleIds": {"$regex": position1}},
76-
{"biosampleIds": {"$regex": position2}},
77-
{"biosampleIds": {"$regex": position3}}
78-
]}
74+
filters=qparams.query.filters
75+
if filters != []:
76+
for filter in filters:
77+
if filter['id']=='GENO:GENO_0000458':
78+
query_cl={"$or": [{ position: "10", "datasetId": dataset}, { position: "01", "datasetId": dataset}]}
79+
qparams.query.filters.remove(filter)
80+
elif filter['id']=='GENO:GENO_0000136':
81+
query_cl={"$or": [{ position: "11", "datasetId": dataset}]}
82+
qparams.query.filters.remove(filter)
83+
else:
84+
query_cl={"$or": [{ position: "10", "datasetId": dataset},{ position: "11", "datasetId": dataset}, { position: "01", "datasetId": dataset}]}
85+
else:
86+
query_cl={"$or": [{ position: "10", "datasetId": dataset},{ position: "11", "datasetId": dataset}, { position: "01", "datasetId": dataset}]}
7987
string_of_ids = client.beacon.caseLevelData \
80-
.find(query_cl, {"id": 1, "_id": 0})
88+
.find(query_cl, {"id": 1, "_id": 0}).limit(qparams.query.pagination.limit).skip(qparams.query.pagination.skip)
8189
HGVSIds=list(string_of_ids)
8290
query={}
8391
queryHGVS={}

‎beacon/connections/mongo/biosamples.py

+17-9
Original file line numberDiff line numberDiff line change
@@ -62,17 +62,25 @@ def get_variants_of_biosample(self, entry_id: Optional[str], qparams: RequestPar
6262
if bioid == entry_id:
6363
break
6464
position+=1
65+
if position == len(bioids):
66+
schema = DefaultSchemas.GENOMICVARIATIONS
67+
return schema, 0, -1, None, dataset
6568
position=str(position)
66-
position1="^"+position+","
67-
position2=","+position+","
68-
position3=","+position+"$"
69-
query_cl={ "$or": [
70-
{"biosampleIds": {"$regex": position1}},
71-
{"biosampleIds": {"$regex": position2}},
72-
{"biosampleIds": {"$regex": position3}}
73-
]}
69+
filters=qparams.query.filters
70+
if filters != []:
71+
for filter in filters:
72+
if filter['id']=='GENO:GENO_0000458':
73+
query_cl={"$or": [{ position: "10", "datasetId": dataset}, { position: "01", "datasetId": dataset}]}
74+
qparams.query.filters.remove(filter)
75+
elif filter['id']=='GENO:GENO_0000136':
76+
query_cl={"$or": [{ position: "11", "datasetId": dataset}]}
77+
qparams.query.filters.remove(filter)
78+
else:
79+
query_cl={"$or": [{ position: "10", "datasetId": dataset},{ position: "11", "datasetId": dataset}, { position: "01", "datasetId": dataset}]}
80+
else:
81+
query_cl={"$or": [{ position: "10", "datasetId": dataset},{ position: "11", "datasetId": dataset}, { position: "01", "datasetId": dataset}]}
7482
string_of_ids = client.beacon.caseLevelData \
75-
.find(query_cl, {"id": 1, "_id": 0})
83+
.find(query_cl, {"id": 1, "_id": 0}).limit(qparams.query.pagination.limit).skip(qparams.query.pagination.skip)
7684
HGVSIds=list(string_of_ids)
7785
query={}
7886
queryHGVS={}

‎beacon/connections/mongo/data/caseLevelData.json

+1-1
Large diffs are not rendered by default.

‎beacon/connections/mongo/data/genomicVariations.json

+1-1
Large diffs are not rendered by default.

‎beacon/connections/mongo/data/targets.json

+1-1
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)