Skip to content

Commit fae6067

Browse files
authored
[DOCS] Solutions' cost information (#109)
* DQ-172 * Adding pricing calculation
1 parent c015173 commit fae6067

File tree

1 file changed

+82
-0
lines changed

1 file changed

+82
-0
lines changed

README.md

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,88 @@ The tool can be used as a standard Terraform module, with deployment examples pr
6464

6565
See the [functions](https://github.com/provectus/data-quality-gate/tree/main/functions) for further details.
6666

67+
## Pricing
68+
69+
This solution is completely free because it is open source. However, if you want to integrate it into a live/production environment, there will be associated costs due to its cloud-based nature. These costs can be divided into two parts: the required infrastructure (which you may already have in place, such as VPCs and subnets) and the AWS services necessary for data quality implementation.
70+
71+
*Note: All the information provided below has been calculated using the maximum score strategy.*
72+
#### Pricing for required infrastructure
73+
74+
| AWS Service | Approximate monthly cost| Description |
75+
| ------------- | ------------- | ------------- |
76+
| AWS S3 and DynamoDB endpoints | - | There is no extra charge for gateway-type endpoints. You only pay for the usage of S3 and DynamoDB itself. |
77+
| AWS Interface VPC endpoints(secrets manager, monitoring, sns) | 3 endpoints * (30 days * 24 hours * 0.01 rate) = 21.6 USD | Interface endpoints charged by hour. 1 hour = $0.01 |
78+
| AWS ECRs (allure, data_test, reports, notifications) | 7 versions * (865mb + 432mb + 380mb) => 11.3gb * 0.1 rate per gb month= 1.13 USD | allure image size = 865mb, data_test image size = 432mb, reports image size = 380mb, notifications image size = 160mb. For the purpose of our calculations, let's assume we are storing 7 versions of each image. |
79+
| AWS QuickSight | $7.3 aprx rate per user * 5 = 36.4 USD | Let's assume you have a team consisting of 5 individuals who are interested in the QuickSight data quality dashboard. They frequently check for changes, typically 2-3 times per day. |
80+
81+
<u>Monthly total is $59.13 US$ per month</u>
82+
___
83+
84+
#### Pricing for data quality specific infrastructure
85+
For most of the services used by Data Quality, AWS offers a free-tier supply. Additionally, the costs for these services are typically just a fraction of a cent. To provide further clarity, below you can find a basic cost formula and a few usage examples with cost estimations.
86+
87+
We are going to count:
88+
- number of AWS Lambda runs
89+
- number of AWS StepFunction transitions
90+
- web reports AWS EC2 instance running(720 hrs per month)
91+
92+
| Description | Formula |
93+
| ------------- | ------------- |
94+
| number of AWS Lambda runs for each | (number of data sources * number of changes * work_days_month) * lambda specific rate(depends on lambda duration and memory used) |
95+
| number of AWS StepFunction transitions | number of lambda runs * 2 |
96+
97+
##### Small
98+
99+
Let's say we have 1000 data sources and half of them changed every day. Number of runs formula for any lambda is **(1000 data sources * 0.5 changed * 30 days)**
100+
101+
| AWS Service | Number of runs | Price |
102+
| ------------ | -------------- | ------ |
103+
| AWS Lambda AllureReport | 15000 | $8.33 |
104+
| AWS Lambda DataTest | 15000 | $67.28 |
105+
| AWS Lambda Reports | 15000 | $2.08 |
106+
| AWS StepFunctions | 15000 | $0.65 |
107+
| AWS EC2 Reports S3 Gateway | 720 hrs | $7.25 |
108+
109+
<u>Monthly total: 85.59 US$</u>
110+
111+
___
112+
113+
##### Medium
114+
115+
Let's say we have 10000 data sources and 70% of them changed every day.
116+
Number of runs formula for any lambda is **(10000 data sources * 0.7 changes * 30 days)**
117+
118+
| AWS Service | Number of runs | Price |
119+
| ------------ | -------------- | ------ |
120+
| AWS Lambda AllureReport | 210k | $203.33 |
121+
| AWS Lambda DataTest | 210k | $1028.57 |
122+
| AWS Lambda Reports | 210k | $115.83 |
123+
| AWS StepFunctions | 210k | $10.40 |
124+
| AWS EC2 Reports S3 Gateway | 720 hrs | $7.25 |
125+
126+
<u>Monthly total: 1 365.38 US$</u>
127+
128+
___
129+
130+
##### Large
131+
132+
Let's say we have 30000 data sources and all of them changed every day.
133+
Number of runs formula for any lambda is **(30000 data sources * 1 changes * 30 days)**
134+
135+
| AWS Service | Number of runs | Price |
136+
| ------------ | -------------- | ------ |
137+
| AWS Lambda AllureReport | 900k | $893.34 |
138+
| AWS Lambda DataTest | 900k | $4430.06 |
139+
| AWS Lambda Reports | 900k | $518.33 |
140+
| AWS StepFunctions | 900k | $44.90 |
141+
| AWS EC2 Reports S3 Gateway | 720 hrs | $7.25 |
142+
143+
<u>Monthly total: 5 893.88 US$</u>
144+
___
145+
146+
**Price per changed data source: 0.006 US$**
147+
148+
67149
## License
68150

69151
Apache 2 Licensed. See [LICENSE](https://github.com/provectus/data-quality-gate/tree/main/LICENSE) for full details.

0 commit comments

Comments
 (0)