Space usage v4 jtl_reporter

[fturibak]

Hi! :-)

I have maybe 9 long runs

that are using almost 30G of space!

It's like PG is storing all the data, not only aggregated ones...

sudo du -sh data/

30G data/

[ludeknovy]

Hi @fturibak,
see: #33

[ludeknovy]

But it could be implemented as delete samples after processing flag for scenario that would delete all the samples after full processing.

[fturibak]

Thanks for info!

I was wondering if you need samples once you've aggregated data. Maybe you can delete all unnecessary data once the aggregation completes..

Anyhow I'll keep this in mind

[ludeknovy]

Yes, that makes sense. In the past, I used them a couple of times in case of a bug. It was possible to fix the already existing data.

[fturibak]

Just from practical side.

If I'm to run and publish results as part of CI/CD process every single day that's approximately 20-30GB of data per day = ~200GB per week = ~1TB per month

It's pretty expensive

[ludeknovy]

DELETE FROM jtl.samples samples WHERE samples.item_id = $1;

[fturibak]

Thanks!

[fturibak]

Hm, I've deleted records from items table.

jtl_report=# SELECT pg_size_pretty( pg_total_relation_size('jtl.items'));

pg_size_pretty

----------------
48 kB

(1 row)

But the size of database is still 28GB!

jtl_report=# SELECT pg_size_pretty( pg_database_size('jtl_report'));

pg_size_pretty

---

28 GB
(1 row)
`

[fturibak]

After VACUUM FULL situation is better

jtl_report=# vacuum full;

VACUUM

jtl_report=# SELECT pg_size_pretty( pg_database_size('jtl_report'));

pg_size_pretty

---

3444 MB

(1 row)

[ludeknovy]

Yes, VACUUM FULL jtl.samples; should make it.