`Development`: Rework user data export server side #1755

sachmii · 2026-01-20T15:04:14Z

‼️ DO NOT DEPLOY TO TEST SERVER - CONTAINS DATABASE CHANGES ‼️
‼️ MERGE #1742 FIRST ‼️

Checklist

General

I tested all changes and their related features with all corresponding user types.
Language: I followed the guidelines for inclusive, diversity-sensitive, and appreciative language.
I chose a title conforming to the naming conventions for pull requests.

Server

Important: I implemented the changes with a very good performance and prevented too many (unnecessary) and too complex database calls.
I strictly followed the principle of data economy for all database calls.
I strictly followed the server coding and design guidelines.
I documented the Java code using JavaDoc style.

Motivation and Context

The user data export needed some rework, see #1749.
The client side solution will be implemented in a separate PR. The client side changes are only a temporary solution to be able to test the feature.

Description

Asynchronous export workflow: Replaced synchronous ZIP streaming with an async request-based system where users request exports, the system processes them in background (via scheduled job), and sends download links via email when ready
Rate limiting and status tracking: Introduced a weekly cooldown period between export requests, new DataExportRequest entity to track export lifecycle (REQUESTED → IN_CREATION → EMAIL_SENT → DOWNLOADED), and status endpoint for users to check their export progress
Token-based download system: Implemented secure, time-limited download tokens with configurable expiration (7 days default) and public download endpoint that doesn't require authentication, with automatic status updates when downloads occur
Enhanced data collection: Expanded export data to include interview-related information (interview processes, slots, interviewees), uploaded images from the image repository, and improved document handling with proper MIME type extensions
New REST API endpoints: Created three new endpoints - POST /api/users/data-export to request exports, GET /api/users/data-export/status to check status, and GET /api/users/data-export/download/{token} for downloading prepared exports
Infrastructure improvements: Added database table for export requests, scheduled cron job for processing pending exports, email notification system (DATA_EXPORT_READY type), and configurable storage paths with proper file system security validation

Steps for Testing

Prerequisites:

Log in to TumApply
Export User data (wait few seconds)
Check exported data corresponds with uploaded documents, applications etc.
(Run integration tests)

Review Progress

Code Review

Code Review 1

Manual Tests

Test 1

Test Coverage

Warning: Both client and server tests failed. Coverage could not be fully measured. Please check the workflow logs.

Last updated: 2026-01-28 16:57:54 UTC

github-actions · 2026-01-20T15:31:24Z

@sachmii Test coverage could not be fully measured because some tests failed. Please check the workflow logs for details.

github-actions · 2026-01-20T15:55:13Z

🤖 No OpenAPI or client changes needed.

github-actions · 2026-01-20T16:21:29Z

@sachmii Test coverage could not be fully measured because some tests failed. Please check the workflow logs for details.

…://github.com/ls1intum/tum-apply into feat/1750-rework-user-data-export-server-side

github-actions · 2026-01-20T18:57:38Z

@sachmii Test coverage could not be fully measured because some tests failed. Please check the workflow logs for details.

github-actions · 2026-01-20T18:58:39Z

🤖 No OpenAPI or client changes needed.

…t/1750-rework-user-data-export-server-side

github-actions · 2026-01-21T13:25:25Z

@sachmii Test coverage could not be fully measured because some tests failed. Please check the workflow logs for details.

github-actions · 2026-01-21T13:27:45Z

🤖 No OpenAPI or client changes needed.

github-actions · 2026-01-21T14:08:45Z

🤖 OpenAPI spec and client code auto-updated and committed.

github-actions · 2026-01-21T14:11:54Z

@sachmii Test coverage could not be fully measured because some tests failed. Please check the workflow logs for details.

github-actions · 2026-01-21T14:15:21Z

@sachmii Test coverage could not be fully measured because some tests failed. Please check the workflow logs for details.

github-actions · 2026-01-21T14:16:23Z

📊 Client Test Coverage Too Low

🔍 View coverage locally:

npm run test:ci
open build/test-results/lcov-report/index.html

🌐 View coverage from GitHub:
Download the "coverage-report-client" artifact from this workflow run.

github-actions · 2026-01-21T14:17:29Z

🤖 No OpenAPI or client changes needed.

github-actions · 2026-01-21T14:17:51Z

📊 Server Test Coverage Too Low

🔍 View coverage locally:

./gradlew test jacocoTestReport
open build/reports/jacoco/test/html/index.html

🌐 View coverage from GitHub:
Download the "coverage-report-server" artifact from this workflow run.

github-actions · 2026-01-24T13:44:51Z

📊 Client Test Coverage Too Low

🔍 View coverage locally:

npm run test:ci
open build/test-results/lcov-report/index.html

🌐 View coverage from GitHub:
Download the "coverage-report-client" artifact from this workflow run.

github-actions · 2026-01-24T13:46:11Z

🤖 No OpenAPI or client changes needed.

github-actions · 2026-01-24T13:48:41Z

@sachmii Test coverage has been automatically updated in the PR description.

github-actions · 2026-01-24T13:49:40Z

@sachmii Test coverage could not be fully measured because some tests failed. Please check the workflow logs for details.

Bofan-Zhu

Thanks for reworking the export 😊.
Generally, what happens if exports are queued but the server reboots or crashes?
Also, does the user have to login to access the export link?

src/main/resources/templates/de/DATA_EXPORT_READY.html

src/main/java/de/tum/cit/aet/core/service/UserDataExportService.java

src/main/java/de/tum/cit/aet/core/dto/DataExportStatusDTO.java

src/main/resources/config/liquibase/changelog/00000000000025_create_data_export_requests.xml

…t/1750-rework-user-data-export-server-side

github-actions · 2026-01-28T16:06:58Z

📊 Client Test Coverage Too Low

🔍 View coverage locally:

npm run test:ci
open build/test-results/lcov-report/index.html

🌐 View coverage from GitHub:
Download the "coverage-report-client" artifact from this workflow run.

github-actions · 2026-01-28T16:08:30Z

🤖 OpenAPI spec and client code auto-updated and committed.

github-actions · 2026-01-28T16:10:05Z

📊 Client Test Coverage Too Low

🔍 View coverage locally:

npm run test:ci
open build/test-results/lcov-report/index.html

🌐 View coverage from GitHub:
Download the "coverage-report-client" artifact from this workflow run.

github-actions · 2026-01-28T16:11:10Z

🤖 No OpenAPI or client changes needed.

github-actions · 2026-01-28T16:11:39Z

📊 Server Test Coverage Too Low

🔍 View coverage locally:

./gradlew test jacocoTestReport
open build/reports/jacoco/test/html/index.html

🌐 View coverage from GitHub:
Download the "coverage-report-server" artifact from this workflow run.

github-actions · 2026-01-28T16:15:33Z

@sachmii Test coverage could not be fully measured because some tests failed. Please check the workflow logs for details.

github-actions · 2026-01-28T16:20:05Z

@sachmii Test coverage could not be fully measured because some tests failed. Please check the workflow logs for details.

…://github.com/ls1intum/tum-apply into feat/1750-rework-user-data-export-server-side

github-actions · 2026-01-28T16:40:06Z

📊 Client Test Coverage Too Low

🔍 View coverage locally:

npm run test:ci
open build/test-results/lcov-report/index.html

🌐 View coverage from GitHub:
Download the "coverage-report-client" artifact from this workflow run.

github-actions · 2026-01-28T16:40:52Z

🤖 No OpenAPI or client changes needed.

github-actions · 2026-01-28T16:41:42Z

@sachmii Test coverage could not be fully measured because some tests failed. Please check the workflow logs for details.

github-actions · 2026-01-28T16:44:42Z

@sachmii Test coverage could not be fully measured because some tests failed. Please check the workflow logs for details.

github-actions · 2026-01-28T16:48:01Z

📊 Client Test Coverage Too Low

🔍 View coverage locally:

npm run test:ci
open build/test-results/lcov-report/index.html

🌐 View coverage from GitHub:
Download the "coverage-report-client" artifact from this workflow run.

github-actions · 2026-01-28T16:48:52Z

🤖 No OpenAPI or client changes needed.

github-actions · 2026-01-28T16:57:55Z

@sachmii Test coverage could not be fully measured because some tests failed. Please check the workflow logs for details.

github-actions · 2026-01-28T16:57:55Z

📊 Client Test Coverage Too Low

🔍 View coverage locally:

npm run test:ci
open build/test-results/lcov-report/index.html

🌐 View coverage from GitHub:
Download the "coverage-report-client" artifact from this workflow run.

github-actions · 2026-01-28T16:58:24Z

🤖 No OpenAPI or client changes needed.

sachmii added 4 commits January 20, 2026 11:31

refine user data export

b82e9e6

classify user

99855d6

add doc

f784f1e

fix detail page

fa4afe7

sachmii self-assigned this Jan 20, 2026

sachmii linked an issue Jan 20, 2026 that may be closed by this pull request

Rework user data export (server side) #1750

Open

github-actions bot added server Pull requests that update Java code. (Added Automatically!) client Pull requests that update TypeScript code. (Added Automatically!) tests labels Jan 20, 2026

Merge branch 'main' into feat/1750-rework-user-data-export-server-side

3ba3a94

sachmii added 2 commits January 20, 2026 19:42

back to normal

437d4a0

Merge branch 'feat/1750-rework-user-data-export-server-side' of https…

06d3459

…://github.com/ls1intum/tum-apply into feat/1750-rework-user-data-export-server-side

Merge branch 'main' of https://github.com/ls1intum/tum-apply into fea…

77694cf

…t/1750-rework-user-data-export-server-side

first steps

2dfdc49

github-actions bot added the database label Jan 21, 2026

chore: update OpenAPI spec and generated client

05d72ac

step 2

e0df5be

Merge branch 'main' into feat/1750-rework-user-data-export-server-side

9c015d3

Bofan-Zhu requested changes Jan 24, 2026

View reviewed changes

sachmii added 2 commits January 28, 2026 16:58

refactor to prevent blob

f3efdb5

Merge branch 'main' of https://github.com/ls1intum/tum-apply into fea…

56993f0

…t/1750-rework-user-data-export-server-side

sachmii requested a review from Bofan-Zhu January 28, 2026 16:04

chore: update OpenAPI spec and generated client

15fa7f8

sachmii added 2 commits January 28, 2026 17:38

add missing doc

86bad25

Merge branch 'feat/1750-rework-user-data-export-server-side' of https…

e530064

…://github.com/ls1intum/tum-apply into feat/1750-rework-user-data-export-server-side

remove unused imports

ff61863

remove unused parameter

5dfbd9f

Development: Rework user data export server side #1755

Are you sure you want to change the base?

Development: Rework user data export server side #1755

Uh oh!

Conversation

sachmii commented Jan 20, 2026 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

General

Server

Motivation and Context

Description

Steps for Testing

Review Progress

Code Review

Manual Tests

Test Coverage

Uh oh!

github-actions bot commented Jan 20, 2026

Uh oh!

github-actions bot commented Jan 20, 2026

Uh oh!

github-actions bot commented Jan 20, 2026

Uh oh!

github-actions bot commented Jan 20, 2026

Uh oh!

github-actions bot commented Jan 20, 2026

Uh oh!

github-actions bot commented Jan 21, 2026

Uh oh!

github-actions bot commented Jan 21, 2026

Uh oh!

github-actions bot commented Jan 21, 2026

Uh oh!

github-actions bot commented Jan 21, 2026

Uh oh!

github-actions bot commented Jan 21, 2026

Uh oh!

github-actions bot commented Jan 21, 2026

Uh oh!

github-actions bot commented Jan 21, 2026

Uh oh!

github-actions bot commented Jan 21, 2026

Uh oh!

github-actions bot commented Jan 24, 2026

Uh oh!

github-actions bot commented Jan 24, 2026

Uh oh!

github-actions bot commented Jan 24, 2026

Uh oh!

github-actions bot commented Jan 24, 2026

Uh oh!

Bofan-Zhu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 28, 2026

Uh oh!

github-actions bot commented Jan 28, 2026

Uh oh!

github-actions bot commented Jan 28, 2026

Uh oh!

github-actions bot commented Jan 28, 2026

Uh oh!

github-actions bot commented Jan 28, 2026

Uh oh!

github-actions bot commented Jan 28, 2026

Uh oh!

github-actions bot commented Jan 28, 2026

Uh oh!

github-actions bot commented Jan 28, 2026

Uh oh!

github-actions bot commented Jan 28, 2026

Uh oh!

github-actions bot commented Jan 28, 2026

`Development`: Rework user data export server side #1755

`Development`: Rework user data export server side #1755

sachmii commented Jan 20, 2026 •

edited by github-actions bot

Loading