Skip to content

Commit 6bc4b48

Browse files
FelixTJDietrichFelixTJDietrich
authored
Documentation: Improve evaluation data export docs (#176)
* add changes * update docs * update docs * revert changes
1 parent 3a8bac8 commit 6bc4b48

File tree

2 files changed

+107
-31
lines changed

2 files changed

+107
-31
lines changed

‎docs/images/load-anonymized-database-dump.png

721 KB
Loading

‎docs/setup/evaluation.rst

Lines changed: 107 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -1,54 +1,130 @@
1-
Evaluation Data
1+
Evaluation Data for Athena Playground
22
===========================================
33

4-
The Playground comes bundled with a basic set of example data to test Athena's functionalities. For more comprehensive evaluation, you can load your own data or use anonymized data from `Artemis <https://github.com/ls1intum/Artemis>`_, an open-source LMS.
4+
The Athena Playground is equipped with a set of example data for initial testing. To conduct a more thorough evaluation, users have the option to use their own datasets or request anonymized data from `Artemis <https://github.com/ls1intum/Artemis>`_, an open-source LMS.
55

6-
Example Data
7-
-------------------------------------------
8-
This data is provided within the `playground/data/example` directory and is automatically utilized when launching the Playground.
96

10-
Evaluation Data
11-
-------------------------------------------
12-
The `playground/data/evaluation` directory is designated for your custom data used for evaluation purposes. Initially, it's left empty for you to populate.
7+
Example Data
8+
------------
139

14-
Artemis Evaluation Data
15-
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
16-
If you're integrating with Artemis LMS and would like to evaluate their data, you can request an anonymized database dump from the Artemis team. This request requires a valid reason and a signed data protection agreement (NDA). For further details, please get in touch with the Artemis team.
10+
Located in ``playground/data/example``, this default dataset is automatically used when the Playground is initiated.
1711

18-
Once the database dump is acquired, follow these steps to export the data to the Playground:
1912

20-
1. **Load the Database Dump:**
13+
Evaluation Data
14+
---------------
2115

22-
.. code-block:: bash
16+
The directory ``playground/data/evaluation`` is reserved for your custom data. It is initially empty, ready to be filled with your evaluation datasets.
2317

24-
npm run export:artemis:1-load-anonymized-database-dump
2518

26-
This command loads the data into your local MySQL database. You can use the same database as Artemis.
19+
Exporting Evaluation Data from Artemis
20+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2721

28-
2. **Export the Data:**
22+
To evaluate using data from Artemis, you can request an anonymized database dump, contingent on a valid justification and a signed data protection agreement. Contact the Artemis team for details.
2923

30-
.. code-block:: bash
24+
Steps to Export Evaluation Data from Artemis:
25+
"""""""""""""""""""""""""""""""""""""""""""""
3126

32-
npm run export:artemis:2-export-evaluation-data
27+
1. **Setup a MySQL database:**
28+
Create a new MySQL database and user. You can use the same database instance as Artemis or a separate one. You can follow the instructions in the `Artemis documentation <https://docs.artemis.cit.tum.de/dev/setup/database.html#mysql-setup>`_ to set up a MySQL database.
3329

34-
This exports exercises listed under `playground/scripts/artemis/evaluation_data` to the `playground/data/evaluation` directory, where you can use it for evaluation purposes.
3530

36-
Artemis Programming Exercises
37-
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
38-
Artemis programming exercises are not included in the anonymized database dump. To access these exercises, you'll need to request them separately from the Artemis team. Once you have the programming exercises, an instructor from the course can export them using the following commands:
31+
2. **Load the Database Dump:**
32+
Use the command below to import the anonymized data into your local MySQL database. You will only need to do this once to populate the database. The script will ask you for the database ``host``, ``port``, ``user``, ``password``, and ``database``. Additionally, you will need to provide the path to the anonymized database dump, e.g. ``/home/user/artemis-database-dump.sql``.
3933

40-
1. **Download the Repositories:**
34+
.. code-block:: bash
4135
42-
.. code-block:: bash
36+
npm run export:artemis:1-load-anonymized-database-dump
4337
44-
npm run export:artemis:3-download-programming-repositories
38+
.. image:: ../images/load-anonymized-database-dump.png
39+
:width: 500px
40+
:alt: Example terminal screenshot of the command to load the anonymized database dump
41+
:align: center
4542

46-
This command exports the programming exercises' materials and submissions to the `playground/data/evaluation` directory. The instructor should then zip these and send them to you.
43+
3. **Export the Data:**
44+
This command exports the data specified in ``playground/scripts/artemis/evaluation_data/text_exercises.json`` to your local ``playground/data/evaluation`` directory.
4745

48-
2. **Link the Repositories:**
46+
.. code-block:: bash
4947
50-
.. code-block:: bash
48+
npm run export:artemis:2-export-evaluation-data
5149
52-
npm run export:artemis:4-link-programming-repositories
5350
54-
This command links the repositories to the `exercise-*.json` files and validates if there are any missing repositories.
51+
Artemis Programming Exercises
52+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
53+
54+
Programming exercises are not part of the anonymized database dump and must be requested separately from the Artemis team. You can find the selected exercise and their participation IDs for export in ``playground/scripts/artemis/evaluation_data/programming_exercises.json``.
55+
56+
Steps for Instructors to Export Programming Exercises:
57+
""""""""""""""""""""""""""""""""""""""""""""""""""""""
58+
59+
4. **Download Repositories:**
60+
Instructors can download materials and submissions from Artemis using the command below, then zip and transfer them to you. Keep in mind that this command will take a long time to run if there are many participations to download.
61+
62+
.. code-block:: bash
63+
64+
npm run export:artemis:3-download-programming-repositories
65+
66+
5. **Link the Repositories:**
67+
Put the downloaded repositories in the ``playground/data/evaluation`` directory and link them to the respective exercises using the following command. This command will also validate if there are any missing repositories. Without this step, the programming repositories will not be available in the Playground.
68+
69+
.. code-block:: bash
70+
71+
npm run export:artemis:4-link-programming-repositories
72+
73+
74+
Generating ``programming_exercises.json``
75+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
76+
77+
The SQL script provided below can be adapted to generate a ``programming_exercises.json`` file, located at ``playground/scripts/artemis/evaluation_data/programming_exercises.json``. Similar logic applies to create ``text_exercises.json``. The script collects data on selected exercises, aggregates participation data, and formats it into a JSON structure suitable for export scripts.
78+
79+
**Note:** The provided SQL script is an example and should be tailored to include the specific IDs of the programming exercises you wish to export. You might want to reduce the number of participations to export if you don't need all of them. ``anonymized_artemis`` should be replaced with the name of your database.
80+
81+
.. code-block:: sql
82+
83+
WITH temp_course_exercises AS (
84+
SELECT
85+
DISTINCT e.id,
86+
c.id AS course_id,
87+
0 as is_exam_exercise -- Course exercises
88+
FROM
89+
anonymized_artemis.exercise e
90+
JOIN anonymized_artemis.course c ON e.course_id = c.id
91+
),
92+
temp_exam_exercises AS (
93+
SELECT
94+
DISTINCT e.id,
95+
c.id AS course_id,
96+
1 as is_exam_exercise -- Exam exercises
97+
FROM
98+
anonymized_artemis.course c
99+
JOIN anonymized_artemis.exam ex ON ex.course_id = c.id
100+
JOIN anonymized_artemis.exercise_group eg ON eg.exam_id = ex.id
101+
JOIN anonymized_artemis.exercise e ON e.exercise_group_id = eg.id
102+
),
103+
temp_exercises AS (
104+
SELECT * FROM temp_course_exercises
105+
UNION
106+
SELECT * FROM temp_exam_exercises
107+
)
108+
SELECT JSON_OBJECT(
109+
c.title, JSON_OBJECT(
110+
'course_id', c.id,
111+
'semester', c.semester,
112+
'exercises', JSON_ARRAYAGG(
113+
JSON_OBJECT(
114+
'id', e.id,
115+
'title', e.title,
116+
'is_exam_exercise', te.is_exam_exercise
117+
)
118+
),
119+
'participations', JSON_ARRAYAGG(
120+
(SELECT JSON_ARRAYAGG(p.id)
121+
FROM anonymized_artemis.participation p -- Note: This contains also participations that are maybe unneccessary
122+
WHERE p.exercise_id = e.id)
123+
)
124+
)
125+
)
126+
FROM temp_exercises te
127+
JOIN anonymized_artemis.exercise e ON te.id = e.id
128+
JOIN anonymized_artemis.course c ON c.id = te.course_id
129+
WHERE e.id IN (2610, 3782, 2111, 2104, 3187, 3781, 6344, 6433, 3942, 3693, 4864, 4896, 3913, 3914, 3908, 3185, 3184) -- Programming exercises to export
130+
GROUP BY c.id, c.title, c.semester;

0 commit comments

Comments
 (0)