feat: add support for complex types in dicts and lists #483

BryanttV · 2025-03-19T01:14:59Z

Description

This PR adds support for complex types in dictionaries and lists. e.g.

List[Dict[str, str]: List of Dictionaries
List[List[int]: List of Lists
List[EventData]: List of Data Attributes
Dict[str, List[str]]: Dict of Lists
Dict[str, Dict[str, bool]]: Dict of Dicts
Dict[str, EventData]: Dict of Data Attributes

Supporting information

Add support for complex types in dictionaries and lists #441

Testing instructions

To test with the event bus:

Install this plugin with the changes in this branch (complex-types-in-dicts-and-lists-samples). That branch modifies some data attributes for testing purposes.
Install an event bus backend like the Redis implementation in the LMS.

Set up the event bus with Tutor. For that, create a Tutor inline plugin with this configuration so the event is produced to Redis:

# openedx-events.py
from tutor import hooks

redis_config = [
    "EVENT_BUS_PRODUCER = 'edx_event_bus_redis.create_producer'",
    "EVENT_BUS_REDIS_CONNECTION_URL = 'redis://@redis:6379/'",
    "EVENT_BUS_TOPIC_PREFIX = 'dev'",
    "EVENT_BUS_CONSUMER = 'edx_event_bus_redis.RedisEventConsumer'",
]

hooks.Filters.ENV_PATCHES.add_item(("openedx-common-settings", "\n".join(redis_config)))

event_bus_config = """
EVENT_BUS_PRODUCER_CONFIG = {
    # To test dict[str, List[str]]
    'org.openedx.learning.course.notification.requested.v1': {
        'course-notification-requested': {'event_key_field': 'course_notification_data.course_key', 'enabled': True},
    },
    # To test List[dict[str, str]]
    'org.openedx.learning.ora.submission.created.v1': {
        'ora-submission-created': {'event_key_field': 'submission.uuid', 'enabled': True},
    },
    # To test List[DiscussionTopicContext]
    'org.openedx.learning.discussions.configuration.changed.v1': {
        'discussion-configuration-changed': {'event_key_field': 'configuration.course_key', 'enabled': True},
    },
}
"""

hooks.Filters.ENV_PATCHES.add_item(("openedx-common-settings", event_bus_config))

Now, to emit each event:

`COURSE_NOTIFICATION_REQUESTED`: `dict[str, List[str]]`

Create a Waffle Flag for your course in {lms_domain}/admin/waffle_utils/waffleflagcourseoverridemodel/
- Waffle Flag: notifications.enable_notifications
- Course id: [Your Course ID]
- Enabled: ✅
You can check this by answering an ORA assignment (Check the next event for more details)

You should see a log in the LMS like this:

2025-03-20 03:49:14,306 INFO 120 [edx_event_bus_redis.internal.producer] [user 4] [ip 172.18.0.1] producer.py:86 - Message delivered to Redis event bus: topic=dev-course-notification-requested, message_id=4a650d68-053e-11f0-85cd-0242ac120002, signal=<OpenEdxPublicSignal: org.openedx.learning.course.notification.requested.v1>, redis_msg_id=b'1742442554306-0'

`ORA_SUBMISSION_CREATED`: `List[dict[str, str]]`

In Studio, create a unit with an ORA assignment.
In the LMS, answer to the assignment.

You should see a log in the LMS like this:

2025-03-20 03:09:26,788 INFO 29 [edx_event_bus_redis.internal.producer] [user 4] [ip 172.18.0.1] producer.py:86 - Message delivered to Redis event bus: topic=dev-ora-submission-created, message_id=bb4ee7f2-0538-11f0-a878-0242ac120002, signal=<OpenEdxPublicSignal: org.openedx.learning.ora.submission.created.v1>, redis_msg_id=b'1742440166787-0'

`COURSE_DISCUSSIONS_CHANGED`: `List[DiscussionTopicContext]`

In Studio, go to Pages & Resources > Discussions ⚙️ > Settings and update the configuration (maybe create a new discussion topic).

You should see a log in the CMS like this:

2025-03-20 03:14:47,761 INFO 30 [edx_event_bus_redis.internal.producer] [user 4] [ip 172.18.0.1] producer.py:86 - Message delivered to Redis event bus: topic=dev-discussion-configuration-changed, message_id=7aa80be2-0539-11f0-a543-0242ac12000a, signal=<OpenEdxPublicSignal: org.openedx.learning.discussions.configuration.changed.v1>, redis_msg_id=b'1742440487760-0'

Other information

The COURSE_NOTIFICATION_REQUESTED is emitted in different ways in edx-platform. This event works with the event bus in some cases, but not all. For example, If we create a new discussion thread (LMS > Discussions > Add a post), that emits an event, but sending to the event bus fails.

The reason for this is that the content_context is a field of type dict, but it can receive different types, such as int, str, or None. Here is an example of the event data.

CourseNotificationData(
    course_key=CourseLocator("OpenedX", "Demo", "XBlocks", None, None),
    app_name="discussion",
    notification_type="new_discussion_post",
    content_url="http://apps.local.openedx.io:2002/discussions/course-v1:OpenedX+Demo+XBlocks/posts/27",
    content_context={
        "replier_name": "admin",
        "post_title": "Sample of Post",
        "course_name": "XBlocks",
        "sender_id": 4,
        "group_by_id": "course-v1:OpenedX+Demo+XBlocks",
        "username": "admin",
        "email_content": '<p style="margin: 0">Description of the Post</p>',
        "thread_id": "27",
        "topic_id": "course",
        "response_id": None,
        "comment_id": None,
    },
    audience_filters={},
)

Currently, the support for Union Types such as List[str | int] or dict[str, str | int | None] is not included, and for that reason is failing. And, the reason the event does not fail to answer an ORA assignment is that the event data looks like this:

CourseNotificationData(
    course_key=CourseLocator("OpenedX", "Demo", "XBlocks", None, None),
    app_name="grading",
    notification_type="ora_staff_notification",
    content_url="http://apps.local.openedx.io:1993/ora-grading/block-v1:OpenedX+Demo+XBlocks+type@openassessment+block@67abc97ee4de4c1f81360a7d5937773c",
    content_context={"ora_name": "Open Response Assessment", "course_name": "XBlocks"},
    audience_filters={"course_roles": ["staff", "instructor"]},
)

All value keys in content_context are strings.

Deadline

None

Checklists

Check off if complete or not applicable:

Merge Checklist:

Post Merge:

Create a tag
Create a release on GitHub
Check new version is pushed to PyPI after tag-triggered build is
finished.
Delete working branch (if not needed anymore)
Upgrade the package in the Open edX platform requirements (if applicable)

openedx-webhooks · 2025-03-19T01:15:04Z

Thanks for the pull request, @BryanttV!

This repository is currently maintained by @openedx/hooks-extension-framework.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
- This process (including the steps you'll need to take) is documented here.
If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

Dependencies

This PR must be merged before / after / at the same time as ...
Blockers

This PR is waiting for OEP-1234 to be accepted.
Timeline information

This PR must be merged by XX date because ...
Partner information

This is for a course on edx.org.
Supporting documentation
Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

The size and impact of the changes that it introduces
The need for product review
Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

mariajgrimaldi · 2025-03-26T16:53:18Z

FYI @bmtcril @robrap @timmc-edx, this is the follow-up work from the discussion in PR #433!

I'll be reviewing this shortly. Thank you, @BryanttV!

bmtcril · 2025-04-11T14:54:18Z

I'm going to attempt testing this locally against the redis and kafka backends today, sorry for the delay

bmtcril · 2025-04-14T15:14:16Z

So far I'm not able to test successfully against the redis bus. I believe I've set things up correctly, but when submitting an ORA response (Demo Course, Module 3, "Intermediate Assessment Tools" -> "Open Response Assessment (ORA)") I get the following error:

2025-04-14 10:54:04 Traceback (most recent call last):
2025-04-14 10:54:04   File "/openedx/venv/lib/python3.11/site-packages/edx_event_bus_redis/internal/producer.py", line 48, in record_producing_error
2025-04-14 10:54:04     raise EventProductionException(error)
2025-04-14 10:54:04 edx_event_bus_redis.internal.producer.EventProductionException: A Dictionary as a list item should have a type annotation.
2025-04-14 10:54:04 2025-04-14 14:54:04,765 INFO 22 [openedx_events.tooling] [user 4] [ip 192.168.65.1] tooling.py:178 - Responses of the Open edX Event <org.openedx.learning.ora.submission.created.v1>: 
2025-04-14 10:54:04 [(<function general_signal_handler at 0xffff9d91eac0>, None)]

Testing of the discussion event is in progress.

BryanttV · 2025-04-14T15:58:29Z

Hi @bmtcril, what branch are you using? You should use this branch with the changes in the data attributes.

bmtcril · 2025-04-14T17:41:20Z

@BryanttV I believe I'm on the correct branch since the error I'm seeing is a new one added in this PR's _get_avro_type_for_list_item. I haven't yet been able to trigger the discussions event, but that's probably some kind of configuration problem with forums on my end, I'm still looking into it.

bmtcril · 2025-04-15T19:30:06Z

@BryanttV I was able to confirm that I'm on the right branch, and testing the discussions case also have an error:

Traceback (most recent call last):
File "/openedx/venv/lib/python3.11/site-packages/edx_event_bus_redis/internal/producer.py", line 130, in send
event_bytes = serialize_event_data_to_bytes(event_data, signal)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/openedx/venv/lib/python3.11/site-packages/openedx_events/event_bus/avro/serializer.py", line 93, in serialize_event_data_to_bytes
serializer = AvroSignalSerializer(signal)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/openedx/venv/lib/python3.11/site-packages/openedx_events/event_bus/avro/serializer.py", line 124, in __init__
self.schema = schema_from_signal(self.signal, custom_type_to_avro_type=self.custom_types)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/openedx/venv/lib/python3.11/site-packages/openedx_events/event_bus/avro/schema.py", line 39, in schema_from_signal
base_schema["fields"].append(_create_avro_field_definition(data_key, data_type,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/openedx/venv/lib/python3.11/site-packages/openedx_events/event_bus/avro/schema.py", line 102, in _create_avro_field_definition
_create_avro_field_definition(attribute.name, attribute.type,
File "/openedx/venv/lib/python3.11/site-packages/openedx_events/event_bus/avro/schema.py", line 73, in _create_avro_field_definition
raise Exception("Unable to generate Avro schema for dict or array fields without annotation types.")
Exception: Unable to generate Avro schema for dict or array fields without annotation types.

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/openedx/venv/lib/python3.11/site-packages/edx_event_bus_redis/internal/producer.py", line 48, in record_producing_error
raise EventProductionException(error)
edx_event_bus_redis.internal.producer.EventProductionException: Unable to generate Avro schema for dict or array fields without annotation types.

BryanttV · 2025-04-16T15:36:20Z

Hi @bmtcril. I tested again, and it works for me. 🤔

I created two branches, the branch of this PR (bav/complex-types-in-dicts-and-lists-support) and (bav/complex-types-in-dicts-and-lists-samples). I'm sorry, the names are similar and can be confusing. I decided to add a temporary commit in the branch of this PR that updates the Data Attributes to work with the event bus (014c404).

bmtcril

Sorry for the long delay @BryanttV I finally got things in a state to test and it worked well for me. I haven't tried the Kafka bus. Thanks for your patience.

robrap · 2025-05-22T16:58:08Z

If and when this merges (or before), can someone ensure kafka CI passes once this gets updated? Does anyone have any concerns about that, or do we think it should pass?

mariajgrimaldi · 2025-06-30T10:37:50Z

openedx_events/tooling.py

+    # "org.openedx.learning.user.notification.requested.v1",
+    # "org.openedx.learning.forum.thread.created.v1",
+    # "org.openedx.learning.forum.thread.response.created.v1",
+    # "org.openedx.learning.forum.thread.response.comment.created.v1",
+    # "org.openedx.learning.course.notification.requested.v1",
+    # "org.openedx.learning.ora.submission.created.v1",


can we move these changes to another PR?

Sure, I moved these changes here: #510

mariajgrimaldi · 2025-06-30T10:51:03Z

@BryanttV, friendly ping to Robert's comment. Can we look more into that?

BryanttV · 2025-07-07T14:22:15Z

Hi @mariajgrimaldi @robrap! I created a PR to test the CI in event-bus-kafka: openedx/event-bus-kafka#319. All checks passed ✅

…d schema

openedx-webhooks added the open-source-contribution PR author is not from Axim or 2U label Mar 19, 2025

openedx-webhooks added this to Contributions Mar 19, 2025

github-project-automation bot moved this to Needs Triage in Contributions Mar 19, 2025

BryanttV force-pushed the bav/complex-types-in-dicts-and-lists-support branch from 62b7edd to e301f54 Compare March 19, 2025 03:36

mphilbrick211 moved this from Needs Triage to Waiting on Author in Contributions Mar 19, 2025

BryanttV marked this pull request as ready for review March 20, 2025 17:19

BryanttV requested a review from a team as a code owner March 20, 2025 17:19

BryanttV force-pushed the bav/complex-types-in-dicts-and-lists-support branch from bc3d3e6 to 7913f38 Compare March 31, 2025 20:12

BryanttV force-pushed the bav/complex-types-in-dicts-and-lists-support branch from 7913f38 to 15356f9 Compare April 16, 2025 14:37

BryanttV force-pushed the bav/complex-types-in-dicts-and-lists-support branch 2 times, most recently from eb20710 to 014c404 Compare April 16, 2025 16:19

bmtcril approved these changes May 22, 2025

View reviewed changes

mariajgrimaldi requested changes Jun 30, 2025

View reviewed changes

BryanttV force-pushed the bav/complex-types-in-dicts-and-lists-support branch from 014c404 to 48a968a Compare July 1, 2025 15:24

BryanttV mentioned this pull request Jul 1, 2025

chore: remove signals from unserializable list #510

Open

15 tasks

BryanttV requested a review from mariajgrimaldi July 7, 2025 14:22

BryanttV added 3 commits July 8, 2025 09:39

feat: support complex types of dicts and lists in avro schema

dda8c8f

feat: support complex types of dicts and lists in deserialization

e968f0a

feat: support complex types of dicts and lists in test functions

f4dc676

BryanttV added 11 commits July 8, 2025 09:39

test: remove tests according new changes

73cba33

feat: update data attributes to support complex types

e9d8d7e

chore: fix quality checks

40162e5

test: add tests for deserialization of complex types in dicts and lists

68596be

chore: add no cover statement in test function

b0394b7

test: add test for unsupported types in deserialization

38d6dc4

test: add test for complex types of list and dict for deserializer an…

752e9e9

…d schema

chore: remove changes in data attrs

1ffd94e

chore: improve inline comments and move code

85750da

chore: update data attrs

51cfe5f

chore: uncomment unserializable signals

027c672

BryanttV force-pushed the bav/complex-types-in-dicts-and-lists-support branch from e9c4c14 to 027c672 Compare July 8, 2025 14:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add support for complex types in dicts and lists #483

feat: add support for complex types in dicts and lists #483

Uh oh!

BryanttV commented Mar 19, 2025 •

edited

Loading

Uh oh!

openedx-webhooks commented Mar 19, 2025 •

edited

Loading

Uh oh!

mariajgrimaldi commented Mar 26, 2025

Uh oh!

bmtcril commented Apr 11, 2025

Uh oh!

bmtcril commented Apr 14, 2025

Uh oh!

BryanttV commented Apr 14, 2025

Uh oh!

bmtcril commented Apr 14, 2025

Uh oh!

bmtcril commented Apr 15, 2025

Uh oh!

BryanttV commented Apr 16, 2025 •

edited

Loading

Uh oh!

bmtcril left a comment

Uh oh!

robrap commented May 22, 2025

Uh oh!

mariajgrimaldi Jun 30, 2025

Uh oh!

BryanttV Jul 1, 2025

Uh oh!

mariajgrimaldi commented Jun 30, 2025

Uh oh!

BryanttV commented Jul 7, 2025

Uh oh!

Uh oh!

feat: add support for complex types in dicts and lists #483

Are you sure you want to change the base?

feat: add support for complex types in dicts and lists #483

Uh oh!

Conversation

BryanttV commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Supporting information

Testing instructions

COURSE_NOTIFICATION_REQUESTED: dict[str, List[str]]

ORA_SUBMISSION_CREATED: List[dict[str, str]]

COURSE_DISCUSSIONS_CHANGED: List[DiscussionTopicContext]

Other information

Deadline

Checklists

Uh oh!

openedx-webhooks commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mariajgrimaldi commented Mar 26, 2025

Uh oh!

bmtcril commented Apr 11, 2025

Uh oh!

bmtcril commented Apr 14, 2025

Uh oh!

BryanttV commented Apr 14, 2025

Uh oh!

bmtcril commented Apr 14, 2025

Uh oh!

bmtcril commented Apr 15, 2025

Uh oh!

BryanttV commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bmtcril left a comment

Choose a reason for hiding this comment

Uh oh!

robrap commented May 22, 2025

Uh oh!

mariajgrimaldi Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

BryanttV Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

mariajgrimaldi commented Jun 30, 2025

Uh oh!

BryanttV commented Jul 7, 2025

Uh oh!

Uh oh!

BryanttV commented Mar 19, 2025 •

edited

Loading

`COURSE_NOTIFICATION_REQUESTED`: `dict[str, List[str]]`

`ORA_SUBMISSION_CREATED`: `List[dict[str, str]]`

`COURSE_DISCUSSIONS_CHANGED`: `List[DiscussionTopicContext]`

openedx-webhooks commented Mar 19, 2025 •

edited

Loading

BryanttV commented Apr 16, 2025 •

edited

Loading