Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding support for transcripts, recording, AI summarization and meeting subscription to channels #377

Open
wants to merge 29 commits into
base: master
Choose a base branch
from

Conversation

jespino
Copy link
Member

@jespino jespino commented May 15, 2024

This PR include 3 important changes:

  1. It allows the user to subscribe channels to meetings. That way, whenever that meeting is started outside mattermost it will post a message with a link to the meeting (identical to the one is create when you click the zoom button).
  2. Whenever a meeting is finish, if there is a recording it is going to report the recording, the transcription and the chat messages in replies to the original zoom post (this works for the meetings in the point 1 and for the meetings that are created using the zoom button).
  3. If you have the mattermost AI Copilot plugin installed and configured, whenever you receive the transcription and the chat, you can summarize them using the AI plugin.

Testing

For testing this, you should test the 3 features independently (there is some degree of overlap, but I going to propose independent test cases). Another important thing is that before you test it you need to configure the zoom application in the zoom marketplace with the right webhook events (it has change, so check out the documentation).

I expect all the test proposed here are executed after properly configuring zoom plugin and AI copilot plugin

Test case 1: Subscription

  • Go to zoom web interface and create a recurrent meeting.
  • Go to a channel and execute /zoom subscribe [meeting id], where [meeting id] is the ID of the created meeting in the pervious step (without spaces)
  • Go to the zoom web interface and initiate the recurrent meeting.
  • The expected behavior is that in some seconds after that, you see the post in the channel that is subscribe.
  • Stop the meeting.
  • After some seconds you should see reflected in the post that the meeting has ended
  • Go back to the channel and execute /zoom unsubscribe [meeting id], with the same meeting id.
  • Go back to the zoom web interface and initiate the recurrent meeting again.
  • The expected behavior is you don't see any post created in the channel.

Test case 2: Recordings, Chats, and Transcriptions (Without AI Copilot enabled)

  • Go to any channel and click the zoom button to start a meeting.
  • You will see the zoom post and you get automatically connect to the meeting.
  • Leave the meeting.
  • The post in some seconds is going to be mark as terminated.
  • No other messages are published later (you can skip this step because you can verify it later).
  • Open another zoom meeting using the same process.
  • Send a chat message
  • Start the recording
  • Say some text to get that in the transcription
  • Send another chat message
  • Stop the recording
  • Leave the meeting.
  • After this, you will see the post message changing to ended (in some seconds).
  • Then, after some minutes (maybe around 5), you will receive 2 new posts. one attaching the transcription, and one attaching the chat and including a link to the recording with the password.
  • Check the transcription and see if that is the one.
  • Check the chat and verify that only the message sent during the recording is there.
  • Check the recording link and password (maybe you need to use an incognito window to require the password)

Test case 3: AI Summarization (With AI Copilot enabled)

  • Follow all the steps of the test case 2 with AI copilot enable
  • In the transcription message and in the chat message you should see a new button that allows you to summarize the content.
  • Click that button in the chat history message and see the AI copilot summarizing it
  • Click that button in the transcription message and see the AI copilot summarizing it
  • If the text in the chat history or the transcription is too short the AI copilot can send you non-sense, if you want to have a better result with the AI copilot use more messages in the chat and speak more to include more text in the transcription.

@jespino jespino requested a review from mickmister as a code owner May 15, 2024 17:56
@jespino jespino added the Do Not Merge Should not be merged until this label is removed label May 15, 2024
var response *http.Response
for retries > 0 {
var err error
response, err = http.DefaultClient.Do(request)

Check failure

Code scanning / CodeQL

Uncontrolled data used in network request Critical

The
URL
of this request depends on a
user-provided value
.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is because we are handling the incoming webhook from zoom as something secure, it should be signed if you configured it properly

return
}
request.Header.Set("Authorization", "Bearer "+webhook.DownloadToken)
response, err := http.DefaultClient.Do(request)

Check failure

Code scanning / CodeQL

Uncontrolled data used in network request Critical

The
URL
of this request depends on a
user-provided value
.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is because we are handling the incoming webhook from zoom as something secure, it should be signed if you configured it properly

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CodeQL code tracing is 👌

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yes, I check the "show paths" and I was really impressed, It is pure gold.

@mickmister mickmister added 2: Dev Review Requires review by a core committer 3: QA Review Requires review by a QA tester labels May 15, 2024
Copy link
Contributor

@mickmister mickmister left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent feature here 🚀

I gave the PR a review and added some comments for discussion. Let me know what you think 👍

return
}
request.Header.Set("Authorization", "Bearer "+webhook.DownloadToken)
response, err := http.DefaultClient.Do(request)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CodeQL code tracing is 👌

server/command.go Show resolved Hide resolved
webapp/src/components/post_type_chat.jsx Outdated Show resolved Hide resolved
webapp/src/components/post_type_chat.jsx Outdated Show resolved Hide resolved
server/command.go Outdated Show resolved Hide resolved
server/webhook.go Show resolved Hide resolved
server/webhook.go Outdated Show resolved Hide resolved
server/webhook.go Show resolved Hide resolved
server/webhook.go Show resolved Hide resolved
server/webhook.go Outdated Show resolved Hide resolved
@jespino jespino requested a review from mickmister May 17, 2024 08:13
@jespino
Copy link
Member Author

jespino commented May 17, 2024

@mickmister PTAL

Copy link
Contributor

@mickmister mickmister left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

server/webhook.go Outdated Show resolved Hide resolved
@@ -241,18 +241,40 @@ func (p *Plugin) runConnectCommand(user *model.User, extra *model.CommandArgs) (
}

func (p *Plugin) runSubscribeCommand(user *model.User, extra *model.CommandArgs, meetingID int) (string, error) {
if !p.API.HasPermissionToChannel(user.Id, extra.ChannelId, model.PermissionCreatePost) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a product question - Who should have permissions to subscribe a channel to a meeting id? Probably good to DRY it up into its own method as well

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a bit of repetition here is not that bad, anyway, I think the way of thinking about it, for me is, if you are able to publish a post, you will be able to do it anyway, but I'm ok if we only allow this to channel admins, for example.

server/webhook_test.go Show resolved Hide resolved
@jespino jespino added the 3: Security Review Review requested from Security Team label Jun 7, 2024
@azigler
Copy link

azigler commented Jun 26, 2024

@jespino trying to build this locally and encountering some issues that are probably really simple to fix -- maybe my environment is configured incorrectly. Can you help me out?

EDIT: Also, I'm looking to configure this Zoom plugin in my own Zoom workspace (which I registered and set up) but I need to know a little more about the scopes and events you set up for this plugin. Can you share your example?

> make server

mkdir -p server/dist;
cd server && env CGO_ENABLED=0 GOOS=linux GOARCH=amd64 /usr/local/go/bin/go build  -ldflags '-X "github.com/mattermost/mattermost/server/public/pluginapi/experimental/telemetry.rudderWriteKey=1d5bMvdrfWClLxgK1FvV3s4U1tg"'  -trimpath -o dist/plugin-linux-amd64;
# github.com/mattermost/mattermost-plugin-zoom/server
./configuration.go:133:105: undefined: manifest
./http.go:459:46: undefined: manifest
./http.go:900:46: undefined: manifest
make: *** [server] Error 1
> make webapp

./build/bin/manifest apply
cd webapp && /Users/andrew/.nvm/versions/node/v18.12.1/bin/npm install
npm ERR! code ERESOLVE
npm ERR! ERESOLVE could not resolve
npm ERR! 
npm ERR! While resolving: [email protected]
npm ERR! Found: [email protected]
npm ERR! node_modules/react
npm ERR!   react@"17.0.2" from the root project
npm ERR!   peer react@">=16.8.0" from @emotion/[email protected]
npm ERR!   node_modules/@emotion/react
npm ERR!     dev @emotion/react@"11.11.0" from the root project
npm ERR!   4 more (@emotion/use-insertion-effect-with-fallbacks, ...)
npm ERR! 
npm ERR! Could not resolve dependency:
npm ERR! peer react@"^18.3.1" from [email protected]
npm ERR! node_modules/react-dom
npm ERR!   peer react-dom@">= 16.8.0" from [email protected]
npm ERR!   node_modules/styled-components
npm ERR!     styled-components@"6.1.10" from the root project
npm ERR! 
npm ERR! Conflicting peer dependency: [email protected]
npm ERR! node_modules/react
npm ERR!   peer react@"^18.3.1" from [email protected]
npm ERR!   node_modules/react-dom
npm ERR!     peer react-dom@">= 16.8.0" from [email protected]
npm ERR!     node_modules/styled-components
npm ERR!       styled-components@"6.1.10" from the root project
npm ERR! 
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force, or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.
npm ERR! 
npm ERR! See /Users/andrew/.npm/eresolve-report.txt for a full report.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/andrew/.npm/_logs/2024-06-26T21_44_49_459Z-debug-0.log
make: *** [webapp/node_modules] Error 1

@azigler
Copy link

azigler commented Jul 2, 2024

I consulted @jespino and was able to build by using nvm install and then make dist.

On the Zoom side, after creating the integration according to the documentation, I went to Features then Access and toggled on Event Subscription and subscribed to these events:

  • Record ➡️ all recordings have completed, recording transcript files have completed
  • Meeting ➡️ start meeting, end meeting

Then I went to Scopes and enabled these scopes:

Scopes Description
meeting:read:meeting View a meeting
meeting:write:meeting Create a meeting for a user
cloud_recording:read:recording View a recording
archiving:read:list_archived_files Get an account's archived meeting or webinar files.
user:read:user View a user

I started following the steps for Test Case 1 but after creating a recurring meeting in the Zoom interface, I was unable to subscribe to it in a channel with /zoom subscribe [meeting ID without spaces]. I get this error: Can not subscribe to meeting: meeting not found

I'm unable to test Test Case 2 because as far as I can tell, this requires a paid Zoom account so you have access to cloud recordings? Please let me know if there's another way to test this.

My main takeaway is that this setup has several steps. You have to configure the Mattermost plugin and the Zoom app in parallel, because they require each other's fields to complete. Then you have to add all the scopes, then configure the webhook events and add them. And then you have to use slash commands to actually set it up. I think there's still some improvement to be done for the admin setup experience. What if we make this a step-by-step prompted guide via a /setup slash command or something? I think the Jira plugin does this successfully.

I propose that an issue is opened to update the docs page for this plugin to reflect these new scopes and events. Right now, the docs for configuring the webhook events don't tell you what events you need enabled (it just suggests a name, maybe reflects an old UI on Zoom?)

In terms of producing video asset(s), I see two objectives:

  • Create a video for social share on LinkedIn that shows Mattermost linked to Zoom for meeting notifications and summaries, highlighting the usefulness. Zoom is a broadly recognizable software with a large audience share, and getting people excited about this is a priority. What is this plugin solving for the user? Why wouldn't the user just enable the Zoom AI summaries and just use those? Why link it to Mattermost? That's what this video needs to address, by show the following:
    • Position Mattermost as the secure collaboration hub where data and communication from platforms are collected for discussion and reference.
    • Show the power of using Copilot to extract summaries. Why use Copilot? You could have a local model and extract AI summaries from them with that instead of Zoom, if you wanted to avoid using a vendor's model. You could also use new custom prompts in multi-LLM support to provide your keywords and terminology to the LLM to improve its accuracy when summarizing Zoom calls with lots of acronyms or special verbiage.
  • Longterm, this setup process and best practices for using the tools (Zoom + Copilot) together might benefit from being in the Academy.

Next actions:

  • @jespino can you help me get Test Case 1 working?
  • @jespino how are you testing Test Case 2 & 3 without a paid Zoom account, because that's a requirement for cloud recordings?
  • @jespino What are your thoughts on why someone would use Zoom + Copilot for AI summarizations instead of just Zoom itself? Is that the real value that this adds, or do you think it's something else?
  • @cwarnermm Can we partner up to make sure these configuration docs reflect the latest and greatest?

cc @jasonblais

@azigler
Copy link

azigler commented Jul 15, 2024

I opened a docs PR to correspond with the changes in this PR.

mattermost/docs#7276

@azigler
Copy link

azigler commented Jul 16, 2024

@jespino Currently there is no way to know what meetings a channel is subscribed to, which makes it hard to know what state it's in. Should we add a slash command (maybe /zoom list) that shows subscriptions for the channel?

@azigler
Copy link

azigler commented Sep 6, 2024

Is there a timeline for merging this?

@jespino
Copy link
Member Author

jespino commented Sep 6, 2024

@azigler not really, we are still waiting for the security review and there are other things that are higher priority.

@wiggin77
Copy link
Member

@jespino Any appetite for getting this merged while we're still using Zoom?

@jespino
Copy link
Member Author

jespino commented Sep 26, 2024

@wiggin77 I would love to see this merged, but it requires the security review and it is still pending, I guess V10 release has keep the security team busy for this kind of things. @esarafianou any news on this?

Copy link
Member

@wiggin77 wiggin77 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this request for change to get this off my review queue until it is ready.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2: Dev Review Requires review by a core committer 3: QA Review Requires review by a QA tester 3: Security Review Review requested from Security Team Do Not Merge Should not be merged until this label is removed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants