Is there a way to feed speaker information to the coref processor?

[amir-zeldes]

I would like to use the coref processor for dialogues where I know the speaker of each sentence. This should help eliminate spurious I/you coref chains that are obviously wrong if we know who is speaking.

I can see in the code that the model is aware of speakers during training, but I can't figure out how to feed it in the input. Is there a way to do this?

[AngledLuffa]

Unfortunately, no. I'm not sure what it would look like as the current paradigm is to feed in raw text. Of course, there's also the mechanism for feeding in a pretokenized document. Perhaps we could attach speaker information to the words in the pretokenized docs, and the coref processor extracts that?

Of course, it would need to be a model trained on a dataset that paid attention to speakers

[amir-zeldes]

Thanks for the quick response! In the code it looks like it's built to train that way, and I assume the numbers on OntoNotes reflects scores from data with speakers (otherwise it's hard to approach SOTA numbers). I can sort of fake it by running just the coref processor with a long list that has one instance of a speaker string per token, like this:

            tokens = ["Hi","!","Hello","!"]
            speakers = ["bob","bob","linda","linda"]

            coref_doc = {"cased_words": tokens, "deprel": deprels, "speaker": speakers, "document_id": "wb000",
                         "subwords": subwords, "word2subword": word2subword, "word_id": word_id, "sent_id": sent_ids,
                         "span_clusters": [], "word_clusters": [], "head2span": []}

            coref_output = coref_processor._model.run(coref_doc)

Or at least I think that's what it's expecting, but it's hard to merge the result back with a full pipeline's output, and like you say, I'd need a model trained this way. It would be nice to just be able to specify the speakers in inputs.

If I was using conllu input I'd expect to just provide a # speaker = linda comment at the beginning of each sentence, and that would also be pre-tokenized since it's conllu. Would you consider allowing conllu as an input format?

If not, a simple alternative could be a list of identifiers per sentence or per token, like I have in the code above, but you'd need to know how many sentences/tokens will exist after tokenization. Finally, we could figure out a way to include them in plain text, e.g. insert {{speaker}} any time a speaker change occurs inside plain text. What do you think?

[AngledLuffa]

Under the hood it's building the internal coref format here:

stanza/stanza/pipeline/coref_processor.py

Line 103 in 8be9ce8

coref_input = {

and that's not using any speaker information. So, if there's a connection from word (or sentence) speaker to the coref model there, it should work. It would either require using a preprocessed Stanza Document, or as you say, we could add the functionality to process conllu as already-tokenized data as well.

[amir-zeldes]

It looks like some version of the code did know about it:

• stanza/stanza/models/coref/pairwise_encoder.py

  Line 81 in 156f86b

  def _speaker_map(doc: Doc) -> List[int]:

• stanza/stanza/models/wl_coref.py

  Line 94 in 156f86b

  argparser.add_argument("--full_pairwise", action="store_true",

It's possible this was just some part of experiments for a paper, maybe @Jemoka can say more? But realistically, if we have multi-party dialog data, we need to know the speakers to do coref properly, otherwise all of the I/yous will get mixed up. Datasets like GUM and probably others in CorefUD have # speaker comments, and OntoNotes has the information in separate files as well.

Personally I would love to have conllu input support, but I'm happy to build whatever input format works best for you.

[Jemoka]

We can have speaker input! A long time ago our OntoNotes model were trained with speaker info (I think we can still find the speaker info / embedding somewhere), and after a while we decided that for production people will not have that information around. I can certainly bake a model with this info if people want, but would love @AngledLuffa's takes on how to supply that information to the Pipeline API (by and large it doesn't really support additional information unless its already in Stanza documents, etc.)

True that in terms of training --full_pairwise can address this, but actually there are additional infrastructure that the flag triggers during research phase that I ripped out. Can put it back in ofc.

[AngledLuffa]

Seems like a short list of things to do:

• have speaker as an attribute on Sentence (do we want it on individual Words?)
• if there's a speaker on the sentences or words, pass that along to the coref model
• CoNLL.conll2doc looks at #speaker and puts it on the Sentence
• convenience method that calls CoNLL.conll2doc and processes it in one call

Broken down like that, every single piece seems quite simple. Anything missing?

[AngledLuffa]

based on @Jemoka 's comments, perhaps also

• rebuild a model with speakers

[amir-zeldes]

Yes, that would be fantastic!

have speaker as an attribute on Sentence (do we want it on individual Words?)

For GUM, we do have this information annotated below the sentence level in genres like fiction, where direct speech only covers part of a sentence, but if it doesn't cover an entire sentence, it's only available in the XML data/CoNLL-U XML MISC attribute. For example compare these two sentence from the same document:

# sent_id = GUM_fiction_pag-57
# speaker = Siri
# addressee = Pag
# text = "Sorry."
1	"	"	PUNCT	``	_	2	punct	2:punct	Discourse=joint-sequence_m:130->128:1:_|SpaceAfter=No
2	Sorry	sorry	ADJ	JJ	Degree=Pos	0	root	0:root	MSeg=Sorr-y|SpaceAfter=No
3	.	.	PUNCT	.	_	2	punct	2:punct	SpaceAfter=No
4	"	"	PUNCT	''	_	2	punct	2:punct	_

In the sentence above, the speaker covers the whole sentence, so we get a # speaker comment and an # addressee comment. But this one is different:

# sent_id = GUM_fiction_pag-60
# speaker = Pag
# addressee = Siri
# text = "Oh shit," Pag panted.
1	"	"	PUNCT	``	_	3	punct	3:punct	Discourse=joint-sequence_m:135->31:1:_|SpaceAfter=No|XML=<sp who:::"#Pag" whom:::"#Siri">
2	Oh	oh	INTJ	UH	_	3	discourse	3:discourse	_
3	shit	shit	INTJ	UH	_	7	ccomp	7:ccomp	SpaceAfter=No|XML=<hi rend:::"italic"></hi>
4	,	,	PUNCT	,	_	3	punct	3:punct	SpaceAfter=No
5	"	"	PUNCT	''	_	3	punct	3:punct	XML=</sp>
6	Pag	Pag	PROPN	NNP	Number=Sing	7	nsubj	7:nsubj	Discourse=attribution-positive:136->135:0:grf-qt-892,896-_+sem-atsrc-897-gold+lex-indwd-898-_|Entity=(14-person-giv:inact-sssss-cf1-1-coref)
7	panted	pant	VERB	VBD	Mood=Ind|Number=Sing|Person=3|Tense=Past|VerbForm=Fin	0	root	0:root	MSeg=pant-ed|SpaceAfter=No
8	.	.	PUNCT	.	_	7	punct	7:punct	_

Since the narrator is the 'speaker' for some of the words, we only know the speaker of the quoted part from the MISC annotation XML in tokens 1 and 5, XML=<sp who:::"#Pag" whom:::"#Siri"> marking the start and XML=</sp> marking the end. AFAIK, this type of markup, which is based on TEI XML, is only found in UD in GUM, GUMReddit and GENTLE.

[Jemoka]

have speaker as an attribute on Sentence (do we want it on individual Words?)

Yup, unless there's code switching, we don't need it on words.

if there's a speaker on the sentences or words, pass that along to the coref model

Yup, should already be done.

CoNLL.conll2doc looks at #speaker and puts it on the Sentence
convenience method that calls CoNLL.conll2doc and processes it in one call3

Yes! and OntoNotes processor already annotates speaker information that was used by previous iteration of the model.

rebuild a model with speakers

Indeed.

[AngledLuffa]

if there's a speaker on the sentences or words, pass that along to the coref model

Yup, should already be done.

Unless you mean in a branch, the coref_processor in the Pipeline does not pass speaker to the model

[Jemoka]

aah no, not on the processor; I meant in training pipeline.