Implement dynamic topic model instead of regular LDA

[juanrloaiza]

No description provided.

[juanrloaiza]

Gensim has an implementation of DTM, but it is incredibly slow. This has been reported as an issue, but no solution has been found yet. Not even changing LdaSeqModel to use LdaMulticore helps.

There is a pull request that improves this implementation, but it hasn't been merged yet:

It is therefore recommended to still use the old DTM wrapper in Gensim 3.8.3 to use the binary from Blei-lab. This requires two files:

• dtm-linux64/darwin64: Blei-lab's C implementation of DTM pre-compiled, depends on the system (could also be compiled from source).
• dtmmodel.py: Gensim's 3.8.3 wrapper (no longer supported in >4.0)

The Gensim 3.8.3 wrapper is included in notebooks/utils, but the binary must be downloaded for each OS.

Finally, I commented the LdaSeqModel code in case the PR above gets merged soon.

latinamerican-philosophy-mining/notebooks/utils/model.py

Lines 40 to 59 in 0c868ad

	"""
	self.ldaseq = LdaSeqModel(
	corpus=corpus_bows,
	time_slice=self.corpus_obj.get_time_slices(time_window=time_window),
	num_topics=self.num_topics,
	id2word=self.id2word,
	passes=15,
	initialize="ldamodel",
	lda_model=self.lda,
	random_state=seed,
	)"""
	self.ldaseq = DtmModel(
	dtm_path="utils/dtm-linux64",
	corpus=corpus_bows,
	time_slices=self.corpus_obj.get_time_slices(time_window=time_window),
	num_topics=self.num_topics,
	id2word=self.id2word,
	rng_seed=0,
	)

[juanrloaiza]

We used Blei's binary. Closing.