Target highly downvoted comments more when scraping Reddit #8
Description
I don't think that scraping Reddit blindly will yield a high enough number of insults in the dataset. I'd estimate that ~98-99% of Reddit comments are NOT insulting. Blind scraping will mean an awfully skewed initial dataset.
In order to try and improve this situation, I think that highly downvoted comments in select Subreddits (eg. r/thedonald, r/subredditdrama) will bring the % of insults in a comment group nearer to maybe 5%.
A possibly better way to do this would be to check for locked threads, as locked threads are those where the conversation has gone to shit, often because users have started to be highly uncivil to each other.