Other potential sources of test corpora

[gvanrossum]

I tweeted about this and got a few tips that I might have to follow up if the current corpus is not sufficiently useful.

• Meg Risdal from Kaggle might have some
• I got a tip about Buggify (looks immature)
• Apparently @Tobias-Kohn may have something
• David Kramer says Sentry might have something (DM for access) (Also Data Dreadnought could help)

FWIW I am currently using a corpus made available by the authors of this paper.

[Tobias-Kohn]

Yes, I do have several thousand programs. What exactly are you looking for: just the programs, or labelled data?

If a summary is helpful: I reported my findings concerning errors in a paper The Error Behind the Message, which is also available from my homepage.

Just as a disclaimer: the programs I have collected are all anonymised (before they were sent to me), i.e. the strings and comments basically comprise only xs, e.g., `s = "Xxxx x xx xxxx", and large numbers are scrambled. Since I am interested in syntax errors and conceptual misunderstandings, this works just fine for me. However, some of the runtime errors might be quite distorted.

We could also ask Philip Guo, who might have some data from his Python Tutor.

[Tobias-Kohn]

By the way: my Python-parser comes with a very small test suite, which contains some erroneous programs. They are in a format where the first two lines contain the expected number of the erroneous line and the expected error message, followed by a small program. Would that kind of format work for you?

[gvanrossum]

Yes, that format works for me. It will encourage me to stop working with straight JSON and just load test cases into a sqlite3 database. :-) I notice that your "expected error message" field is some kind of enum. Do you have a way to map these to what CPython produces? (No worries if you don't.)