Data Source Identification 2025 plan

[josh-chamberlain]

Context

We have some pieces coming together, but it's time to focus efforts around a few central goals. Our timeline is through June 2025.

Overall Goals

1. Grow the database with quality sources according to our priorities
   • common topics: analyzing response effectiveness (calls for service, dispatch); documenting interactions & outcomes (stops, arrests, use of force); understanding basics of systems (agency completion & metadata, personnel)
   • geographic: go deep on one county at a time, starting with the most populous
   • followed areas: add sources to searches followed by our users
2. Improve labeling models
   • eventually, we should be able to get a fairly accurate relevancy, record type, and agency from trained language models. This is easier the more quality sources there are in the database.
3. Quickly investigate requests

System overview

Check out the diagram in the README!

Q3–4

Close the loop

• Automatically update training data from source collector #89
• Tweak Data Collection #141

Show off

• Data Collection Status map pdap.io#342

New Source Collectors

• New Source Collector: "research" ML model to generate batches #299
• Scraper templates / strategies for common data portals scrapers#216
• "citation follower": track down sources used in published analysis

Encourage participation, add Big Value

• Data Collection Status map pdap.io#342
• Automatically create batches periodically based on followed locations
  • Add "location" to batch creation #260
• One-step source submission
  • all we need is a URL; we'll look up the rest of the metadata for you.
• Gradually (visibly!!!) pre-fill high-confidence annotations for human approval.

Maintain the dataset

• when sources break (a change in URL status) we should try to find a replacement source
• agency crawler automation: check known websites for sources were moved, updated, or added
• Log usefulness of sources
  • how often are people clicking?
  • capture sentiment (did you find what you needed with this search? is this source useful?)\

Improve the models

• Try Out New Machine Learning Models #142
• make training happen on digital ocean #41
• Replace the tag collector with an enhanced version, or an image snapshot.

Q1–2 (we got a lot done!)

Build the engine

• Data Source Identifier: decide DB & API structure #112
• Data Source Identifier app #109

Refinements (feature creep!)

• Source Collector duplicate checking #198
• Source Collector "agency homepage" labeling #199
• Relevancy Labeling: add options #223

Use metrics to measure success

• Automate data cleaning #117
• "Collector priorities" tab of the source collector #167
• Keep track of the cost of running the collector, and generating batches, in the form of compute time. Ideally, we'll come up with a cost for different types of crawls, and use them to direct donations.
• Data Source ID: Simple reporting dashboard #125

Run the engine

• Automated search engine calls - Auto-Googler #53
• Data source ID: Retool interface #126

[maxachis]

@josh-chamberlain One thing to consider in this plan is how Machine Learning will play into it. One thing I'm thinking about is that we may be able to also put our ML stuff on the cloud so that we could command it to run as needed.

Now that might come with some pricing considerations: If we don't use the model all that often, it may not make sense to use a Digital Ocean pricing model, which charges not by usage but per month.

[josh-chamberlain]

@maxachis for sure. if we get to the point where our models are actually reliably useful, this is probably a good idea. Simplest is probably to just upgrade our HF plan to allow cloud compute: https://huggingface.co/pricing