Resolution plan for divergent types issues

[pudo]

So these are all the properties identified by @tillprochaska's script which have different types in different schemata and thus create a mess when trying to generate an ES schema from FtM entities. Here's some proposals for how to address each:

author:

• Assessment:author - entity
• Document:author - string

Resolution: Unsure. Can you @tillprochaska determine the exposure of OCCRP on Assessment in general? I never quite figured out what that's supposed to mean. We could rename Assessment:author to Assessment:authorEntity, but I wonder if we can explore abolishing Assessment entirely, or at least clarify its semantics?

organization:

• Position:organization - entity
• Post:organization - string
• Membership:organization - entity
• Directorship:organization - entity

Resolution: Post is already meant to be removed in FtM 4.0

number:

• UserAccount:number - phone
• Identification:number - identifier

Resolution: propose moving UserAccount:number to UserAccount:phone. Can we get a sense of how many such values are set, @tillprochaska and also possibly @catileptic/@simonwoerpel?

authority:

• Sanction:authority - string
• CallForTenders:authority - entity
• Contract:authority - entity
• Identification:authority - string

Resolution: complete mess. I don't know. We use both Sanction and Identification extensively and would need to post a change announcement to our customers if we move that.

duration:

• Video:duration - number
• Call:duration - number
• Audio:duration - number
• Sanction:duration - string

Resolution: shift Sanction:duration to Sanction:term - or Risk:something?

area:

• RealEstate:area - number
• License:area - string

Resolution: License is likely much less used. We could rename to scope? Or location with type: address? @tillprochaska Can you get an index count here, too?

subject:

• Message:subject - string
• Email:subject - string
• UnknownLink:subject - entity

Resolution: nasty. subject is a key part of UnknownLink, so I'm actually wondering if we want to replace the whole schema (eg. with a new OtherLink and make the transition schema -> schema, rather than breaking UnknownLink really hard.

sender:

• Email:sender - string
• Message:sender - entity
• EconomicActivity:sender - entity

Resolution: really nasty. This is emitted for basically every email by the ingestors, so moving Email:sender to Email:senderName is a massive installed base problem.

[tillprochaska]

I’m going to leave a few separate comments as I go through these schema changes, starting with Assesment:

author

Can you @tillprochaska determine the exposure of OCCRP on Assessment in general? I never quite figured out what that's supposed to mean. We could rename Assessment:author to Assessment:authorEntity, but I wonder if we can explore abolishing Assessment entirely, or at least clarify its semantics?

Except for one dataset, use of Assesment:author is very rare. Assesment in general is used occasionally, checking with our data team.

[tillprochaska]

number

Resolution: propose moving UserAccount:number to UserAccount:phone. Can we get a sense of how many such values are set, @tillprochaska and also possibly @catileptic/@simonwoerpel?

This should be fine.

[tillprochaska]

authority

Resolution: complete mess. I don't know. We use both Sanction and Identification extensively and would need to post a change announcement to our customers if we move that.

Same here.

[tillprochaska]

duration

Resolution: shift Sanction:duration to Sanction:term - or Risk:something?

Except for one dataset (+ OpenSanctions data) no usage over here. Checking with the data team, but I think this should be fine.

[tillprochaska]

area

Resolution: License is likely much less used. We could rename to scope? Or location with type: address? @tillprochaska Can you get an index count here, too?

License is actually used quite often, although inconsistently for any of the following use cases:

• the size of the geographic area covered by a (mining) license
• name of the geographic area/region
• address
• scope of the license (industry sector, type of business)

License already inherits address and addressEntity, so that case is covered, but adding scope (string) and area (number) as additional properties might make sense?

[tillprochaska]

Tagging @kdeden @pudo @uhhhuh for visibility/feedback

[tillprochaska]

sender

Resolution: really nasty. This is emitted for basically every email by the ingestors, so moving Email:sender to Email:senderName is a massive installed base problem.

Wondering how useful Email:sender is. In most cases, it should be the same as Email:from -- although the cases where it isn’t might be interesting.

[tillprochaska]

Quick note so I don’t forget about it: In cases where we rename existing properties, it might make sense to already add them in a 3.x version while marking the old property as deprecated, to make it possible to migrate crawlers etc. before actually making the switch.

[pudo]

Regarding the Email:sender thing - I remember digging into this while working on ingest-file, and it's definitely a difference in the spec of what an email is, but I'm not sure it is important in how emails are actually used. The idea is that bossman would be From, and their personal assistant would be Sender.

[tillprochaska]

Yes, exactly. I’ll try to figure out whether I can find any real-world cases where From and Sender are different and whether it’s of any actual use in investigations.