NR-347971 | OpenLineage event consumer writer #1959
Conversation
NR-347971 | Added conditions
NR-347871 | Remove redundant line
…-nr/entity-definitions into NR-347971-add-condition
NR-347971 Add condition and tests
NR-347971 | Added filter condition on unique attribute
Nr 347971 | Fix test case
devendra-nr
changed the title
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| attributes: | ||
| - dataset.namespace | ||
| - dataset.name | ||
| - updatedAt |
There was a problem hiding this comment.
why updatedAt?
Wouldn't this create a new entity every time the field is updated? is that desired?
Can we talk about how many entities are we expecting here?
There was a problem hiding this comment.
The number of events received with updatedAt field are quite low ~10 per month per account.
This event is generated when schema of dataset changes and such events occur rarely but they are very important.
We want to track the changes as separate entity.
| entityTagName: olin.dataset.updatedAt | ||
| multiValue: false | ||
| configuration: | ||
| entityExpirationTime: MANUAL |
There was a problem hiding this comment.
Entities that use synthesis can't be manual they must have a TTL
There was a problem hiding this comment.
For our use case we want to store the history of changes to dataset schema for very long period 12+ months.
Also the number of entities generated are quite low ~10 per month per account.
So we want to handle the delete logic manually in our codebase.
|
marked as internal review see https://newrelic.slack.com/archives/C07ELLRFDG9/p1741961591913679?thread_ts=1741942875.625519&cid=C07ELLRFDG9 |
NR-347971 | Update expiry ttl
Relevant information
Added domain and entities for Data Pipeline Observability product
Checklist
I've read the guidelines and understand the acceptance criteria.
The value of the attribute marked as identifier will be unique and valid.
I've confirmed that my entity type wasn't already defined. If it is I'm providing an explanation above.