Duplicate logs with GitHub API data ingestions

[l-koppuravuri-BL]

Describe the bug
We tried using function apps and logic apps to ingest GitHub data to Sentinel, and we found that both solutions were producing duplicate data. We wanted to make sure before looking into the code to see if this was a known problem or if there were any imitations.

To Reproduce
Steps to reproduce the behavior:
1As mentioned in the documentation, deployed solutions and updated orgs , lastjobruntime.json files. jobs running with default schedule of 10 minutes.

Expected behavior
should not see the duplicate logs.

Screenshots

[l-koppuravuri-BL]

I have already looked into issue # #9356, but the solution offered has not helped because I am already using an org.json file with the new structure and do not see any rate limits.

[v-rusraut]

Hi @l-koppuravuri-BL , Thanks for flagging this issue, we will investigate this issue and get back to you with some updates. Thanks!

[v-sudkharat]

@l-koppuravuri-BL, you are using this data connector to pull the data ? - https://github.com/Azure/Azure-Sentinel/tree/master/DataConnectors/GithubFunction

And Is there any other connector configured and pointing out to the same workspace?

[l-koppuravuri-BL]

@v-sudkharat No additional connectors are configured on this workspace.

[onyigbo]

In the logic app step where you have the API url, were you able to specific the time interval and format (start and end time, some might want ISO format) based on Github documentation on how the API should be used?

[v-sudkharat]

@l-koppuravuri-BL, Could you please confirm this connector has been configured - https://github.com/Azure/Azure-Sentinel/tree/master/DataConnectors/GithubFunction

Or please let us know if any logic app has been set into env mentioned in above comment. Thanks!

[l-koppuravuri-BL]

@v-sudkharat We have set up connectors based on both logic apps and function apps, and we have observed that they behave in the same way.

Since both connectors report to distinct log analytics workspaces, there should not be any conflicts or duplicate logs.

@onyigbo : I am not sure to whom you are addressing the time interval question, but I have not set it up that way and did not notice any input parameters during deployment. Image