Use the new OGM to export cpg as json #2392

peckto · 2025-07-17T07:05:43Z

Transition from the Neo4j OGM to the cpg internal OGM to persist cpg as json, analog to the Neo4j persistence.

One problem I'm currently facing with this approach is, that in some cases nodes with duplicate id (uuid) occur.
Not sure if those nodes are actually the same, or if there is a collision with the uuid.

Example:
https://github.com/pq-crystals/dilithium

    {
      "id": "ffffffff-ad71-abff-ffff-ffffcb38da74",
      "labels": [
        "Node",
        "Declaration",
        "ValueDeclaration",
        "VariableDeclaration"
      ],
      "properties": {
        "isArray": false,
        "isGlobal": true,
        "isImplicitInitializerAllowed": false,
        "argumentIndex": 0,
        "isImplicit": false,
        "isInferred": true,
        "fullName": "pqcrystals_dilithium2_ref_idxlut",
        "localName": "pqcrystals_dilithium2_ref_idxlut",
        "nameDelimiter": "::",
        "name": "pqcrystals_dilithium2_ref_idxlut"
      }
    },
    {
      "id": "ffffffff-ad71-abff-ffff-ffffcb38da74",
      "labels": [
        "Node",
        "Declaration",
        "ValueDeclaration",
        "VariableDeclaration"
      ],
      "properties": {
        "isArray": false,
        "isGlobal": true,
        "isImplicitInitializerAllowed": false,
        "argumentIndex": 0,
        "isImplicit": false,
        "isInferred": true,
        "fullName": "pqcrystals_dilithium2_ref_idxlut",
        "localName": "pqcrystals_dilithium2_ref_idxlut",
        "nameDelimiter": "::",
        "name": "pqcrystals_dilithium2_ref_idxlut"
      }
    },
    {
      "id": "ffffffff-ad71-abff-ffff-ffffcb38da74",
      "labels": [
        "Node",
        "Declaration",
        "ValueDeclaration",
        "VariableDeclaration"
      ],
      "properties": {
        "isArray": false,
        "isGlobal": true,
        "isImplicitInitializerAllowed": false,
        "argumentIndex": 0,
        "isImplicit": false,
        "isInferred": true,
        "fullName": "pqcrystals_dilithium2_ref_idxlut",
        "localName": "pqcrystals_dilithium2_ref_idxlut",
        "nameDelimiter": "::",
        "name": "pqcrystals_dilithium2_ref_idxlut"
      }
    },
[...]

codecov · 2025-07-17T07:34:43Z

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
851	2	849	12

View the top 2 failed test(s) by shortest run time

de.fraunhofer.aisec.cpg.IntegrationTest::testExportToJson()

Stack Traces | 0.058s run time

java.lang.NullPointerException
	at de.fraunhofer.aisec.cpg.persistence.JsonKt.createJsonGraph(Json.kt:69)
	at de.fraunhofer.aisec.cpg.persistence.JsonKt.persistJson(Json.kt:113)
	at de.fraunhofer.aisec.cpg.IntegrationTest.testExportToJson(IntegrationTest.kt:80)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)

de.fraunhofer.aisec.cpg.IntegrationTest::testBuildJsonGraph()

Stack Traces | 11.3s run time

java.lang.NullPointerException
	at de.fraunhofer.aisec.cpg.persistence.JsonKt.createJsonGraph(Json.kt:69)
	at de.fraunhofer.aisec.cpg.IntegrationTest.testBuildJsonGraph(IntegrationTest.kt:51)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

oxisto

Nice! the implementation even looks simpler than before. Next step would be to use the same functions in the schema export thingie

oxisto · 2025-07-18T11:01:55Z

cpg-neo4j/src/main/kotlin/de/fraunhofer/aisec/cpg/persistence/Json.kt

@@ -0,0 +1,116 @@
+/*


I think we can even move this file to cog-core? Should not have any dependencies on neo4j anymore

yes, good point, moved it to core

oxisto · 2025-07-18T11:02:13Z

cpg-neo4j/src/main/kotlin/de/fraunhofer/aisec/cpg/persistence/Json.kt

+import de.fraunhofer.aisec.cpg.helpers.Benchmark
+import java.io.File
+
+data class JsonNode(val id: String, val labels: Set<String>, val properties: Map<String, Any?>)


some doc here would be nice

oxisto · 2025-07-18T11:02:17Z

cpg-neo4j/src/main/kotlin/de/fraunhofer/aisec/cpg/persistence/Json.kt

+
+data class JsonNode(val id: String, val labels: Set<String>, val properties: Map<String, Any?>)
+
+data class JsonEdge(


some doc here would be nice

oxisto · 2025-07-18T11:02:21Z

cpg-neo4j/src/main/kotlin/de/fraunhofer/aisec/cpg/persistence/Json.kt

+    val properties: Map<String, Any?>,
+)
+
+data class JsonGraph(val nodes: List<JsonNode>, val edges: List<JsonEdge>)


some doc here would be nice

peckto · 2025-08-07T07:03:12Z

I noticed, the duplicate node UUID problem is not specific to inferred nodes.
Here is another example, the node is in the same file, but in a different location:

    [
        {
            "id": "ffffffff-b571-ae40-ffff-ffff8b131677",
            "labels": [
                "Node",
                "AstNode",
                "Statement",
                "Expression",
                "Reference"
            ],
            "properties": {
                "access": "READ",
                "isStaticAccess": false,
                "referenceTag": -1839066876,
                "argumentIndex": 0,
                "code": "a",
                "isImplicit": false,
                "isInferred": false,
                "artifact": "file:///dilithium/ref/poly.c",
                "endLine": 253,
                "endColumn": 30,
                "startColumn": 29,
                "startLine": 253,
                "fullName": "a",
                "localName": "a",
                "nameDelimiter": "::",
                "name": "a"
            }
        },
        {
            "id": "ffffffff-b571-ae40-ffff-ffff8b131677",
            "labels": [
                "Node",
                "AstNode",
                "Statement",
                "Expression",
                "Reference"
            ],
            "properties": {
                "access": "READ",
                "isStaticAccess": false,
                "referenceTag": -1497655690,
                "argumentIndex": 0,
                "code": "a",
                "isImplicit": false,
                "isInferred": false,
                "artifact": "file:///dilithium/ref/poly.c",
                "endLine": 625,
                "endColumn": 18,
                "startColumn": 17,
                "startLine": 625,
                "fullName": "a",
                "localName": "a",
                "nameDelimiter": "::",
                "name": "a"
            }
        }
    ]

peckto · 2025-08-07T07:07:27Z

cpg-core/src/integrationTest/kotlin/de/fraunhofer/aisec/cpg/IntegrationTest.kt

-import picocli.CommandLine

-fun createTranslationResult(file: String = "client.cpp"): Pair<Application, TranslationResult> {
+fun createTranslationResult(file: String = "client.cpp"): TranslationResult {


No being in the cpg-core we probably cannot access the cxx language frontend anymore ^^
Is there an alternative way to create a graph that we can export for the test-case?

peckto force-pushed the json-export-new-ogm branch from 0ea2193 to bb2eefa Compare July 17, 2025 07:31

oxisto approved these changes Jul 18, 2025

View reviewed changes

oxisto mentioned this pull request Jul 23, 2025

Self-Loop EOG Edge on C++ Default Argument Values #2395

Open

Use the new OGM to export cpg as json

3c93f63

peckto force-pushed the json-export-new-ogm branch from bb2eefa to 176b407 Compare August 7, 2025 06:28

peckto requested review from KuechA and konradweiss as code owners August 7, 2025 06:28

peckto force-pushed the json-export-new-ogm branch 2 times, most recently from e36f3a4 to a5f56cb Compare August 7, 2025 06:42

peckto commented Aug 7, 2025

View reviewed changes

peckto force-pushed the json-export-new-ogm branch from 2454689 to 13d76da Compare August 7, 2025 09:13

peckto added 4 commits August 7, 2025 13:01

move json persistence into cpg-core

413587d

use GraphExamples for integration test

30a859a

move GraphExamples into testFixtures

aa63201

use legacyId for now, as long as the UUIDs are not sufficiently unique

464b4be

peckto force-pushed the json-export-new-ogm branch from 13d76da to 464b4be Compare August 7, 2025 11:01


		data class JsonNode(val id: String, val labels: Set<String>, val properties: Map<String, Any?>)

		data class JsonEdge(

Use the new OGM to export cpg as json #2392

Are you sure you want to change the base?

Use the new OGM to export cpg as json #2392

Uh oh!

Conversation

peckto commented Jul 17, 2025

Uh oh!

codecov bot commented Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 2 Tests Failed:

Uh oh!

oxisto left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peckto commented Aug 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov bot commented Jul 17, 2025 •

edited

Loading