Help with Correlation Mappings

[danieldonoghue]

I would like to be able to see all related calls flows when I try to view the flow for any call legs.

In my setup, I have outgoing calls, comprising of 2 legs; these calls have a P-Routing-Params header which consists of. series of key=value Pairs. I am using the id field 's value as a correlation id to link the calls legs together.

Additionally, I have incoming calls that trigger an outgoing call; in this scenario, I use the X-CID header for correlation.

I have adjusted my heplify-server configuration to extract the correlation id, and it does this successfully. I can see the correlation id and search by it within the homer UI.

AlegIDs               = ["X-CID","P-Routing-Params,id=(.*)$"]

(the regex is ok here as the id field is always the final property in the header)

Despite the correlation ID being extracted properly, I am, so far, unable to have all the related calls in a single flow diagram. I have tried configuring the field mappings using various configurations that I've seen suggestd online. The current looks like...

[
    {
        "source_field": "data_header.call_id",
        "lookup_id": 1,
        "lookup_profile": "call",
        "append_sid": true,
        "lookup_field": "protocol_header->>'correlation_id'",
        "lookup_range": [
            -300,
            200
        ]
    },
    {
        "source_field": "protocol_header.correlation_id",
        "lookup_id": 1,
        "lookup_profile": "call",
        "append_sid": true,
        "lookup_field": "protocol_header->>'correlation_id'",
        "lookup_range": [
            -300,
            200
        ]
    },
    {
        "source_field": "protocol_header.correlation_id",
        "lookup_id": 1,
        "lookup_profile": "call",
        "append_sid": true,
        "lookup_field": "data_header->>'callid'",
        "lookup_range": [
            -300,
            200
        ]
    },
    {
        "source_field": "data_header.callid",
        "lookup_id": 1,
        "lookup_profile": "call",
        "lookup_field": "sid",
        "lookup_range": [
            -300,
            200
        ]
    }
]

The correlation ID coming from the P-Routing-Params header is not a call-id but is the same in all the calls that are related.

I should also mention that it is only the initial incoming INVITE request which contains the relevant correlation fields. and I would like to see all the requests, responses , logs and RTCP reports related to the calls in question.

Some of the resources I've looked at to try and get this working correctly include..

• How do mappings work? #278
• https://github.com/sipcapture/homer/wiki/Custom-Header-Searching-and-Correlation
• https://groups.google.com/g/homer-discuss/c/_XoRM6jQeRk

I would be grateful for some assistance here :)

[hamidrhashmi]

Hi @danieldonoghue

are the following headers in the INVITE packet?

• X-CID
• P-Routing-Params
• id=(.*)$

Can you see them as extracted when you click on the INVITE and go to the details tab ? as shown here as first step ?

If not then please first try with X-CID and P-Routing-Prams only and confirm if you can see those headers as extracted.

[danieldonoghue]

hi @hamidrhashmi

thanks for the suggestion. I followed the resources you showed there and can confirm that the fields were indeed shown in the details tab.

I have simplified my setup somewhat for the purposes of getting something working. I have temporarily dropped the X-CID requirement and am focussing on the P-Routing-Params header.

I am using lua in heplify-server to create a new sip header (X-Group-Id) that contains the id from the P-Routing-Params header to make it easier to handle in the web app.

in the web app, I now have something like this for my mappings...

[
    {
        "source_field": "data_header.callid",
        "lookup_id": 1,
        "lookup_profile": "call",
        "lookup_field": "sid",
        "lookup_range": [
            -300,
            200
        ]
    },
    {
        "source_field": "data_header.X-Group-Id",
        "lookup_id": 1,
        "lookup_profile": "call",
        "lookup_field": "sid",
        "lookup_range": [
            -300,
            200
        ]
    },
    {
        "source_field": "data_header.callid",
        "lookup_id": 1,
        "lookup_profile": "call",
        "lookup_field": "data_header->>'callid'",
        "lookup_range": [
            -300,
            200
        ]
    },
    {
        "source_field": "data_header.X-Group-Id",
        "lookup_id": 1,
        "lookup_profile": "call",
        "append_sid": true,
        "lookup_field": "data_header->>'X-Group-Id'",
        "lookup_range": [
            -300,
            200
        ]
    },
    {
        "source_field": "data_header.callid",
        "lookup_id": 1,
        "lookup_profile": "call",
        "append_sid": true,
        "lookup_field": "data_header->>'X-Group-Id'",
        "lookup_range": [
            -300,
            200
        ]
    },
    {
        "source_field": "data_header.X-Group-Id",
        "lookup_id": 1,
        "lookup_profile": "call",
        "append_sid": true,
        "lookup_field": "data_header->>'callid'",
        "lookup_range": [
            -300,
            200
        ]
    }
]

with this setup, I can search, specifically, for the X-Group-Id (I've also added that as a field) and find all the INVITEs and I also now actually see the other INVITE in my call flow diagrams, which is a step forward.

However, I only see the INVITE for the additional call and not the rest of the call's requests. To solve this, I tried using the append_sid option but it doesn't to solve my problem (no difference in the call flow). In fact, if I add append_sid to this block, then I note that I stop seeing the additional INVITE in the call flow altogether...

{
        "source_field": "data_header.callid",
        "lookup_id": 1,
        "lookup_profile": "call",
        "lookup_field": "data_header->>'callid'",
        "lookup_range": [
            -300,
            200
        ]
}

[danieldonoghue]

I have simplified my mapping further so it looks like this...

[
    {
        "source_field": "data_header.callid",
        "lookup_id": 1,
        "lookup_profile": "call",
        "lookup_field": "sid",
        "lookup_range": [
            -300,
            200
        ]
    },
    {
        "source_field": "data_header.X-Group-Id",
        "lookup_id": 1,
        "lookup_profile": "call",
        "append_sid": true,
        "lookup_field": "data_header->>'X-Group-Id'",
        "lookup_range": [
            -300,
            200
        ]
    },
    {
        "source_field": "data_header.callid",
        "lookup_id": 1,
        "lookup_profile": "call",
        "append_sid": true,
        "lookup_field": "data_header->>'X-Group-Id'",
        "lookup_range": [
            -300,
            200
        ]
    },
    {
        "source_field": "data_header.X-Group-Id",
        "lookup_id": 1,
        "lookup_profile": "call",
        "append_sid": true,
        "lookup_field": "data_header->>'callid'",
        "lookup_range": [
            -300,
            200
        ]
    }
]

no change though; still see the INVITE from the additional legs but not all the requests.

I feel I should reiterate, too, that X-Group-Id doesnt contain a Call ID. it's an internal reference, linked to one or more call legs.

[danieldonoghue]

just looked at the logs for homer-app and noticed the following occurred when I try to view the call flow, with the mappings presented in my previous comment..

time="2025-02-12T11:37:27Z" level=info msg="pq: got 5 parameters but the statement requires 3" module=gorm type=log
time="2025-02-12T11:37:27Z" level=info msg="SELECT * FROM \"hep_proto_1_call\"  WHERE (create_date between $1 AND $2 AND data_header->>'X-Group-Id' in ($3))" module=gorm type=sql
{"level":"error","msg":"GetTransactionData: We have got error: pq: got 5 parameters but the statement requires 3","time":"2025-02-12T11:37:27Z"}
{"level":"debug","msg":"GetTransactionData: Len: 0","time":"2025-02-12T11:37:27Z"}
{"level":"debug","msg":"Correlation data len:20","time":"2025-02-12T11:37:27Z"}
{"level":"debug","msg":"ISOLATEGROUP ","time":"2025-02-12T11:37:27Z"}
{"level":"debug","msg":"USERGROUP admin","time":"2025-02-12T11:37:27Z"}
time="2025-02-12T11:37:27Z" level=info msg="pq: got 5 parameters but the statement requires 3" module=gorm type=log
time="2025-02-12T11:37:27Z" level=info msg="SELECT * FROM \"hep_proto_1_call\"  WHERE (create_date between $1 AND $2 AND data_header->>'callid' in ($3))" module=gorm type=sql
{"level":"error","msg":"GetTransactionData: We have got error: pq: got 5 parameters but the statement requires 3","time":"2025-02-12T11:37:27Z"}
{"level":"debug","msg":"GetTransactionData: Len: 0","time":"2025-02-12T11:37:27Z"}
{"level":"debug","msg":"Correlation data len:20","time":"2025-02-12T11:37:27Z"}

not sure whether there's any relevance to the problem?

[danieldonoghue]

I suspect this is a bug in the DB query logic in GetTransactionData in service/search.go.

I think the following block:

	if likeSearch {
		query += "AND " + fieldKey + " LIKE ANY(ARRAY[?])"
	} else {
		query += "AND " + fieldKey + " in (?)"
	}

should be changed to:

	if likeSearch {
		query += "AND " + fieldKey + " LIKE ANY(ARRAY[?])"
	} else {
		query += "AND " + fieldKey + " = ANY(?)"
	}

[danieldonoghue]

PR: #596

[nttranbao]

Hi,

For some reasons, on latest master branch as of today, when selecting more than 1 Call-ID in the GUI, the search still returns blank with "No data in Database"... and the logs say...

INFO[0069] pq: got 4 parameters but the statement requires 3 module=gorm type=log

UPDATE: when double-clicking on 1 selected Call-ID, then webgui is showing the call diagram... The above log line is still generated though...
The issue is with 2 or more Call-IDs selected, then Webapp shows blank page, with the above log ling printed...

[adubovikov]

@nttranbao it has been fixed in 1.5.8