R04J32 2025-09-02

R04J32 Merge to Master (#95)

* implement user-defined adj site mapping to allow fine grained control over neighboring site announcements;

* BUGFIX: patch issue where the FNE would not maintain the source peer ID for certain routing scenarios;

* fix typo in original implementation, peerId should be the peerId and ssrc should become the originating peer ID; remove peer check throwing a warning;

* log ssrc for various data points (the ssrc is the RTP originating sync source which would be the peer ID of the origination of a RTP packet);

* bump build number;

* add peer ID masking support (this is useful for FNEs that are "public" links where internal peer IDs don't need to be retained); more work for maintaining the originating stream sync source; refactor how the FNE handles peer network connections;

* throw a warning if the user configured more then 8 upstream peer connections (more than this number can cause performance problems);

* make non-peer-link peer ID masking optional;

* add checking for traffic repeat to verify we are not trying to send traffic back to the source;

* add checking for traffic repeat to verify we are not trying to send traffic back to an external source;

* convert peer network protocol packet processing to a threaded model;

* update copyright dates;

* implement REST API for manipulating adjacent site map entries;

* fix missed REST API initializer, need to pass the adj site map;

* add DFSI/V.24 full duplex option (this allows dvmhost to repeat incoming frames back out);

* Fix incorrect check of DFSI and FSC operating modes that resulted in a segfault (#92)

Co-authored-by: faulty <[email protected]>

* document p25CorrCount in depth more; set the default p25CorrCount to 2;

* properly call PacketBuffers clear() before leaving a scope to ensure contained buffers are deleted (otherwise we'll leak memory);

* skip the first 6 bits of the TIA-102 DFSI VHDR (this is very strange, and is probably not kosher);

* transparently pass voice frames with FID $20 (assumed to be Kenwood) when the PF flag is set (this allows Kenwoods flavor of encryption to pass);

* relabel FID_DMRA to FID_MOT (this feature ID is assigned to Motorola); handle more conditions for FID $20 (Kenwood) on voice frames;

* remove filterHeaders (this is deprecated, HDUs aren't sent over the network); perform frame length validation for P25 network frames (unlike DMR and NXDN P25 frames are variable length, requring length validation to prevent buffer under or overflows); fix issue with addr variable not being freed in InfluxDB handler;

* [EXPERIMENTAL] add very initial support for dvmpatch to talk to a MMDVM P25 Gateway in P25 mode;

* fixup dvmpatch support for talking to MMDVM P25 Gateway;

* fix up handling of MMDVM call termination;

* more work on better signalling end of call for MMDVM P25 gateway patches; better log traffic from MMDVM;

* correct some incorrect timing;

* add support for properly authorizing a peer to send Inhibit/Uninhibit commands; add some NXDN constants for remote control;

* add columns to main peered peer list to represent if a peer is allowed to inhibit;

* fixup informational logging;

* add auto generation of a peer password; slightly increase the dialog size of the peer edit window;

* fix TIA VHDR incorrect length; correct gatekeep accidentally setting TIA StartOfStream to 4 bytes instead of 3, because you know the TIA spec is incomprehensible with its bit alignment;

* add some permanent log trace for DFSI over UDP tracing; correct startOfStreamTIA incorrectly sending a LDU1 NID; correct some offsets;

* make sure we send out heartbeats *before* the heartbeat time;

* send Start of Stream with voice data;

* free memory in error case; remove unused variables;

* correct some buffer allocations; ensure the AES class is deleted after use;

* implement support for legacy radio affiliation (like P25) for DMR and NXDN (this fixes an issue where conventional DMR and NXDN systems would be unable to transmit onto affiliation only TGs); correct bad CSBK decoding in the FNE (forgot to check for dataSync);

* code cleanup to correct compiler warnings;

* add string file meta data info to Win32 EXEs; add icons to Win32 EXEs;

* migrate and condense analog audio helper and G.711 codec routines into a common class called AnalogAudio for easy reuse; stub network protocol entries to eventually handle analog audio;

* add support in the DVM network protocol for analog FM audio traffic; add support to the DVM FNE to properly switch analog FM audio traffic; bump revision from H to J;

* add analog enable flag to fne config YAML;

* prelim work to make dvmbridge pass analog audio directly into FNE network;

* handle locking inside deleteBuffers();

* refactor FrameQueue mutex handling; don't start and stop the thread pool when the peer network opens/closes;

* use unique_ptr for compression class instead of passing raw pointers;

* refactor FrameQueue from using a std::unordered_map to a simple fixed array that is directly controlled;

* refactor compression slightly;

* better handle multi-threaded problems;

* use a std::vector instead of a classical C allocated array for timestamp list; add error/logic checking for V.24/DFSI where the VHDR may send a TGID0 during a call causing a call rejection;

* make timestamps vector static;

* dump out of VHDR processing if call is in progress;

* bump tarball version build stamp;

* add option to forcibly allow TGID0 for badly behaved systems this will cause dvmhost to accept the call and not drop it, with the caveat that the TGID will be rewritten to TGID1, the network stack simply cannot service a TGID of 0 and as such we must rewrite the TGID a TIA-102 valid TGID;

* simplify late entry handling where V.24 may send a dstId of 0;

* whoops correct my own stupidity;

* whoops correct my own stupidity (again);

* add opcodes for Harris User Alias and GPS; add opcode for Motorola GPS on PTT; decode Harris User Alias in LDU LCs;

* attempt at decoding the LDU RS values for both V.24 and TIA DFSI, this does not abort decoding in the ModemV24 right now and will simply log an error if the RS values for the LDU1 or LDU2 are to badly invalid;

* alter field data based on recent testing; enhance and update debug log entries; add debug logging for Voice 1 and 10 start of voice frames for V.24;

* add missing tag information for V.24 PDU;

* fix incorrect handling of RFSS ID in host setup;

* experimental fix to ignore wildly illegal TGIDs in the HDU;

* simplify the V.24 HDU TGID logic, entirely ignore the TGID presented in the HDU and just wait for the LDU1 to set the rfLastDstId;

* add grantDemand documentation to dvmpatch;

* [EXPERIMENTAL] add experimental support for cross-encrypting patched P25 traffic;

* reset call algo when a P25 call ends;

* properly handle algo ID's for source and destinations;

* missed reverseCrypto check;

* [EXPERIMENTAL] for some configurations (**you know who you are god damn it**), allow dvmhost to be configured to "idle" on a non-zero TG when the RF talkgroup hangtimer stops, this effectively disables the promiscuous network Rx of the host for certain conventional operations, and requires RF-based TG steering to change the active talkgroup; cleanup the macros for DMR and NXDN that perform various repeated checks;

* major refactor for V.24 support;

* hide RSSI2 ICW errors; report on any voice frames reporting more then 0 errors;

* properly sent VHDR1 and VHDR2 with proper opcodes (whoops);

* add experimental support for TDULC over V.24; add some documentation for the V.24 and TIA voice header byte layout;

* experimental support for transporting V.24 over IP;

* initial super frame should start at 1; make sure to use proper constants instead of magic numbers;

* reorganize code slightly;

* cleanup format for, and make slightly more precise trace and error dumping log messaging;

* add option for the FNE to directly log traffic denials to the system log;

* centralize string for illegal RID access;

* log total voice frame errors for TIA/DFSI mode;

* minor PDU refactoring on when network PDU data is sent; correct issue with accepting a conventional data reg;

* disable allowExplicitSourceId by default (not all subscribers support this); correct some bad handling of LC data;

* log sysId and netId info for call start and end on the FNE log;

* add setting netId and sysId to dvmbridge and dvmpatch for future use;

* reorganize and expose decodeLC/encodeLC from LC; correct typo in TDULC headers; add explicit source id TDULC; add LC_GROUP explicit ID flag;

* use shorthand macros where able;

* enable or disable explicit source ID directly, don't require control to be enabled (important for trunked VC);

* minor correction to handling explicit source ID;

* properly set dummy SiteData for dvmbridge and dvmpatch (necessary to set the WACN and SysId);

* handle condition defaulting WACN and SysId if network data for WACN and SysId is 0;

* ensure unencrypted parameters if encryption is disabled;

* copy dvmbridge change to dvmpatch for: ensure unencrypted parameters if encryption is disabled;

* ignore extra ICW opcodes;

* use raw LDU values from the call data instead of processed values if the MFId on the LC isn't TIA standard;

* collate bit errors reported from V.24 or TIA;

* typos;

* allow a group affiliation on Conv+ (DVRS) to terminate a running TG hang timer;

* add missing stop to the timer;

* add WACN/SysId logging to SysView;

* add global to disable transmitting logging to the network (this is used when the UDP socket fails to send data to prevent a crash condition talking to a null socket); unify errno to string message processing for clearity in logs; refactor network reconnect and retry logic to better handle a full connection reset cycle (this is useful for conditions where the network loss isn't transient and is something like a Ethernet or WiFi link drop causing the entire interface to become down temporarily);

* remove unnecessary debug logging;

* allow encoding user alias at the LC level;

* see if we can calculate error percentages;

* get rid of magic numbers for properly defined constants;

* implement proper support for P25 LDU1 and LDU2 to pass call control bytes; allow a end-point to signal that a call is handing over/switching over from one stream ID/source ID to another; implement support on dvmbridge to properly handle stream/source ID switch over;

* log call source switch over events;

* whoops this should be a OR equals not equals;

* correct issue where network frames would be ignored during RF calls causing buffer overflows;

* cleanup and remove unnecessary and confusing C compiler macros;

* cleanup and unify design a bit, for P25 move traffic collision checking into separate function calls;

* add colorize-host.sh helper tool; update colorize-fne.sh tool for DMR and NXDN;

* store status data for a private call;

* during unit registration store the originating source peer ID for a given unit registration; use unit registration source ID to select the destination peer to send private call data to;

* initial experimental implementation of private call routing:
	this introduces a new configuration flag "restrictPrivateCallToRegOnly", when set, this flag will influence how private calls are routed through the system
	private calls using restrictPrivateCallToRegOnly require both the source and destination subscribers to be unit registered with the system, when a private call
	occurs the system will utilize the source peer ID to restrict repeating the private call traffic only to the 2 peers involved in the private call

	FNEs will *always* receive private call traffic

* rework stream validation for U2U calls slightly for P25; ensure P25 always sends the U2U control byte for private calls;

* enhance handling of NXDN LC on the FNE; fix issue where dvmhost would not process a network RCCH burst;

* refactor NXDN CC handling a bit;

* add support to translate a raw DENY/QUEUE/CAUSE/REASON value for DMR/P25/NXDN into a human readable string;

* whoops hastily missed std::string -> c_str conversion;

* ensure SSRC is maintained for unit registration announcements;

* more implementation for private call routing;

* add missing clear flag to DMR payload activate RPC;

* correct crash for fsc set to true when not using DFSI TIA-102/UDP;

* add support to disable a failed CRC-32 for P25 PDU data; ignore CRC-32 errors for AMBT PDUs;

* Add support in dvmbridge for a serial PTT activation switch.  RTS is asserted on the serial port defined in bridge-config.yml for the duration of audio received, then is removed. (#102)

* P25 PDU packet handling refactor (for future use);

* remove test code;

* fix issue where PDU RSPs weren't being sent to the FNE; correct timing around ARP and packet retry when subscriber is not ready;

* add some more verbose logging;

* experimental changes to PDU data handling;

* correct CSV parsing for iden, peer list and RID lookup tables (we would skip parameters if they were empty); make the FNE P25 packet data handler operate in the TIA-102 asymmetric addressing mode (as would be required with FNE configurations);

---------

Co-authored-by: Jamie <[email protected]>
Co-authored-by: faulty <[email protected]>
Co-authored-by: Lorenzo L. Romero <[email protected]>

R04H31 2025-05-25

R04H31 Merge to Master (#89)

* simplify and refactor peer-link packet handling;

* bump version number to match dev branch; alter logging for peer-link data;

* warn on packets generated over 4K;

* don't clear map directly, use clear() helper;

* use parens vs braces for initializers; correct case fall-thru on SysView;

* for clarity relabel RPT_CLOSING and MST_CLOSING to RPT_DISC and MST_DISC;

* discontinue support for TRANSFER command on the traffic port entirely; hide useAlternatePortForDiagnostics and allowActivityTransfer options in FNE configuration; add stern warnings and alerts if the FNE disabled either useAlternatePortForDiagnostics or allowActivityTransfer (these are really critical operations); alert in peer log if the master does not report support for the alternate port for diagnostics;

* make oversized packet warning clearer (this warning is not a end-user warning, the user can't do antyhing about this this is a developer BUGBUG warning);

* apply recv timeout on InfluxDB operations (this hopefully will prevent stuck queries if the InfluxDB server dies while query is in progress);

* prevent InfluxDB query from becoming stuck in a tight infinite loop within select();

* don't bother handling responses from the influx query, remove fluxQL (we don't make queries with this library);

* cleanup unused vars;

* generate an error log if a fluxql worker fails to write data to influx;

* implement support to save/retain configuration pushed from peer link connections;

* fix memory leak caused by not deleting unused packetbuffer;

* refactor ACL updates;

* refactor log messages for channels to be <channel ID>-<channel No> in most cases where applicable;

* add more <channel ID>-<channel No> changes;

* whoops wrong variable type;

* correct fatal bug where update lookup timer was never started;

* check the RF state at the bottom of a talkgroup hang expiration timer for RF talkgroup activity, if the state is not listening, handle the end of call as if it was a frame loss;

* fix potential memory leak in PacketBuffer fragmenter, the encoder was not deleting the storage buffer before end of scope;

* ensure decompressed buffer is deleted;

* ensure decompressed buffer is deleted in error conditions too;

* add support to disable adjacent site broadcast;

* lay the groundwork for explicit channel configuration;

* [EXPERIMENTAL] implement experimental support for a CC to define explicit channel identity information for voice channels (NOTE: this does *NOT* implement support for hotspots to operate in explicit channel mode!);

* don't use the concurrent::unordered_map for this (this gives me great pause for a problem...);

* add some extra debug trace;

* add some extra debug trace;

* display MI data during calibration; update modem submodule;

* this is ugly, and I hate it, but to fix very strange WIN32 deadlock situations, we'll selectively use our concurrent unordered_map on non-WIN32, and on WIN32 use the std unordered_map + a local mutex;

* update modem submodule; change P25 Voice MI logging messages from DEBUG to MESSAGE; fix issue in HostSetup/HostCal where PDU header decoding was broken;

* only dump MI data if the call is encrypted (don't bother displaying zeros for a clear call);

* whoops check right variable;

* cleanup and clarify MI logging;

* cleanup and clarify MI logging (again LDU2 was out of log sequence);

* hate me later, dont use a colon as the string separator for the MI bytes, use equals;

* log DMR PI MI dta;

* refactor debug messaging for HDU;

* reorganize core meta-programming macros, move class property macros into ClassProperties.h, move bit manipulation into BitManipulation.h, move VLA into VariableLengthArray.h; rename meta-programming macros for clarity most of these macros declare some sort of variable and/or functions, as such I'm dropping the notation for __BLAHBLAH and using DECLARE_BLAHBLAH instead; better document class property meta-programming macros; refactor __GET/__SET_UINT16 into __GET/__SET_UINT24 these macros set 24-bit values from and to buffers, this will reduce programming confusion;

* add property documentation for some KMM classes;

* correct file copyright headers;

* change commenting to reflect section more clearly;

* use proper GPL license for file;

* detect arch and properly update internal variable;

* handle instance where the system does not return a valid processor type for arch detection;

* display target arch along side host arch if we're cross compiling;

* strip newlines;

* fix missing parens around response encoding;

R04H30 2025-05-01

R04H30 Merge to Master (#88)

* implement support for LC_GROUP_UPDT (this includes the RPC between the CC and VC's to announce active TG lists); completely and entirely refactor how packet handling threads are done, use a new method introduced for this, thread pool resources. this will ultimately be more resource consuming depending on configuration as the worker threads for packet processing stay alive along side the main process. this should be more performant (because we're not constantly creating and destroying threads) and will prevent error conditions that can cause an extreme number of threads to spawn;

* reduce influx and V.24 thread pool sizes; add calculation of how long between when a packet was Rx to when it began proper processing;

* don't use void* for the task routines;

* rename ThreadPoolCallback to ThreadPoolTask; add some checking around task validness; correct a valgrind issue with RawFrameQueue() write not deleting the buffer before return;

* valgrind cleanups;

* gate active TG from CC to VC updates at 5s (prevent API spam);

* update package version;

* don't attempt to send active TG updates to 0.0.0.0;

* make notification of active TGs CC -> VC optional;

* correct ThreadPool issue on Win32; split UDP PCM audio processing into its own thread; implement user control of inter-audio frame delay and jitter buffer (if using inter-audio frame delay);

* lock queue;

* add custom classes for STL containers that support mutex locking for thread safe operation; modify ChannelLookup and AffiliationLookup to use concurrent containers; modify FNE to use concurrent containers for internal lists;

* remove mutex used for protecting udp packet deque;

* more concurrency solidification;

* further concurrency class usage; bump version number to R04H30;

* add --boot commandline argument to reboot modem into bootloader without any interactive interface;

* cleanup program -h usage display;

* incorrect opcode define;

* update README.md;

* correct incorrect string format for non-useAlternatePortForDiagnostics;

* update README.md;

* simplify influxdb worker task function;

* refactor PL_ACT_PEER_LIST opcode entirely, use zlib and compress list sent and properly block data sent;

* don't waste cycles on building the peer list repeatedly, build it once for the cycle;

* deduplicate compress/decompress code into a static C++ class;

* stylecop file formatting;

* add table locking and remove at find;

* deduplicate implementation;

* cleanup unused label; change FrameQueue's unordered_map to a concurrent one; fix incorrect setting of __lock() for concurrent containers;

* add more timestream map locking;

* fix incorrect behavior when deriving initial timestamp for a call stream; correct incorrect behavior inserting new stream in to timestamps table for frame queue; fix incorrect behavior deriving timestamp for bridge RTP;

* fix issue with naive approach to handling PL_ACT_PEER_LIST data fragementation;

* disable accidental debug code;

* instead of asserting, throw a log error message and discard network packet;

* lock the talkgroup tables when a find is in progress (this will prevent some weird concurrency behaviors because talkgroup rules does not use the concurrent vector);

* if filter headers or terminators is enabled, and the target peer is in the exclusion list, do not send headers or terminators;

* exclusion check should happen before the rewrite check;

* implement dvmpatch, this is a new utility that allows simple TG to TG patching;

* ensure FNE downstream peers that report as peer link have implicit always rules applied to them (i.e. they will *always* receive *all* traffic); correct order of operations when deleting a peer entry (delete connection *AFTER* removing from peers table);

* document concern over possible null ref concurrency issue;

* don't be overly aggressive with FNE process niceness, nice of -10 is more then sufficient;

R04G21 Build 2025-04-16

okay now that I've rewritten history, selectively for amd64 we'll bui…

Release 2023-09-08

fix wrong variable used to populate MI data for encryption for networ…

Release 2023-09-05

add support to grant demand to support denying a grant and supporting…