Skip to content

Client library

Jump to bottom
remiges-shuvam edited this page Mar 11, 2024 · 28 revisions

LogHarbour provides client libraries, in Java and Go, to allow business applications to perform all operations on a LogHarbour instance. The documentation below specifies the function signatures for the Go version. The semantics are identical in the two versions.

All functions require a token. A token is tied to a realm, and is generated by mechanisms outside the client library. See the page on access controls in LogHarbour.

Writing log entries

These functions are needed to write log entries into the various logs of LogHarbour.

The Go library for writing log entries already exists. Check https://pkg.go.dev/github.com/remiges-tech/logharbour/logharbour

Log retrieval

GetLogs()

This function is the general purpose function to fetch log entries from the activity logs and data-change logs in the repository.

See https://pkg.go.dev/github.com/remiges-tech/logharbour/logharbour#LogEntry for the data structure definition for a single log entry.

func LogHarbour.GetLogs(querytoken, app, type, who, class, instance, op string,
                                fromts, tots time.Time, ndays int,
                                remoteIP string, pri LogPri_t,
                                searchAfterTS time.Time, searchAfterDocID string)
                               (logs []LogEntry, nrec int, err error)

where

  • querytoken: mandatory, is the query token of the realm
  • app: optional, to extract log entries for just the app specified
  • type: optional, A meaning activity logs, C meaning data-change logs, D meaning debug logs, and omitted meaning all three
  • who: optional, to extract log entries inserted only by actions performed by the specified user
  • class: optional, to extract logs related to objects of the specified class only
  • instance: optional, to extract logs only for a specific object instance. This parameter must be nil if class is not specified.
  • op: optional, extract log entries which carry this specific value in their op field
  • fromts, tots: optional, timestamps, to extract log entries whose when value falls in this time range.
  • ndays: optional, an integer specifying how many days back in time the retrieval must attempt, counting backwards in time from the current date
  • remoteIP: optional, an IP address in string form specifying the remote IP from where the operation was triggered which generated the log entry
  • pri: optional, specifies that only logs of priority equal to or higher than the value given here will be returned. If this parameter is present in the call, then data-change log entries are omitted from the result, because those log entries have no priority.
  • searchAfterTS: optional. If specified, it indicates that only logs with timestamps older than the specified parameter must be retrieved
  • searchAfterDocID: optional. If specified, it indicates that only logs whose document IDs are earlier than the one specified must be retrieved.

The three parameters, fromts, tots, and ndays have inter-dependency as follows:

  • If fromts and tots are specified, then tots must be after fromts.
  • If either or both of fromts and tots is specified, then ndays is ignored.
  • If just one of fronts or tots is specified, then the function will return all records which satisfy this filter, up to a maximum of LOGHARBOUR_GETLOGS_MAXREC entries, where the value of this parameter is a global configuration entry. If fromts is specified, then entries will be returned sorted forward in time (i.e. oldest to newest) based on their when field. If tots is specified then entries will be sorted from newest to oldest.

Under all circumstances, the response carries an array of up to LOGHARBOUR_GETLOGS_MAXREC entries. Each entry has fields as mentioned in the annotated data structure definition above.

The searchAfterTS and searchAfterDocID parameters must have the value nil the first time this function is called with a specific set of parameters. This call returns at most LOGHARBOUR_GETLOGS_MAXREC entries. The actual number of entries matching this request is returned in nrecs. If this number exceeds the value of LOGHARBOUR_GETLOGS_MAXREC, then the function may be called again with exactly the same set of parameters but with specific values for searchAfterTS and searchAfterDocID. This makes the function skip the earlier block(s) of entries and fetch only the next set. This way, the same function can retrieve a large data set over multiple calls, getting up to LOGHARBOUR_GETLOGS_MAXREC entries in every call.

The LogEntry data type has a field called data, which is of type any. When a log entry is retrieved, the type of the data field will always be string, and it will hold a JSON data structure. The structure of this JSON block is fixed for data-change log entries and for debug log entries. For data-change log JSON blocks, the utility function JSONtoDataChg() has been provided. The data structure which gets loaded is ChangeInfo (see https://pkg.go.dev/github.com/remiges-tech/logharbour/logharbour#ChangeInfo) which refers to ChangeDetail For debug JSON blocks, the JSONtoDebug() function has been provided. The data structure which gets loaded after the unmarshalling is DebugInfo (see https://pkg.go.dev/github.com/remiges-tech/logharbour/logharbour#DebugInfo) For activity log entries, the data block will be mapped to a simple hashmap of map[string]string, no separate utility function is being provided.

GetSet()

This function gets a set of values for an attribute from the log entries specified. This is a faceted search for one attribute.

func LogHarbour.GetSet(querytoken, app, type, who, class, instance, op string,
                       fromts, tots time.Time, ndays int,
                       remoteIP string, pri LogPri_t, setattr string)
                      (setvalues map[string]int, err error)

where

  • querytoken: mandatory, is the query token of the realm
  • app: optional, to extract log entries for just the app specified
  • type: optional, A meaning activity logs, C meaning data-change logs, D meaning debug logs, and omitted meaning all three
  • who: optional, to extract log entries inserted only by actions performed by the specified user
  • class: optional, to extract logs related to objects of the specified class only
  • instance: optional, to extract logs only for a specific object instance. This parameter must be nil if class is not specified.
  • op: optional, extract log entries which carry this specific value in their op field
  • fromts, tots: optional, timestamps, to extract log entries whose when value falls in this time range.
  • ndays: optional, an integer specifying how many days back in time the retrieval must attempt, counting backwards in time from the current date
  • remoteIP: optional, an IP address in string form specifying the remote IP from where the operation was triggered which generated the log entry
  • pri: optional, specifies that only logs of priority equal to or higher than the value given here will be returned. If this parameter is present in the call, then data-change log entries are omitted from the result, because those log entries have no priority.
  • setattr: mandatory, the name of the attribute whose values are requested in the form of a set. The attribute named can only be one of those which have discrete values, i.e. they are conceptually enumerated types. There is no point attempting to perform this operation on an attribute whose values are continuously variable, e.g. when or message. It makes sense only calling this operation for attributes which have a finite number of discrete values, e.g. pri, op, instance, app, type, status, remoteIP, etc. The function will internally do a sanity check that only one of these attributes has been named, and if not, will return an error.

The three parameters, fromts, tots, and ndays have inter-dependency as follows:

  • If fromts and tots are specified, then tots must be after fromts.
  • If either or both of fromts and tots is specified, then ndays is ignored.
  • If just fromts is specified, it means that the log entries from the given fromts timestamp till the current time will be examined. If only tots is specified, then log entries from the oldest log entry till the given timestamp will be examined.

The function will look through the log entries which match the query parameters, and will pull out the values of the attribute whose name is specified in setattr. It will create a frequency table of occurrences of the values of this attribute, and load these into a hashmap called setvalues where each key will be one unique value of that attribute, and the value of that key will be the count of occurrences of that value in the log entries.

GetApps()

func LogHarbour.GetApps(querytoken) (apps []string, err error)

This function will be used by a business application to retrieve the list of apps under a realm. The response will carry an array of apps. An app is a one-word string, and is an attribute of every log entry. This data will be pulled out from the log entries in the repository, and making a unique set out of them. If there are no log entries in the repository at this time, the array will be of zero length.

This function is a wrapper on top of GetSet().

GetUnusualIP()

This function will go through the logs of the last ndays days which match the search criteria, and pull out all the remote IP addresses which account for a low enough percentage of the total to be treated as unusual or suspicious.

func LogHarbour.GetUnusualIP(querytoken, app, who, class, op string, ndays int, unusualpercent float32) (unusualIPs []string, err error)

where

  • querytoken is the query token for the realm
  • app: optional, the application ID whose matching log entries must be searched
  • who: optional, the user identity whose log entries must be searched
  • class: optional, the class of entities whose log entries must be searched
  • op: optional, the operation which triggered the generation of the log entries which must be searched. This parameter may carry a meaningful value only if class has been specified, and will be ignored if class is nil.
  • ndays: optional, an integer specifying how many days back in time the retrieval must attempt, counting backwards in time from the current date. If this is omitted, then the most recent LOGHARBOUR_GETLOG_MAXREC log entries will be processed
  • unusualpercent: mandatory, specifies the percent upper limit of an IP address' occurrence in the list of remote IPs for it to be considered "unusual". Typical values will range between 0.5 and 5.0. A value above 50 will be rejected and the function will return with an error.
function GetUnusualIP(filtering parameters, ndays, unusualpercent):
  # Aggregate logs by IP and count occurrences
  aggregatedIPs = select remoteIPs of log entries matching the request parameters
  
  # Calculate total number of logs to find what 1% represents
  var PercentThreshold float = count_of(aggregatedIPs) * unusualpercent / 100.0
  if PercentThreshold < 1, then PercentThreshold = 1, endif

  # Initialize an empty list to hold unusual IPs
  unusualIPs = []

  # For each IP in the aggregated IP list
  for each IP in aggregatedIPs do
      # If the count of logs for this IP is less than the threshold
      if IP.count <= onePercentThreshold, then
          if IP.address != "LOCAL" then
              unusualIPs.append(IP.address)
          endif
      endif
  done

  return unusualIPs

This function will internally use GetSet().

The response will return a list of "unusual" IP addresses from all the matching log entries.

GetChanges()

This function reads only the data-change log and gets the changes done on one object instance, or one field of one object, over a time period. The output will be the log entries, with all fields per entry, which record any change in the given object instance (or field of object). Therefore, typically, the oldest log entry returned will be the one where the object instance was created. Subsequent entries, sorted from oldest to newest, will report log entries where the object instance had some values modified. The last entry may (or may not) report a deletion of the object.

func LogHarbour.GetChanges(querytoken, app, who, class, instance, field string,
                       fromts, tots time.Time, ndays int)
                      (logs []LogEntry, err error)

where

  • querytoken: mandatory, is the query token of the realm
  • app: mandatory, identifies the app whose logs will be examined for these changes
  • class: mandatory, to extract logs related to objects of the specified class only
  • instance: mandatory, to specify the object instance whose changes are being extracted
  • field: optional, the name of a field whose changes are to be extracted
  • fromts, tots: optional, timestamps, to extract log entries whose when value falls in this time range.
  • ndays: optional, an integer specifying how many days back in time the retrieval must attempt, counting backwards in time from the current date
  • logs: will be an array of zero or more log entries which will report changes done to the object instance or the named field of the object instance which is being examined

The three parameters, fromts, tots, and ndays have inter-dependency as follows:

  • If fromts and tots are specified, then tots must be after fromts.
  • If either or both of fromts and tots is specified, then ndays is ignored.
  • If just fromts is specified, it means that the log entries from the given fromts timestamp till the current time will be examined. If only tots is specified, then log entries from the oldest log entry till the given timestamp will be examined.

The function will pull out log entries which refer to the specific object instance as identified by class and instance. If field is also supplied, the function will only pull out log entries which record a change to the field named. Typically, data-change logs have one entry for the creation of an object instance, therefore this log entry will be there in the result set of the function in all cases. Further log entries will be present as per the changes recorded. There is typically a log entry inserted when an object is deleted or permanently deactivated. Such log entries will be present in the result if no specific field was named in the parameters.

Clone this wiki locally