Add reporting to xrt::runner (#8922)

The runner infrastructure reports metrics upon request.  The metrics
are currently loosely defined and most metrics are collected upon
request.

Below is a sample of what is reported by the runner and returned
as a json std::string.

The schema will change before it is finalized and versioned.

```
{
  "cpu": {
    "elapsed": 491726,    # execution elapsed time (us)
    "latency": 21,        # execution computed latency (us)
    "throughput": 45793   # execution average throughput (op/s)
  },
  "hwctx": {
    "columns": 0          # Number of columns (not implemented)
  },
  "resources": {
    "buffers": 5,         # Number of xrt::bo objects created
    "kernels": 1,         # Number of xrt::kernel objects created
    "runs": 9,            # Number of xrt::run objects created
    "total_buffer_size": 100 # Total buffer size in bytes
  },
  "xclbin": {
    "uuid": "44968c89-7cfa-a9d6-38e0-17cfa13445f2"
  }
}
```

Signed-off-by: Soren Soe <[email protected]>

Author: stsoe

src/runtime_src/core/common/api/hw_context_int.h

@@ -51,6 +51,11 @@ create_hw_context_from_implementation(void* hwctx_impl);
 xrt::module
 get_module(const xrt::hw_context& hwctx, const std::string& kname);
 
+// Get the partition size (number of columns).  May not be available
+// in xclbin mode.    
+size_t
+get_partition_size(const xrt::hw_context&);
+
 }} // hw_context_int, xrt_core
 
 #endif

src/runtime_src/core/common/api/xrt_hw_context.cpp

@@ -176,6 +176,12 @@ class hw_context_impl : public std::enable_shared_from_this<hw_context_impl>
     return m_mode;
   }
 
+  size_t
+  get_partition_size() const
+  {
+    return m_partition_size;
+  }
+
   xrt_core::hwctx_handle*
   get_hwctx_handle()
   {
@@ -239,6 +245,12 @@ get_module(const xrt::hw_context& ctx, const std::string& kname)
   return ctx.get_handle()->get_module(kname);
 }
 
+size_t
+get_partition_size(const xrt::hw_context& ctx)
+{
+  return ctx.get_handle()->get_partition_size();
+}
+
 } // xrt_core::hw_context_int
 
 ////////////////////////////////////////////////////////////////

src/runtime_src/core/common/runner/README.md

@@ -1,6 +1,6 @@
 <!-- SPDX-License-Identifier: Apache-2.0 -->
 <!-- Copyright (C) 2024-2025 Advanced Micro Devices, Inc. All rights reserved. -->
-# Runner instrastructure
+# Runner infrastructure
 
 This directory contains xrt::runner infrastructure. The runner is
 broken into two json components.  First is the recipe that defines a
@@ -10,3 +10,49 @@ under what constraints how the model is executed.
 - [recipe](recipe.md)
 - [profile](profile.md)
 
+## Instantiating an xrt::runner 
+
+An execution profile is useless without a run recipe.  But a run
+recipe can have several execution profiles, and less likely a
+profile could be used with multiple recipes.
+
+The profile (and the recipe) may reference file artifacts. These
+artifacts can be passed to the runner in two different ways. (1) the
+runner can be instantiated with a path to a directory containing the
+referenced artifacts (files), or (2) it can be instantiated with an
+artifacts repository that have all the artifacts in memory pre-loaded.
+
+See [runner.h](runner.h) or details.
+
+## Reporting
+
+The runner infrastructure reports metrics upon request.  The metrics
+are currently loosely defined and most metrics are collected upon 
+request.
+
+Below is a sample of what is reported by the runner and returned
+as a json std::string. 
+
+The schema will change before it is finalized and versioned.
+
+```
+{
+  "cpu": {
+    "elapsed": 491726,    # execution elapsed time (us)
+    "latency": 21,        # execution computed latency (us)
+    "throughput": 45793   # execution average throughput (op/s)
+  },
+  "hwctx": {
+    "columns": 0          # Number of columns (not implemented)
+  },
+  "resources": {
+    "buffers": 5,         # Number of xrt::bo objects created
+    "kernels": 1,         # Number of xrt::kernel objects created
+    "runs": 9,            # Number of xrt::run objects created
+    "total_buffer_size": 100 # Total buffer size in bytes
+  },
+  "xclbin": {
+    "uuid": "44968c89-7cfa-a9d6-38e0-17cfa13445f2"
+  }
+}
+```

src/runtime_src/core/common/runner/main.cpp

@@ -39,12 +39,15 @@ usage()
 static void
 run(const std::string& recipe,
     const std::string& profile,
-    const std::string& dir)
+    const std::string& dir,
+    bool report)
 {
   xrt::device device{0};
   xrt_core::runner runner {device, recipe, profile, dir};
   runner.execute();
   runner.wait();
+  if (report)
+    std::cout << runner.get_report() << "\n";
 }
 
 static void
@@ -55,12 +58,18 @@ run(int argc, char* argv[])
   std::string recipe;
   std::string profile;
   std::string dir = ".";
+  bool report = false;
   for (auto& arg : args) {
     if (arg == "-h") {
       usage();
       return;
     }
 
+    if (arg == "--report") {
+      report = true;
+      continue;
+    }
+
     if (arg[0] == '-') {
       cur = arg;
       continue;
@@ -76,7 +85,7 @@ run(int argc, char* argv[])
       throw std::runtime_error("Unknown option value " + cur + " " + arg);
   }
 
-  run(recipe, profile, dir);
+  run(recipe, profile, dir, report);
 }
 
 int

src/runtime_src/core/common/runner/profile.md

@@ -13,7 +13,7 @@ resources.
 
 An execution profile is useful for testing of a run recipe.  It allows
 for one external application controlling execution of a run recipe by
-defininng:
+defining:
 
 - how data is bound to resources
 - how validation is performed
@@ -35,6 +35,8 @@ how many times along with what should be done in each iteration.
 
 ## QoS
 
+This section is optional.
+
 Simple key/value pairs designating configuration parameters for
 hardware context creation. 
 ```
@@ -44,11 +46,14 @@ hardware context creation.
   },
 ```
 The json schema doesn't enforce key names or value ranges, XRT will
-warn but ignore unrecoqnized keys. Improper values are implementation 
+warn but ignore unrecognized keys. Improper values are implementation 
 defined.
 
 ## Bindings
 
+This section is optional if all buffer resources specified in the
+recipe are specified with `size` (see [recipe](recipe.md#buffer)).
+
 The bindings section specifies how external buffers should be created,
 initialized, and validated. The section is an array of binding elements,
 where each binding element must reference a resource in the run-recipe.
@@ -68,7 +73,7 @@ file, but is otherwise required.
     { ... }
   ],
 ```
-A binding elememnt has some simple attributes and more complex
+A binding element has some simple attributes and more complex
 nested attributes
 
 ### Simple attributes of a binding element
@@ -111,7 +116,7 @@ should be initialized. There are multiple ways to initialize a buffer.
 ### Initialization of a resource buffer
 
 A bindings element results in the creation of the `xrt::bo` buffer
-object.  Upon creation, the buffer object is initialzied per the init
+object.  Upon creation, the buffer object is initialized per the init
 sub-element.  There are several ways to initialize a buffer only one
 of which can be specified.
 
@@ -137,10 +142,10 @@ have been specified.
       }
 ```
 File initialization implies that the resource buffer should be
-intialized with content from `file`.  The `file` must reference
+initialized with content from `file`.  The `file` must reference
 a key that locates a file on disk or in an artifacts repository used
 during construction of the `xrt::runner`.  If the binding element
-specifies a `size` value, then this size takes precendence over the
+specifies a `size` value, then this size takes precedence over the
 size of the file, otherwise the size of the file will be the size of
 the buffer. The optional `skip` element
 allows skipping first bytes of the file during initialization of the
@@ -157,7 +162,7 @@ of the file are used.
 
 If a bindings element specifies `reinit`, then the buffer is
 reinitialized with bytes from the file in each iteration of the
-recipe.  The initalization in an iteration picks up from an offset
+recipe.  The initialization in an iteration picks up from an offset
 into the file at the point where the previous iteration stopped
 copying.  Again, the file wraps around when end-of-file is reached
 without filling all the bytes of the buffer.
@@ -166,9 +171,9 @@ without filling all the bytes of the buffer.
 
 ```
       "init": {
-        "stride": 1,    // stide bytes
+        "stride": 1,    // stride bytes
         "value": 239,   // value to write at each stride
-        "begin": 0,     // begining of range to write at
+        "begin": 0,     // beginning of range to write at
         "end": 524288   // end of range
       }
 ```
@@ -179,7 +184,7 @@ is referred to as strided buffer initialization.
 In the above example, the buffer range specified by `begin` and `end`
 is written with the byte value `239` at every byte (`stride` is
 1). The value can be any uint64_t value, but must be a decimal value
-in order to be valid json.  To initalize a buffer with with
+in order to be valid json.  To initialize a buffer with with
 0xdeadbeef, convert to decimal and write:
 
 ```
@@ -191,8 +196,8 @@ in order to be valid json.  To initalize a buffer with with
 
 #### Validation
 
-After executon, a buffer can be validated against golden data. This is
-specifed using a `validate` element for the binding.
+After execution, a buffer can be validated against golden data. This is
+specified using a `validate` element for the binding.
 
 ```
     {
@@ -218,13 +223,23 @@ specifics as needed.
 
 ## Execution
 
+This section is optional.  If not present, the recipe will execute
+one iteration.
+
 The execution section of a profile specifies how many times the recipe
 should be executed and how.  It controls what should happen after each
-iteration and before next iteration.
+iteration and before next iteration. if `iterations` is not specified,
+then the recipe will execute one iteration.
+
+By default the profile execution will display to stdout elapsed,
+throughput, and latency computed from running the recipe specified
+number of iterations.  
+This can be disabled by setting `verbose: false`.
 
 ```
   "execution" : {
     "iterations": 500,
+    "verbose": false,   // disable reporting of cpu time
     "iteration" : {
       "bind": false,    // re-bind binding buffers
       "init": true,     // re-initialize binding buffers
@@ -235,15 +250,15 @@ iteration and before next iteration.
   }
 ```
 
-The iteration element specifies what should happen before after each
-iteration of the run recipe.
+The iteration element is optional, but if present specifies what
+should happen before after each iteration of the run recipe.
 
 - `bind` indicates if buffers should be re-bound to the
-recipe before an iteration.  Only buffers whos binding element
+recipe before an iteration.  Only buffers whose binding element
 specifies `rebind` are re-bound.  All buffers are by default
 bound to the recipe upon creation.
 - `init` indicates if buffer should be re-initialized before an
-iteration. Only buffers whos binding element specifies has an `init`
+iteration. Only buffers whose binding element specifies has an `init`
 element and specifies `reinit` are re-initialized.  All buffers are by
 default initialized upon creation if their binding element has an
 `init` element.
@@ -256,18 +271,3 @@ iterations and after last iteration.
 - `validate` means buffer validation per what is specified in
 the binding element.
 
-# Instatiating an xrt::runner with a profile
-
-An execution profile is useless without a run recipe.  But a run
-recipe can have several execution profiles, and less likely a
-profile could be used with multiple recipes.
-
-The profile (and the recipe) may reference file artifacts. These
-artifacts can be passed to the runner in two different ways. (1) the
-runner can be instantied with a path to a directory containing the
-referenced artifacts (files), or (2) it can be instantiated with an
-artifacts repository that have all the artifacts in memory pre-loaded.
-
-See [runner.h](runner.h) or details.
-
-

src/runtime_src/core/common/runner/recipe.md

@@ -6,10 +6,10 @@ A run recipe defines a graph model that can be executed by XRT.
 
 This directory contains a stand-alone `xrt::runner` class that reads and 
 executes a run recipe json file.   The idea is to have tools, e.g. VAIML
-geneate the run recipe along with xclbin and control code for kernels.
+generate the run recipe along with xclbin and control code for kernels.
 
 The schema of the recipe json is defined in `schema/recipe.schema.json`. The
-implementation of the runner drove some of the defintion of the json
+implementation of the runner drove some of the definition of the json
 format.
 
 A run recipe is associated with exactly one configuration (xclbin or
@@ -75,7 +75,7 @@ be listed in the resources section.
 
 ### Kernel functions
 
-Kernels listed in the resoruces section result in runner creating
+Kernels listed in the resources section result in runner creating
 `xrt::kernel` objects.  In XRT, the kernel objects are identified by
 name, which must match a kernel instance name in the xclbin.
 
@@ -426,7 +426,7 @@ input and output are consumed during one kernel execution.  See the
 
 # Runner API
 
-The runner is contructed from a recipe json file and a device object.
+The runner is constructed from a recipe json file and a device object.
 The runner is a standard XRT C++ first class object with the following
 API.  Include documentation will be beefed up when the runner code is 
 moved to public XRT.