the result by getBytesReadFromPMM and result by ”ipmctl show -performance TotalReadRequests“ are quite different

[Spear-Neil]

I'm writing some code to trace pmem/nvm access in my small benchmark, and the following snippet shows how I use pcm:

  pcm::PCM* pcm = nullptr;
  pcm::SystemCounterState before, after;
  if(enable_pcm) {
    pcm = pcm::PCM::getInstance();
    pcm->checkError(pcm->program());
    before = pcm->getSystemCounterState();
  }

  Do some work with multiple threads

  if(enable_pcm) {
    after = pcm->getSystemCounterState();
    std::cout << "[INFO]: L3 Miss Ratio: " << 1 - pcm::getL3CacheHitRatio(before, after) << std::endl;

    double mem_reads = (double) pcm::getBytesReadFromMC(before, after) / (0x01ul << 20);
    double mem_writes = (double) pcm::getBytesWrittenToMC(before, after) / (0x01ul << 20);
    std::cout << "[INFO]: Mem Reads: " << mem_reads << " MiB, " << mem_reads * 1000000 / drt << " MiB/S" << std::endl;
    std::cout << "[INFO]: Mem Writes: " << mem_writes << " MiB, " << mem_writes * 1000000 / drt << " MiB/S"
              << std::endl;

    double pmem_reads = (double) pcm::getBytesReadFromPMM(before, after) / (0x01ul << 20);
    double pmem_writes = (double) pcm::getBytesWrittenToPMM(before, after) / (0x01ul << 20);
    std::cout << "[INFO]: PMM Reads: " << pmem_reads << " MiB, " << pmem_reads * 1000000 / drt << " MiB/S" << std::endl;
    std::cout << "[INFO]: PMM Writes: " << pmem_writes << " MiB, " << pmem_writes * 1000000 / drt << " MiB/S"
              << std::endl;
}

But I find the result (sometimes even unreasonable, like 37 GiB/second, much higher than the ideal read bandwidth with 4 * 128 GB Optane DC PMem modules, configured in 2133MT/s) is quite different from the result using the following command

ipmctl show -performance TotalReadRequests > read-req.before
ipmctl show -performance TotalReadRequests > read-req.after

def extract_total_access(filename, field_name="TotalMediaWrites"):
    total = 0
    with open(filename, 'r') as f:
        for line in f:
            if field_name in line:
                match = re.search(rf'{field_name}=([0-9a-fx]+)', line, re.IGNORECASE)
                if match:
                    hex_str = match.group(1).lower().replace('0x', '').lstrip('0')
                    if hex_str:
                        total += int(hex_str, 16)
    return total


def main():
    if len(sys.argv) < 3:
        print("usage: python3 script.py before.txt after.txt [field-name]")
        print("default field: TotalMediaWrites")
        return

    before_file, after_file = sys.argv[1], sys.argv[2]
    field_name = sys.argv[3] if len(sys.argv) > 3 else "TotalMediaWrites"

    before_total = extract_total_access(before_file, field_name)
    after_total = extract_total_access(after_file, field_name)
    difference = after_total - before_total

    print(f"{field_name} (bytes): {difference * 64}")

Is this because there is something wrong with my code?
By the way, the pcm version used by https://github.com/sfu-dis/pibench does not have this issue.

Platform:
Ubuntu 22.04 LTS
kernel: 5.15.0-25-generic
CPU: Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz