high memory usage

[rafaeljesus]

Hello, I using pprof-rs to generate pprof data and send it to GCloud,

I have a background task that runs every 5s as follows:

const FREQUENCY: i32 = 10;

// this fn is called every 5s in a loop
pub async fn cpu(&self) -> Result<protos::Profile, ProfileError> {
    let guard = pprof::ProfilerGuardBuilder::default()
        .frequency(FREQUENCY)
        .blocklist(&["libc", "libgcc", "pthread", "vdso"])
        .build()
        .map_err(|e| ProfileError::CpuError(e.to_string()))?;

    // Build the profile and return it in profile.proto format:
    guard
        .report()
        .build()
        .map_err(|e| ProfileError::CpuError(e.to_string()))?
        .pprof()
        .map_err(|e| ProfileError::CpuError(e.to_string()))
}

the caller side, take_profile runs every 5 seconds in a loop

async fn take_profile(&self) -> Result<(), ProfileError> {
    let mut profile_handle = // crate_profile RPC
    let profile_type = ...

    let profile_bytes = match profile_type {
        crate::ProfileType::Cpu => {
            let profile = self.inner.profiler.cpu().await?; // calling the cpu fn declared above
            let mut buf = bytes::BytesMut::new();
            profile
                .encode(&mut buf)
                .map_err(|e| ProfileError::EncodingError(e.to_string()))?;
            Ok(buf.freeze())
        }
        _ => Err(ProfileError::UnsupportedType(format!("{:?}", profile_type))),
    };

    match profile_bytes {
        Ok(bytes) => {
            profile_handle.profile_bytes = bytes;
            self.client()
                .upload_profile(profile_handle.clone())
                .await
                .map_err(|e| ProfileError::UpdateError(e.to_string()))?;
            Ok(())
        }
        Err(e) => Err(e),
    }
}

I have this enabled in different rust apps in our GKE cluster, and in all of them memory increased + 10x, below is a screenshot from a pod that was using 15 MB, and when the pprof-rs was enabled it jumped to +300MB

metric: jemalloc_active_bytes

I took heap profile data using polarsignals/rust-jemalloc-pprof

Does that ring any bell for you? I appreciate any help you can provide.

[rafaeljesus]

I have a background task that runs every 5s

I've reduced to 60s and still the same memory consuption

[rafaeljesus]

just to rule out any possible problem with the background task, I took a profile hitting the pprof/profile endpoint in a app with the background task mentioned above disabled, I took the profile just once and memory increased this much and continued that way,

in my pprof endpoint, I am also using the ProfilerGuardBuilder

[ttys3]

yes, I got this problem too.

We limited the process memory to 128M before, since it actully usage is around 10MB.

but after enabled pyroscope-rs (which uses tikv/pprof-rs), the memory increased to near 220MB+