Write latency of ADLS Gen2

[johnxpan]

I run my application on a Ubuntu VM, the app uses spark to write about 10M of parquet data to ADLS blob container (mounted by blobfuse2) every 5 minutes. We found that each write took around 4 minutes to complete. I copied the same amount of data directly (without spark session) to the ADLS, it still took over 2 minutes. The Ubuntu box is not under pressure ( lower CPU and memory usage), the upload speed of the internet link is over 1500Mb/sec. My questions are:

1. Top shows the blobfuse2 CPU usage is over 100 most of the time, sometimes over 250. The VM has 16 cores, so it is not under pressure. However, is it normal for Blobfuse2 has such a high CPU usage?
2. Under my setting, is it reasonable to have 2~4 minutes latency for writing 10M data in burst mode?
3. I did observe that in several cases, we had over 30 minutes delay for a single write. All of them happened at midnight (UTC time). Is it possible that Azure is doing some maintanence that caused this huge delay?
4. I am using the block-cache for blobfuse2. Any tuning suggestion? Will blobfuse2 logging help on debugging this issue?

[ashruti-msft]

Hello, for your case where you are creating files of size 10MB its better to use file_cache instead. Please try that and let me know if you still see latency, thanks!