could not reduce file size

[sandip094]

Which version of blobfuse was used?

blobfuse2 version: 2.4.0

Which OS distribution and version are you using?

CentOS Linux release 7.9.2009 (Core)

What was the issue encountered?
Getting the below error while running the RMAN backup.
released channel: C1 RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03002: failure of backup plus archivelog command at 01/06/2025 11:53:13 ORA-19510: failed to set size of 1758986 blocks for file "/rman-backup/step/0/2025-01-03_1146/STEP_148867_1_c33ei92k_20250106.incr0" (block size=8192) ORA-27059: could not reduce file size Linux-x86_64 Error: 103: Software caused connection abort Additional information: 2

Configuration file is as below -/etc/blobfuse/blobfuseconfig.yaml
`# Refer ./setup/baseConfig.yaml for full set of config parameters
#allow-other: false

logging:
type: base
level: log_info
max-file-size-mb: 32
file-count: 10
track-time: true
max-concurrency: 8
components:

• libfuse
• file_cache
• azstorage

libfuse:
default-permission: 0644
attribute-expiration-sec: 120
entry-expiration-sec: 120
negative-entry-expiration-sec: 240
ignore-open-flags: true

file_cache:
path: /mnt/blobfusetmp
timeout-sec: 20
max-size-mb: 75776
allow-non-empty-temp: true
cleanup-on-start: true

azstorage:
type: block
account-name: x
account-key: x
mode: key
container: x
blobfuse2.log
`

So far tried these options:

• Changed the RMAN backup location from blobfuse container to disk and its completed successfully
• Changed the temp storage ~74GB and still it fails. Database is not big so not sure the reason for the failure
• Attached the logs for further reference

[vibhansa-msft]

I do not see any failure logs in the shared log file. Neither could I find any reference to ""/rman-backup/step/0/2025-01-03_1146/STEP_148867_1_c33ei92k_20250106.incr0" file in the logs. Are you sure correct logs are shared. As per logs the last mount was just trying to create blobfuse2.log file on the container, which makes me curious about the log file location. Are you trying to save blobfuse logs in the mounted container itself ?

[sandip094]

@vibhansa-msft , @ashruti-msft : Attached the fresh logs now.

Starting backup at 07-JAN-2025 11:27:58
channel C1: starting incremental level 0 datafile backup set
channel C1: specifying datafile(s) in backup set
input datafile file number=00015 name=/database/db1/oradata/step/stepsysdata_step_3.dbf
channel C1: starting piece 1 at 07-JAN-2025 11:27:59
released channel: C1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup plus archivelog command at 01/07/2025 11:29:34
ORA-19510: failed to set size of 1758986 blocks for file "/rman-backup/step/0/2025-01-03_1146/STEP_148978_1_fi3eks1v_20250107.incr0" (block size=8192)
ORA-27059: could not reduce file size
Linux-x86_64 Error: 103: Software caused connection abort
Additional information: 2

blobfuse2-07-01.log

[ashruti-msft]

Hi @sandip094 your logs end at timestamp 11:29:25 and after a break logs indicate blobfuse was mounted again. Can you explain what might be the reason for this? Since the error occurred at 11:29:34 and there are no logs around that time I cannot debug further.

[sandip094]

Hello @ashruti-msft ,

As soon as the error occurs /rman-backup gets disconnected and had to remout manually which i did.
I have tried to recereate the issue again and was able to. Attached the complete logs for the reference.
blobfuselogs-08.zip

Starting backup at 08-JAN-2025 12:38:24
channel C1: starting incremental level 0 datafile backup set
channel C1: specifying datafile(s) in backup set
input datafile file number=00015 name=/database/db1/oradata/step/stepsysdata_step_3.dbf
channel C1: starting piece 1 at 08-JAN-2025 12:38:25
released channel: C1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup plus archivelog command at 01/08/2025 12:40:00
ORA-19510: failed to set size of 1758986 blocks for file "/rman-backup/step/0/2025-01-03_1146/STEP_149101_1_jd3enki1_20250108.incr0" (block size=8192)
ORA-27059: could not reduce file size
Linux-x86_64 Error: 103: Software caused connection abort
Additional information: 2

[ashruti-msft]

Hi @sandip094 after analyzing the logs I could not find any indication of an issue from Blobfuse during the transfer. It looks like there is a connection abort due to which the backup fails. Could you please confirm if there are no network issues or any other reason causing this error?

[sandip094]

@ashruti-msft : I can confirm that there has been no network issue. However, what’s puzzling is why the /rman-backup mount gets disconnected every time the error occurs. Is there a way to ensure that this mount remains connected and doesn’t get disconnected?

[vibhansa-msft]

Do you mean to say in case of error the mount is not longer valid ?
If so, I suspect blobfuse might be crashing. To root-cause that we want you to mount blobfuse with "--foreground" cli option. This will block the shell where you mount and blobfuse will run as a user application. From another shell you can try doing a backup and in case of crash the mount shell will dump the backtrace on the console. You can share that output and we can take it up from there.