We are having multiple Task failures for Cloud run, the error message in Cloud Run says "md5 checksum error"

[Gowtham589]

We are getting the md5 checksum error again across multiple pipelines. this error is causing failures for our data pipelines.
Earlier, we received this error and Google provided the following findings:
"GCS doesn't store MD5 checksums for composite objects. A composite object isn't a single entity; it's a collection of individual objects. Verifying the integrity of the composite object itself isn't possible using MD5. This behavior was already in place and there haven’t been any recent changes on the GCS side.
There are chances that earlier object uploaded were not the composite objects and that's why the MD5 checksum consistency check was not the issue for them. From the error, it is confirmed that MD5 checksum verification is done for the composite objects and for composite objects GCS doesn't add any MD5 checksum values.
Since, updating the gcsfs helped here, I believe it might be related to gcsfs releases, please note that gcsfs is not the GCP tool it's a third-party tool. I believe GCS service doesn't have any role in the error you observed."
Solution: We had upgraded the libraries to the latest version which eventually fixed the issue for certain pipelines, however, this time this solution alone is not fixing the issue.
We wanted to connect with you to understand why this error is getting triggered again, is there any change being done on the environment or any release again?
Please let us know at the earliest.

[martindurant]

Can you please show what you tried to run, the full traceback and the versions of gcsfs, fsspec and google* packages?

[Gowtham589]

These are following packages we have used.

.

We are receiving multiple files into GCS bucket in same naming convention at a time, & we are reading the data from these files using python libraries utilizing Cloud run service & loading into Big Query, But recently we are facing MD5 check sum when we have multiples at a time in GCS.

If you still having any doubts we can connect over a call, we are available from 9:00 AM IST to 6:00 PM IST.

Thank you.

[martindurant]

Can you please copy&paste the exception that you are facing? (be sure to anonymize any sensitive values first)

[Gowtham589]

plz find the exception:
No md5 checksum available to do consistency check. GCS does not provide md5 sums for composite objects

[martindurant]

Sorry, I need to see your actual exception and traceback. I assume those logs have more information.
Also, I need to see what code you are executing.

[nageswar-loreal]

Hi Team,

Please help us to setup a call with you troubleshoot this further fastly because our production pipelines are getting impacted.
we are available from 9:00 AM IST to 6:00 PM IST.

We have faced another instance where md5 checksum error has caused 2 of the DAGs for a particular application (ASEAN) to fail. When we clear the tasks of the DAG, the pipleline is successful until feb 10th.Now it was not successful even updating the latest versions. However, this error randomly keeps coming back for these 2 pipelines.

We had raised a Google Support case for the same and they have redirected us to contact the gcsfs team for further analysis of the RCA. Please help us understand the root cause.

[martindurant]

What timezone is IST? It could be Indian, Irish or Israel according to wikipedia. How do I contact you?

[Gowtham589]

its Indian, we can connect though Google meet or MSteams

[nageswar-loreal]

Hi @martindurant ,

You can contact us through this mail - "GCPAPPLICATIONOPS_NA@loreal.com"

[nageswar-loreal]

Hi @martindurant ,

can you please share your email or team_DL to communicate more on this issue

[martindurant]

Following live discussions, some notes for anyone following this issue:

• md5 checksums are optional (consistency= keyword) in the current codebase. It is conceivable that this was not the case in previous version, but I don't see an indication of this. This doesn't appear to be an issue (any more) with the current code
• the NotImplemented: calling sync() error is a consequence of calling the blocking API from within a running event loop. This was not obvious, because the immediate calling function was not async, but it was being called from an async context. There are two possible options:
  • make the functions async all the way down and call only async methods (those with leading "_" in the name). So open(..).read() -> await _cat_file().
  • call the blocking function as it is now, but on a new thread, e.g., with threading.Thread(target=)

Both solutions to the second point would result in the main application event loop being responsive, rather than waiting for IO to finish.