0

Resources used

  • Azure Log Analytics
  • Blob Storage v2
  • Azure Databricks job

Log analytics is writing data to blob storage account in a container using export rule. Databricks is having same container mounted and running pipeline which is reading data every one hour and doing transformations. Databricks pipeline is running sometimes properly and sometimes it is giving following error. I can understand there is race condition while reading and writing data to blob storage. Log analytics data export rule doesn't have fixed time threshold to send data to storage. Any idea to handle this race condition?

Caused by: java.io.IOException: Operation failed: "The condition specified using HTTP conditional header(s) is not met.", 412, GET, https://xxx.dfs.core.windows.net/xx-xxx/WorkspaceResourceId%3D/subscriptions/xxx.json?timeout=90, ConditionNotMet, "The condition specified using HTTP conditional header(s) is not met. RequestId:xxx-xxxx-xxxx-xxxx-xxx Time:xxx-04-03T20:11:21.xxxx"
    at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsInputStream.readRemote(AbfsInputStream.java:673)
    at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsInputStream.readInternal(AbfsInputStream.java:619)
    at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsInputStream.readOneBlock(AbfsInputStream.java:409)
    at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsInputStream.read(AbfsInputStream.java:346)
    at java.io.DataInputStream.read(DataInputStream.java:149)
    at com.databricks.common.filesystem.LokiAbfsInputStream.$anonfun$read$3(LokiABFS.scala:204)
    at scala.runtime.java8.JFunction0$mcI$sp.apply(JFunction0$mcI$sp.java:23)
    at com.databricks.common.filesystem.LokiAbfsInputStream.withExceptionRewrites(LokiABFS.scala:194)
Improve this question
1
  • Could you please provide sample code which you have used to read json from blob storage? Commented Apr 8, 2024 at 8:12

1 Answer 1

Reset to default
1 
"The condition specified using HTTP conditional header(s) is not met.", 412, GET, https://xxx.dfs.core.windows.net/xx-xxx/WorkspaceResourceId%3D/subscriptions/xxx.json?timeout=90, ConditionNotMet, "The condition specified using HTTP conditional header(s) is not met."

As per this,

When a write operation is done on a blob, the ETag of the Blob is reset, let's say 0x8CDA1BF0593B660. And, before it is triggered (with ETag value 0x8CDA1BF0593B660), the blob is updated by another service, and its ETag is changed to 0x8CDA1BF0593B661.

That might be the reason for getting the above error while reading a JSON file from a storage account in Databricks. The concurrency behavior can be configured according to the hadoop-azure library documentation. It is the library used to access ADLS (abfss).

For more information, you can refer to the links below:

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.