Azure Data Factory trigger is not initiated when uploading a file using Java SDK

[Guest mahmoudsamy]

Symptoms:

 

 

 

• Uploading files using Java SDK class DataLakeFileClient fails to initiate the ADF trigger.
  
• Uploading files using Azure Portal initiates the trigger as expected.
  
• ADF trigger is configured to ignore empty blobs.
  

 

 

 

Analysis:

 

 

 

• Whenever a file is created using ADLS Gen2 REST API, there are 3 operations called at the backend i.e. CreateFile, AppendFile and FlushFile.
  
• Microsoft.Storage.BlobCreated event is triggered when a blob is created or replaced. Specifically, this event is triggered when clients use the CreateFile and FlushWithClose operations that are available in the Azure Data Lake Storage Gen2 REST API.
  
• As CreateFile API created a 0 byte blob, the event will not be triggered.
  

 

 

 

Resolution:

 

 

 

• You can initiate the trigger by adding FlushWithResponse method with close parameter as true.
  

 

Example:

 

 

 

boolean close = true;

 

File file = new File("C:\\Users\\mahmoudsamy\\route.txt");

 

fileClient.uploadFromFile("C:\\Users\\mahmoudsamy\\route.txt");

 

fileClient.flushWithResponse(file.length(), retainUncomittedData, close, httpHeaders, requestConditions, timeout, contect);

 

 

 

Note: Please make sure to pass the correct parameters when using FlushWithResponse to avoid unexpected results.

 

Continue reading...