Practice Test 1 | Google Cloud Certified Professional Data Engineer | Dumps | Mock Test
You have a dataflow pipeline reads a CSV file daily at 6am, applies the needed cleansing & transformation on it, then loads it to BigQuery. Occassionally, the CSV file might be modified within the day due to human error or incomplete data. This causes you to manually re-run dataflow pipeline again. Is there a way to fix this by automatically re-run the pipeline if file has been modified?
A. Use Cloud Scheduler to re-run dataflow after 6am. Check what is the average time the file is modified and schedule based on it.
B. Use Dataproc to reprocess the file after 6am. You can use Cloud Functions to launch a Dataproc cluster.
C. Use Cloud Composer to rerun dataflow and reprocess the file. Create a custom sensor to detect file condition if changed.
D. Use a compute engine to schedule a cron job to run every 10 minutes to check if the file was modified to rerun dataflow.
Answer: C
Cloud Composer is a fully managed workflow orchestration service built on Apache Airflow. Cloud composer is built specifically to schedule and monitor workflows and take required actions. You can use Cloud Composer to orchestrate dataflow pipeline and create a custom sensor to detect file’s condition if any changes occurred, then it triggers the dataflow pipeline to run again.
Answer A is incorrect: Guessing what time scheduler should rerun dataflow is not efficient.
Answer B is incorrect: Dataproc is unnecessary in this scenario.
Answer D is incorrect: Using a Compute Engine VM is unnecessary because Cloud Composer can orchestrate Dataflow pipeline’s failure.
Source(s):
Cloud Composer:
Comments are closed, but trackbacks and pingbacks are open.