Practice Test 3 | Google Cloud Certified Professional Data Engineer | Dumps | Mock Test
You are assigned to take care of a data pipeline which runs several steps such as ingesting, processing, and loading data to the data warehouse. The steps are triggered using App Engine cron job. Each step is triggered 30 minutes apart from the previous one.
As you are maintaining the pipeline, you observe when a step fails, the rest of the steps will be triggered and will fail respectively since their input is missing. This generates unnecessary alarms and you have to manually stop all remaining steps before fixing the incident. How would you solve this issue?
A. Using a Cloud Function, write a script to check if the job was done. If not, try to cover all possible cases to fix the issue caused and re-run the failed step.
B. Using a Cloud Function, write a script to check if the job was done. If not, it checks the last successful step and re-runs the failed steps.
C. Use Cloud Composer to orchestrate the pipeline’s steps.
D. Use Cloud Scheduler to schedule the pipeline steps, with setting a time period of 60 minutes between each step.
Correct Answer: C.
Cloud Composer is a fully managed workflow orchestration service built on Apache Airflow. Cloud composer is built specifically to schedule and monitor workflows and take required actions. You can use Cloud Composer to orchestrate dataflow pipeline and create a custom sensor to detect file’s condition if any changes occurred, then it triggers the dataflow pipeline to run again.
Options A & B are incorrect: Both approaches still do not fix the root issue which is to find a fault-tolerant and resilient tool to handle pipeline failures.
Option D is incorrect: Cloud Scheduler is also a cron job product from Google which relies on App Engine. This approach is not different from the current scenario.
Source(s):
Cloud Composer:
Comments are closed, but trackbacks and pingbacks are open.