Practice Test 2 | Google Cloud Certified Professional Data Engineer | Dumps | Mock Test
You have a complex data pipeline which has a combination of shell scripts, python code and spark jobs. These task are scheduled by cron jobs to run. The problem with this approach is, in case of failure, the whole pipeline breaks and failure control with stopping next tasks from running after a certain task fails and re-running the pipeline again is difficult and messy. You want a solution which can manage the pipeline’s different jobs to be failure-resilient, scalable and easy to monitor.
What approach is best for this scenario?
A. Use Cloud Composer to orchestrate the pipeline workflow.
B. Use Dataproc for Apache Spark jobs and migrate all other tasks to use Apache Spark instead.
C. Use Cloud Scheduler to schedule pipeline’s tasks.
D. Use Dataflow to re-build the data pipeline.
Answer: A
Cloud Composer is a fully managed workflow orchestration service built on Apache Airflow. Cloud composer is built specifically to schedule and monitor workflows and take required actions. You can use Cloud Composer to orchestrate diverse task types, whether they are shell scripts, python code or spark jobs.
Answer B is incorrect: Dataproc does not orchestrate workflows and migrating all tasks can be a difficult job.
Answer C is incorrect: Using Cloud Scheduler does not solve the core problem which is how to manage the workflow.
Answer D is incorrect: Dataflow pipelines are effective for simple and direct workflows which are similar in environment and purpose. This scenario is dealing with a complicated pipeline with various tech and code snippets used.
Source(s):
Cloud Composer: https://cloud.google.com/composer/
Comments are closed, but trackbacks and pingbacks are open.