Practice Test 3 | Google Cloud Certified Professional Data Engineer | Dumps | Mock Test

Question 19

A financial services company which offers credit card and loan package services uses BigQuery as a data warehouse to store clients details in the denormalized structure. Data analysts are experimenting on Apache Spark for more data transformation and enrichment and after a few presentations, the head of data decided to move forward and use Apache Spark. As the data engineer, you are assigned to provide the required tech stack. What would you do?

A. Create a Dataproc cluster. Install Dataproc’s BigQuery connector on the cluster using initialization actions. Dataproc temporarily loads data from BigQuery to Google Storage. If failed, you need to manually delete temp files.
B. Create a Dataproc cluster. Install Dataproc’s BigQuery connector using initialization actions. Dataproc temporarily loads data from BigQuery to Google Storage. If failed, Dataproc deletes temp files before finishing the job.
C. Create a Dataproc cluster. Export data from BigQuery to Google Storage in JSON format. Dataproc cluster reads data from Google Storage using a connector. You need to manually delete data files after Dataproc is done.
D. Create a Dataproc cluster. Export data from BigQuery to Google Storage in CSV format. Dataproc cluster reads data from Google Storage using a connector. Dataproc cluster deletes data from Google Storage after Dataproc is done.

Answer

Correct Answer: A

You can use a BigQuery connector to enable programmatic read/write access to BigQuery. This is an ideal way to process data that is stored in BigQuery. No command-line access is exposed. The BigQuery connector is a Java library that enables Hadoop to process data from BigQuery using abstracted versions of the Apache Hadoop InputFormat and OutputFormat classes.

You can access BigQuery from Dataproc by installing BigQuery connector to Dataproc cluster using initialization actions. When a Dataproc spark job reads from BigQuery, it writes the BigQuery table’s content temporarily to Google Storage using Dataproc cluster’s assigned bucket. If the job completes successfully, temporary files are automatically deleted from the cluster. If the job fails, you need to delete temp files manually.

Option B is incorrect: If the job fails, you need to delete temp files manually.

Options C and D are incorrect: Dataproc can read from BigQuery by installing the connector. No need to export data from BigQuery to Google Storage manually.

Source(s):

BigQuery Connector: https://cloud.google.com/dataproc/docs/concepts/connectors/bigquery

Initialization Actions:

https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/init-actions