Practice Test 3 | Google Cloud Certified Professional Data Engineer | Dumps | Mock Test
You have Apache Spark jobs running on on-premise machines. The team decided to migrate all on-premise resources to Google Cloud and Dataproc was considered to be used to run Spark jobs on the cloud while data was migrated from on-premise HDFS to Google Storage. Dataproc will read and write data from and to Google Storage using the connector.
After a while, you noticed that Spark jobs running on Dataproc are I/O intensive and this is causing latencies reading and writing data in Storage. How would you solve this?
A. Increase persistent disk size for Dataproc cluster’s nodes.
B. Increase RAM capacity of Dataproc cluster’s worker nodes.
C. Use local HDFS storage of Dataproc cluster nodes instead of Google Storage.
D. Increase RAM capacity of Dataproc cluster’s master node.
Correct Answer: C
It’s recommended to use Dataproc to run Apache Spark & Hadoop clusters When you want to move Hadoop & Spark workloads from an on-premises environment to Google Cloud Platform (GCP).
Local HDFS storage is a good option if you have workloads that involve heavy I/O. For example, you have a lot of partitioned writes. It is a good option if you also have I/O workloads that are especially sensitive to latency. For example, you require single-digit millisecond latency per storage operation.
Option A is incorrect: Increasing disk size for worker nodes alone is not enough. You should move data to local HDFS storage of Dataproc. Increasing size may help to increase HDFS storage.
Options B and D are incorrect: Increasing memory will not help fix the issue because the problem is because of intensive disk read/write.
Source(s):
Migrating Apache Spark Jobs to Cloud Dataproc:
https://cloud.google.com/solutions/migration/hadoop/migrating-apache-spark-jobs-to-cloud-dataproc
Comments are closed, but trackbacks and pingbacks are open.