Practice Test 3 | Google Cloud Certified Professional Data Engineer | Dumps | Mock Test
You are building an ETL pipeline to process over 25GB of data every day. You decide to use Dataproc for custom Apache Spark jobs for data cleansing and transformation. Jobs running on Dataproc require dependencies that are not installed in Dataproc by default while launching a cluster. Security measures in your company don’t allow resources to connect to the internet and Dataproc cannot install these dependencies online. What would you do in this situation?
A. Launch a Compute Engine instance and use it as NAT instance to install the dependencies from.
B. Store required dependencies in Google Storage. Install the dependencies to Dataproc nodes using initialization actions.
C. Launch a Compute Engine instance. Download the dependencies on a persistent disk using VM instance. Stop the VM instance and use persistent disk for Dataproc cluster to install the dependencies from.
D. While launching Dataproc cluster, you may provide the URIs with which you can download the dependencies from. Dependencies will be installed before the cluster is up.
Correct Answer: B
You can create a Cloud Dataproc cluster with internal IP addresses only. However, attempts to access the Internet in an initialization action will fail unless you have configured routes to direct the traffic through a NAT or a VPN gateway. Without having access to the Internet, you can enable Private Google Access, and place job dependencies in Cloud Storage; cluster nodes can download the dependencies from Cloud Storage from internal IPs.
Options A & C are incorrect: There is no need to use a Compute Engine instance as answer B provides a better and cheaper solution.
Option D is incorrect: This is not possible because the cluster will not be able to access the internet if it was set with internal IP addresses only. This also does not comply with security measures which do not allow installing dependencies from the internet.
Source(s):
Initialization actions:
https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/init-actions
Comments are closed, but trackbacks and pingbacks are open.