Practice Test 2 | Google Cloud Certified Professional Data Engineer | Dumps | Mock Test
You receive a daily comma-separated (CSV) file which should be imported to BigQuery. You need to scan the file in case of incomplete or improperly aligned column values which will cause importing to BigQuery fail.
What should you do to handle invalid inputs?
A. Import data to BigQuery, then run queries to check if data skew exists among table fields.
B. Import file to BigQuery with setting –max_bad_records.
C. Use Google Stackdriver to monitor import status and create an alert for failed imports.
D. Build a Dataflow batch pipeline to scan CSV file, import valid rows to BigQuery and push invalid ones to a dead-letter file on Google Storage for later analysis.
Answer: D.
A better way to solve this problem would be to have a dead letter file where all of the failing inputs are written for later analysis and reprocessing. We can use a side output in Dataflow to accomplish this goal.
Answer A is incorrect: This won’t avoid the fact data will fail loading into BigQuery.
Answer B is incorrect: Setting –max_bad_records option won’t help processing the file, and if number of errors exceed the value set for this option, it will return an error and cause the job to fail.
Answer C is incorrect: Stackdriver doesn’t have a native support for detecting BigQuery import failures. Also, this does not help fixing corrupted file.
Source(s):
Handling Invalid Inputs in Dataflow: https://cloud.google.com/blog/products/gcp/handling-invalid- inputs-in-dataflow
BigQuery – Loading CSV Data from Google Storage: https://cloud.google.com/bigquery/docs/loading- data-cloud-storage-csv
Comments are closed, but trackbacks and pingbacks are open.