Datastream to Dataflow to Postgres Part 2

This continue from part 1. After you got streaming files in your Cloud Storage, you need to import to PostgreSQL via Dataflow.

https://cloud.google.com/dataflow

First, Go to Dataflow and create Pipeline.

Select From Dataflow Template > Datastream to SQL

Fill in your CloudStorage info from part 1. Fill in your Database info and credential. I use Internal IP to connect to Database. Make sure your Destination Cloud SQL enable Private IP and Private Path.

Then, go back and create pipeline.

Fill in your database name in Optional Parameter. Delete starting date time, I got error from default value. I don’t know why.

Create pipeline and go see your Job running.

Check data. You need to create schema and table beforehand in your destination Database. Check the log if error occurs. For me, I mistakenly created table in wrong schema in destination database.

Hope this help.