Datastream to Dataflow to Postgres Part 1
This note on POC Project to transfer data from CloudSQL MySQL to PostgreSQL using GCP tools. It quite long so I Split to 2 parts. First part is Datastream. I used Cloud Storage, CloudSQL, Datastream and Dataflow. This method will cost you. You might need some budget to try. Flow is as follow but change from BigQuery to Postgres.
First, Create Streams from Datastream menu.
Select Destination Type = Cloud Storage
Review Prerequisite and do as they say.
Continue to Next step. Create Source Connection.
POC By Allowing IP
Go to Your CloudSQL Source Database and allow IP from this List
Come back and runtest.
Go to Next Step. Select Database and table to transfer data.
Next define Cloud Storage. Your Destination for Datastream.
Fill in your CloudStorage info. Create and you will get connection profile.
Set Path Prefix and format.
Run Validation at last step. If pass, create and start!
It will redirect you to Stream detail page. After it done, check your Cloud Storage for the files it stream from your DB.
