I'm confronting a strange situation here with Google Cloud SQL.
I'm migrating a 15.7Gb mysql database to Google cloud. I've followed the migration process exactly as the doc says.... moreI'm confronting a strange situation here with Google Cloud SQL.
I'm migrating a 15.7Gb mysql database to Google cloud. I've followed the migration process exactly as the doc says. And everything worked perfectly. Absolutely no issue during the process, my application works just fine. The only problem here is that the size used by the DB shown on Google Cloud is much bigger that the original DB. Right now I have a 39Gb sql database, from a 15.7Gb database.
After some research and testing I've come to the conclusion that it's the way that Google count the data on their side.
I just wanted to know if somebody have any idea, or can confirm what I'm saying.
Thank you for your answers. less
I need to ETL data into my Cloud SQL instance. This data comes from API calls. Currently, I'm running a custom Java ETL code in Kubernetes with Cronjobs that makes request to... moreI need to ETL data into my Cloud SQL instance. This data comes from API calls. Currently, I'm running a custom Java ETL code in Kubernetes with Cronjobs that makes request to collect this data and load it on Cloud SQL. The problem comes with managing the ETL code and monitoring the ETL jobs. The current solution may not scale well when more ETL processes are incorporated. In this context, I need to use an ETL tool.
My Cloud SQL instance contains two types of tables: common transactional tables and tables that contains data that comes from the API. The second type is mostly read-only in a "operational database perspective" and a huge part of the tables are bulk updated every hour (in batch) to discard the old data and refresh the values.
Considering this context, I noticed that Cloud Dataflow is the ETL tool provided by GCP. However, it seems that this tool is more suitable for big data applications that needs to do complex transformations and ingest data in multiple formats. Also, in Dataflow, the... less