traceback:
Traceback (most recent call last):
File "/Users/soubhagyapradhan/Desktop/upwork/baby/data-science/env/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
return callable_(*args, **kwargs)
File "/Users/soubhagyapradhan/Desktop/upwork/baby/data-science/env/lib/python3.8/site-packages/grpc/_channel.py", line 923, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/Users/soubhagyapradhan/Desktop/upwork/baby/data-science/env/lib/python3.8/site-packages/grpc/_channel.py", line 826, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.PERMISSION_DENIED
details =... less
I'm relatively new to GCP and just starting to setup/evaluate my organizations architecture on GCP.
Scenario:Data will flow into a pub/sub topic (high frequency, low amount of... moreI'm relatively new to GCP and just starting to setup/evaluate my organizations architecture on GCP.
Scenario:Data will flow into a pub/sub topic (high frequency, low amount of data). The goal is to move that data into Big Table. From my understanding you can do that either with a having a cloud function triggering on the topic or with Dataflow.
Now I have previous experience with cloud functions which I am satisfied with, so that would be my pick.
I fail to see the benefit of choosing one over the other. So my question is when to choose what of these products?
Thanks less
I'm currently learning how to use new Cloud Functions for Firebase and the problem I'm having is that I can't access the function I wrote through an AJAX request. I get the "No... moreI'm currently learning how to use new Cloud Functions for Firebase and the problem I'm having is that I can't access the function I wrote through an AJAX request. I get the "No 'Access-Control-Allow-Origin'" error. Here's an example of the function I wrote:
exports.test = functions.https.onRequest((request, response) => {
response.status(500).send({test: 'Testing functions'});
})
The function sits in this url: https://us-central1-fba-shipper-140ae.cloudfunctions.net/test
Firebase docs suggests to add CORS middleware inside the function, I've tried it but it's not working for me: https://firebase.google.com/docs/functions/http-events
This is how I did it:
var cors = require('cors');
exports.test = functions.https.onRequest((request, response) => {
cors(request, response, () => {
response.status(500).send({test: 'Testing functions'});
})
})
What am I doing wrong? I would appreciate any help with this.UPDATE:Doug Stevenson's answer helped. Adding ({origin: true}) fixed the issue, I... less
I need to ETL data into my Cloud SQL instance. This data comes from API calls. Currently, I'm running a custom Java ETL code in Kubernetes with Cronjobs that makes request to... moreI need to ETL data into my Cloud SQL instance. This data comes from API calls. Currently, I'm running a custom Java ETL code in Kubernetes with Cronjobs that makes request to collect this data and load it on Cloud SQL. The problem comes with managing the ETL code and monitoring the ETL jobs. The current solution may not scale well when more ETL processes are incorporated. In this context, I need to use an ETL tool.
My Cloud SQL instance contains two types of tables: common transactional tables and tables that contains data that comes from the API. The second type is mostly read-only in a "operational database perspective" and a huge part of the tables are bulk updated every hour (in batch) to discard the old data and refresh the values.
Considering this context, I noticed that Cloud Dataflow is the ETL tool provided by GCP. However, it seems that this tool is more suitable for big data applications that needs to do complex transformations and ingest data in multiple formats. Also, in Dataflow, the... less
Google just released Cloud Firestore, their new Document Database for apps.
I have been reading the documentation but I don't see a lot of differences between Firestore and... moreGoogle just released Cloud Firestore, their new Document Database for apps.
I have been reading the documentation but I don't see a lot of differences between Firestore and Firebase DB.
The main point is that Firestore uses documents and collections which allow the easy use of querying compared to Firebase, which is a traditional noSQL database with a JSON base.
I would like to know a bit more about their differences, or usages, or whether Firestore just came to replace Firebase DB?
I am wanting to start a data warehouse in Google Big Query but I'm not sure how to actually schedule jobs to get the data into the cloud.
To give some background. I have a MySQL... moreI am wanting to start a data warehouse in Google Big Query but I'm not sure how to actually schedule jobs to get the data into the cloud.
To give some background. I have a MySQL database hosted on-prem which I currently take a demp of each night as a backup. My idea is that I can send this dump to the Google Cloud and have it import the data into Big Query. I have thought that I could send the dump and probably use a cloud scheduler function to then run something that opens the dump and does this but I'm unsure how these services all fit together.
I'm a bit of a newby with the Google Cloud so if there is a better way to achieve this then I'm happy to change my plan of action.
Thanks in advance. less
Task: We have to setup a periodic sync of records from Spanner to Big Query. Our Spanner database has a relational table hierarchy.
Option Considered I was thinking of using... moreTask: We have to setup a periodic sync of records from Spanner to Big Query. Our Spanner database has a relational table hierarchy.
Option Considered I was thinking of using Dataflow templates to setup this data pipeline.
Option1: Setup a job with Dataflow template 'Cloud Spanner to Cloud Storage Text' and then another with Dataflow template 'Cloud Storage Text to BigQuery'. Con: The first template works only on a single table and we have many tables to export.
Option2: Use 'Cloud Spanner to Cloud Storage Avro' template which exports the entire database. Con: I only need to export selected tables within a database and I don't see a template to import Avro into Big Query.
Questions: Please suggest what is the best option for setting up this pipeline less
Google announced beta version of a new machine learning environment today. Can someone update me on where Google CloudML stands? The docs seemed to have just changed overnight.... moreGoogle announced beta version of a new machine learning environment today. Can someone update me on where Google CloudML stands? The docs seemed to have just changed overnight. I'm guessing that commands that were
gcloud beta ml
are now
gcloud ml-engine
Having trouble parsing the notes released today:
https://cloud.google.com/ml-engine/docs/resources/release-notes
I was following the tutorial here.
https://cloud.google.com/blog/big-data/2016/12/how-to-train-and-classify-images-using-google-cloud-machine-learning-and-cloud-dataflow
What else should I be look out for? less
I wonder if it is possible to use Cloud Armor with GAE Flex? Because in Cloud Armor's documentation, it says that you have to use an HTTPS Load Balancer. Since GAE Flex doesn't... moreI wonder if it is possible to use Cloud Armor with GAE Flex? Because in Cloud Armor's documentation, it says that you have to use an HTTPS Load Balancer. Since GAE Flex doesn't have a load balancer, how can we use Cloud Armor with GAE Flex? We have to use a WAF to prevent DDOS attacks. Is it possible to use Cloud Armor with GAE Flex through HTTPS Load Balancer? If so, can you explain how I can achieve this goal?
Thank you.