To ingest data to BigQuery from Pub/Sub in real-time, one must leverage Dataflow pipelines along with the schema definition and required transformations (if any). Still, the same can be a stretch if one only needs to ingest raw data or with NO transformation from Pub/Sub to BigQuery. Thus, to simplify…

Let’s clear the fog around CloudSQL Auth Proxy. Google Cloud SQL Auth proxy is a binary that provides IAM-based authorization and encryption when connecting to a Cloud SQL instance. …

As a Data Specialist Customer Engineer @ Google Cloud, I (Ravish Garg) assist a lot of our customers in their journey to Cloud or Application Modernization and how choosing the right database platform in proportion to their business requirement can make all the difference in overall user experience and ensure their edge in the competitive market. Do you know that as of Jun’2021, db-engines has listed 371 different databases in 15 different database models and if you are wondering on how to choose the right-fit database platform for your application, no matter if it’s a Greenfield application or an existing application which you are planning to modernize?

As per Harvard Business Review (HBR) less than 1% of unstructured and 50% of structured data is ever used in decision making as the organizations can’t figure out how to efficiently integrate, manage and extract value from their data. In addition, to remain competitive in the market, organizations required data…

Part-2: Build Pipeline from CloudSQL SQL Server to BigQuery In Part-1, we have taken up a scenario: “A cost-sensitive customer, who has SQL Server 2014 standard edition and looking to build a CDC pipeline to their warehouse platform i.e. BigQuery.” and mitigate the first challenge i.e. …

Part1: Online replication from On-Prem MS-SQL to CloudSQL SQL Server Why the need? Legacy versions of enterprise relational databases have been an integral part of our application stack but as the organizations are transitioning into the digital era, they start realizing the value of untapped data and the plethora of opportunities they are…

Apache Beam is a unified programming model for both Batch & Stream processing — Why do you need Data Pipelines? In the 21st century, most enterprises rely on scalable platforms & datafication of their services or products to remain competitive in the market. And with the proliferation of data from disparate sources with varying volume, velocity , and variety, Enterprises need a new data strategy. Thus, the need for data…

If you are not aware with Google Cloud Spanner database or haven’t checked my last post on Why Google Cloud Spanner, I highly recommend to check that first before proceeding further. It will just take 5mins. …

WHY GOOGLE CLOUD SPANNER ? Many organizations start small and with simplistic architecture including selection of traditional RDBMS engine, but as their product gain thrust, it unfolds the primary challenges with their database engine selection that is: Ensuring reliable performance with distributed scale. Resilient application data to system failures. Improving user experience by limiting latency.