Cloud Run Functions : Retour d’expérience FinOps

How To Build A Google Tag Manager Monitor

How to build a monitor system for Google Tag Manager, where the rate and success of tags firing on your website is logged into BigQuery for analysis and anomaly detection.

https://www.simoahava.com/analytics/google-tag-manager-monitor/#313-technology-of-choice-cloud-functions

python
from google.cloud import bigquery
import datetime
import logging

PROJECT='me-gtm-monitoring' # Update to match your project name
DATASET='gtm_monitoring'    # Update to match your dataset name
TABLE='raw_data'            # Update to match your table name

def stream_bq(uri):

    client = bigquery.Client()

    table_ref = client.dataset(DATASET).table(TABLE)
    table = client.get_table(table_ref)

    # Stream the URI of the request
    errors = client.insert_rows(table, [{'URI':uri, 
                                         'timestamp': datetime.datetime.now()}])

    if errors:
        logging.error(errors)


def gtm_monitor(request):

    if request.url:
        stream_bq(request.url)

python
from google.cloud import bigquery
import datetime
import logging
from flask import Request  # Add type hints for better readability

PROJECT = 'me-gtm-monitoring'  # Update to match your project name
DATASET = 'src_gtm'  # Update to match your dataset name
TABLE = 'raw_data'  # Update to match your table name

# Initialize the BigQuery client globally to avoid re-initialization on each request
client = bigquery.Client()

def stream_bq(uri: str) -> None:
    """
    Streams a single URI to BigQuery.
    
    Args:
        uri (str): The URI to be logged in BigQuery.
    """
    try:
        table_ref = client.dataset(DATASET).table(TABLE)
        #table = client.get_table(table_ref)  # Cache this to avoid unnecessary lookups
    
        # Insert the URI and current timestamp into the table
        rows_to_insert = [{
            'URI': uri, 
            'timestamp': datetime.datetime.utcnow().isoformat()  # Use UTC for consistency
        }]
        errors = client.insert_rows_json(table_ref, rows_to_insert)  # Use insert_rows_json instead of insert_rows

        if errors:  # Log specific errors for better debugging
            logging.error("Errors while inserting rows into BigQuery: %s", errors)
    except Exception as e:
        logging.exception("An unexpected error occurred while streaming to BigQuery: %s", e)


def gtm_monitor(request: Request) -> str:
    """
    Cloud Function entry point. Processes an incoming request and streams the request URL to BigQuery.
    
    Args:
        request (flask.Request): The incoming request object.
    
    Returns:
        str: Response message indicating success or failure.
    """
    try:
        if request and request.url:  # Ensure request object and URL are valid
            logging.info("Received request with URL: %s", request.url)
            stream_bq(request.url)
            return 'Request URL has been successfully streamed to BigQuery.'
        else:
            logging.warning("No URL found in the request.")
            return 'No URL found in the request.'
    except Exception as e:
        logging.exception("An unexpected error occurred in gtm_monitor: %s", e)
        return 'An error occurred while processing the request.'

Configure Cloud Run functions  |  Cloud Run functions Documentation  |  Google Cloud

When you deploy or update a function created with the Cloud Functions v2 API, you have access to all of Cloud Run's configuration capabilities. Use this page to understand a few of the configuration options that are available in Cloud Run, how to control the behavior of your functions, and what the best practices are for each type of configuration. See Cloud Run documentation for a full list of configuration options.

https://cloud.google.com/functions/docs/configuring/concurrency

Maximum concurrent requests for services  |  Cloud Run Documentation  |  Google Cloud

For Cloud Run services, each revision is automatically scaled to the number of instances needed to handle all incoming requests.

https://cloud.google.com/run/docs/about-concurrency

Concurrency vs. Parallelism

Concurrency and parallelism are two terms often used in relation to multithreaded applications. This tutorial explains the difference between concurrency and parallelism.

https://jenkov.com/tutorials/java-concurrency/concurrency-vs-parallelism.html

💡

En utilisant une concurrence > 1, il est important de ne plus utiliser table = client.get_table(table_ref) car chaque appel simultané augmente le temps d’exécution à cause de la latence de l’API. En cas de fort traffic, les instances vont bloquée une à une nécessitant d’en créer des nouvelles en générant des coûts supplémentaires.

• Passage d’une concurrency de 1 à 10 (nécessitant le passage de 0.167 vCPU à 1 CPU) et réduction du timeout de 60s à 15s.

• Passage d’une concurrency de 10 à 20

• Passage d’une concurrency de 20 à 1000 (la limite)