Architecture for Real-Time Dynamic Web Content

As a case study project I wanted to dynamically update a web page showcasing my YouTube channel content. The goal was to automate the process of updating the web page with new video uploads, without requiring any manual intervention or redeploying the website.

This article explains the architectural solution I developed for this use case, using serverless functions and real-time notifications to keep content up-to-date automatically.

Check out the videos page to view a summary of the latest YouTube video content uploaded to my channel.

---

Architecture

The architecture below illustrates an overview of the solution that enables a page on my website to receive real-time push notifications for new content uploaded to my YouTube channel.

A public webhook receives notifications from my YouTube channel via PubSubHubbub. The webhook is a Google Cloud Run function that processes these notifications and pushes the data to a Google Cloud Pub/Sub topic.

Another Google Cloud Run function then subscribes to this topic and uploads the data to Supabase, which leverages its real-time capabilities to notify a NextJS web page of new video data instantly, ensuring the page is always up-to-date.

YouTube WebSub Feed Updates

YouTube channels publish content updates, including new video uploads, to a WebSub topic.

These updates are made available through Google’s PubSubHubbub hub service (based on the WebSub protocol). This is the official and primary implementation for YouTube notifications, enabling subscribers to receive real-time updates via webhooks, eliminating the need for constant polling.

Refer to my article and the YouTube Push Notifications Guide for further details.

Google Cloud Run Function - Webhook

My architecture uses a public, unauthenticated Google Cloud Run function to implement the webhook that receives push notifications from PubSubHubbub for my YouTube channel. This webhook is designed to handle incoming notifications with minimal security, using HMAC (Hash-based Message Authentication Code) to verify the integrity and authenticity of the data sent from PubSubHubbub.

The webhook parses the push notification to extract video metadata and pushes the data to an internal Google Cloud Pub/Sub topic.

A service account is used to run the webhook, granting it the pubsub.topics.publish permission to publish messages to the Pub/Sub topic.

Google Cloud Pub/Sub

Once the webhook processes the push notification, it pushes the video metadata to an internal Google Cloud Pub/Sub topic. This enables seamless communication between different components of the system and ensures reliable message delivery.

Google Cloud Run Function - Subscriber

An internal Google Cloud Run function acts as a subscriber to the internal Pub/Sub topic. This function is triggered by an authenticated Pub/Sub push subscription to ensure that only authorised services receive messages from the topic.

Pub/Sub automatically handles the authentication, verifying that the subscriber has the necessary permissions to receive push notifications.

The function processes the notifications, extracts relevant video metadata and uploads the data to Supabase for storage.

A service account with the roles/run.invoker (Cloud Run Invoker) role is used to securely trigger the internal Cloud Run Function that publishes video data to Supabase.

Supabase Real-time Database

Supabase is used for storing video metadata. Furthermore, Supabase's real-time capabilities are leveraged to push real-time notifications to a NextJS web page whenever new video data is inserted into the videos table. This ensures the video data is instantly available on the web page without needing to rebuild or redeploy the site.