How To Subscribe To Pubsubhubub

As a case study project I wanted to dynamically update a web page showcasing my YouTube channel content. The goal was to automate the process of updating the web page with new video uploads, without requiring any manual intervention or redeploying the website.

Part 2 of the real time web notification series explains how I subscribed to pubsubhubbub and developed a webhook to receive content updates for my YouTube channel.

Check out the videos page to view a summary of the latest YouTube video content uploaded to my channel.

WebSub Protocol Overview

The WebSub protocol (formerly known as PubSubHubbub) is a real-time content distribution mechanism based on the publish-subscribe (pub/sub) model. It allows subscribers to receive updates from a publisher via an intermediary called a hub. This eliminates the need for continuous polling, making content delivery more efficient.

Components

WebSub involves three key components:

• Publisher: The content provider (e.g., YouTube) that updates feeds and notifies a hub.
• Hub: An intermediary (e.g., YouTube’s PubSubHubbub service) that manages subscriptions and pushes updates to subscribers.
• Subscriber: A service (e.g., a webhook) that subscribes to updates and processes notifications.

Workflow

The diagram below illustrates how the WebSub protocol works, from subscription initiation to content notification.

Subscription Process

A subscriber initiates a subscription by sending a GET request to the hub, specifying the topic (e.g., a YouTube channel) and a callback URL where notifications should be sent. The hub responds with a challenge string, which the subscriber must echo back to confirm the validity of its callback URL. Once the hub verifies the challenge response, the subscription is successfully registered.

Content Publication and Notification Delivery

When a publisher (e.g., a YouTube channel) updates its feed, it sends a POST request to the hub with the new content. The hub then forwards the update to all registered subscribers by making a POST request to each subscriber’s callback URL. The notification payload includes an XML feed containing metadata about the new content.

Subscriber Processing and HMAC Verification

To ensure the notification is authentic, the subscriber first verifies the HMAC signature included in the x-hub-signature header. This signature is computed using the request body and a shared secret key. If the computed signature does not match, the request is rejected with a 403 Forbidden response.

If the signature is valid, the subscriber proceeds to parse the XML payload and extract key metadata, such as video title, link, and publication time. This extracted data can then be processed further, such as being stored in a database or published to another system like Google Pub/Sub.

Acknowledgement and Completion

Once the subscriber successfully processes the notification, it sends a 200 OK response back to the hub. This confirms that the notification has been received and processed, preventing the hub from retrying delivery. If any errors occur during processing, the subscriber may return an appropriate error response, prompting the hub to retry the request.

Summary

The WebSub protocol provides an efficient way to distribute content updates using a hub-based pub/sub model. By leveraging HMAC verification, it ensures notifications are secure and authentic. This approach significantly reduces the overhead of polling for updates while improving the scalability and reliability of real-time content delivery.

How To Subscribe?

Google provides pubsubhubbub for subscribing to receive YouTube notifications for updates. The curl command below highlights the subscription parameters:

Parameters

Usage

curl -X POST \
-d "hub.mode=subscribe" \
-d "hub.topic=https://www.youtube.com/xml/feeds/videos.xml?channel_id=${YOUTUBE_CHANNEL_ID} \" \ -d \"hub.callback=${CALLBACK_URL}" \
-d "hub.callback=${CALLBACK_URL}" \
-d "hub.verify_token=${HUB_VERIFY_TOKEN}" \
-d "hub.secret=${HMAC_SECRET}" \
"https://pubsubhubbub.appspot.com/subscribe

Webhook Implementation

This section explains the webhook callback handler, implemented using Node.js with TypeScript as a Google Cloud Run function. It covers the GET verification callback request and the POST notification handler that processes YouTube channel data feeds and publishes them to Google Pub/Sub.

GET Verification Callback

The GET request is sent by PubSubHubbub for subscription verification. It includes three query parameters:

• mode: Indicates whether the request is for subscribing or unsubscribing.
• challenge: A string that must be echoed back in the response.
• verify_token: A pre-agreed token used to verify authenticity. If the token does not match the expected value, a 403 error is returned.

The implementation below extracts the query parameters from the request and validates verify_token. If the token is valid, the challenge string is echoed back with a 200 response. Otherwise, a 403 error is returned.

/**
 * Handles GET requests for subscription verification using the PubSubHubbub protocol.
 *
 * This function validates the 'hub.verify_token' query parameter against the expected token.
 * If valid, it responds with the 'hub.challenge' query parameter to confirm the subscription.
 * Otherwise, it logs a warning and responds with a 403 status.
 *
 * @param req - The incoming HTTP request.
 * @param res - The outgoing HTTP response.
 */
const getHandler = (req: Request, res: Response): void => {
  const mode = req.query["hub.mode"];
  const challenge = req.query["hub.challenge"] as string;
  const verifyToken = req.query["hub.verify_token"] as string;
 
  if (verifyToken !== expectedToken) {
    logger.warn("Invalid verify token during subscription verification", {
      verifyToken,
    });
    res.status(403).send("Invalid verify token");
    return;
  }
 
  logger.info(`Subscription ${mode} verified successfully.`);
  res.status(200).send(challenge);
};

POST notification callback

The POST request is sent by PubSubHubbub when new content is published. It contains an x-hub-signature header, which provides an HMAC signature to verify the request's authenticity, and a body containing the XML data feed with YouTube video details.

The handler first extracts and verifies the HMAC signature. If the signature is invalid, the request is rejected with a 403 response. If valid, the XML payload is parsed to extract video metadata, such as the title and link. The extracted data is then published to Google Pub/Sub. If the publication succeeds, a 200 response is returned. If an error occurs during publishing, a 500 response is sent, and retries are attempted before failing.

/**
 * Processes POST requests for YouTube webhook notifications.
 *
 * This asynchronous function performs the following steps:
 * 1. Validates the HMAC signature from the 'x-hub-signature' header.
 * 2. Parses the XML payload to extract video details.
 * 3. Publishes the parsed data to Google Pub/Sub.
 *
 * It responds with appropriate HTTP status codes:
 * - 403 if the HMAC signature is invalid.
 * - 500 if an error occurs during parsing or publishing.
 * - 200 if the notification is processed successfully.
 *
 * @param req - The incoming HTTP request with the raw body and headers.
 * @param res - The outgoing HTTP response.
 */
const postHandler = async (req: Request, res: Response): Promise<void> => {
  try {
    const receivedSignature = req.headers["x-hub-signature"] as string;
 
    // Verify the HMAC signature
    if (!verifyHmacSignature(receivedSignature, req.body)) {
      logger.warn("Invalid signature detected in notification", {
        receivedSignature,
      });
      res.status(403).send("Invalid signature");
      return;
    }
 
    // Parse the XML payload to extract video details
    const parsedData = await parseYouTubeFeed(req.body.toString());
 
    // Publish valid notification to Google Pub/Sub
    try {
      const messageId = await publishToPubSub(parsedData);
      logger.info(`Message ${messageId} published to Pub/Sub.`);
      res.status(200).send("Notification received and processed successfully");
    } catch (err) {
      logger.error("Error publishing message to Pub/Sub", { error: err });
      res.status(500).send("Error publishing message");
    }
  } catch (err) {
    logger.error("Unexpected error in notification handler", { error: err });
    res.status(500).send("Unexpected server error");
  }
};