Signup pages are the first impression of any platform, and for students, they need to be more than just functional—they should be fun, engaging, and relatable. Imagine this: you’re excited to join a game or a new learning adventure, but the first thing you face is a dull, lifeless form. In today’s digital-first world, that can feel boring or even intimidating.

This is where personality makes all the difference. By making the signup process interactive and enjoyable, we can turn a routine task into a moment of delight. A friendly, playful touch not only keeps students engaged but also sets the tone for their entire journey.

Designing the Interaction: From Idea to Execution

The Idea: Engaging Students with QBit

We started by putting ourselves in the shoes of our primary users: students. We asked a simple question—how can we make the often mundane task of filling out a signup form feel exciting? The answer was clear: create a playful, interactive element that could entertain and guide students simultaneously.

Enter QBit, From the moment students arrive at the signup page, QBit springs to life:

• Curious Eyes: QBit tracks the cursor with his eyes, giving students a sense of connection and playful interaction.

• Hiding Act: When students click on the password field, Qbit shyly ducks out of view, respecting their privacy charmingly and humorously.

• Motivational Prompts: QBit nudges students with friendly reminders, like “Sign up now and join the activity!”

These small touches create a sense of delight while subtly guiding students through the signup process.

Drumroll 🥁, Please... "And Now, the Moment You’ve Been Waiting For: Let’s Build This Magic!"

1. Understanding the Math Behind the Magic

The first step in making QBit's eyes follow the cursor is calculating the angle of sight between the cursor position and the center of QBit’s eyes. Here's how:

Step 1.1: Capture the coordinates of the cursor (x1, y1) and the coordinates of QBit's eye center (x2, y2)

Step 1.2: Use the slope calculation formula to find the angle θ.

tanθ = (y2​−y1​) (x2​−x1​​)

Step 1.3: Convert this slope into an angle using the arctangent function:

θ = arctan ((y2​−y1) (​x2​−x1​​))

This angle tells us the direction in which QBit's eyes should move relative to the cursor.

2. Calculating New Eye Coordinates

Now, let’s move QBit’s eyes along the calculated line of sight by a fixed distance, say 10px. To do this, we'll use basic trigonometry:

Step 2.1: Use the angle θ to calculate the new x and y offsets:

 offsetX ​= sinθ 10
 offsetY ​= cosθ 10 

Step 2.2: Adjust these offsets by adding them to QBit’s initial eye coordinates (x2, y2) to get the final position of the eyes:

newX​ = x2​ + offsetX​
newY​ = y2​ + offsetY​

Putting It All Together

Here’s a pseudocode example for implementation:

Conclusion

Delightful UX isn’t just about aesthetics—it’s about creating meaningful, engaging moments for users. QBit’s playful interaction transformed a routine signup process into an enjoyable experience, proving that small, thoughtful details can make a big impact. By adding personality and fun, we’ve not only enhanced usability but also built a stronger connection with our student users, making their journey on Quizizz more memorable and exciting.

In a world where digital interactions often feel impersonal, adding a touch of personality can set your platform apart. QBit’s playful interaction reminded us that UX design is not just about solving problems but also about creating joy. By embracing the power of delightful UX, we’re not only enhancing functionality but also building meaningful connections with our users—one small smile at a time.

If you want to build similar or better experiences, we’re looking out for people like you. Head over to our careers page and apply to our open positions.

We're hiring!

Web browsers, particularly Chrome are finally taking a stand to reshape how cross-site tracking and authentication work. Third-party cookies, once a ubiquitous method for maintaining user sessions across different domains, are being phased out due to privacy concerns. Most privacy focused browsers already sandbox the session data like Cookies, Storage APIs to within the origin domain. But with Chrome also set to enable this, it requires web applications to adopt more privacy-conscious approaches to user authentication and data management. Though the deadline for this deprecation is delayed until early 2025, it is already rolled-out to 1% of users and the exposure is set to increase to 100% by the time the deadline is finalized.

Just to give a quick rundown of how unregulated 3rd party cookies can jeopardize your privacy, here’s what can happen,

• You visited your favourite social media XYZ, it sets a cookie

• That same social media XYZ also sells Ads

• Now any other site that you visit, if it has XYZ ads/trackers/any kind of integration, that XYZ cookie will get sent

• This way XYZ keeps track of anything and everything you do.

I’m sure nobody wants that to happen, so 3PCD is definitely a need to avoid this from happening, but the collateral damage for doing that is benevolent actors who use 3rd party cookies for legitimate use cases.

Quizizz is one of those, where the content embedded in 3rd party websites is a critical use case for us, so we decided to pre-empt and prepare for this deprecation well ahead of the timeline. While there are many approaches to handle this deprecation, we decided to implement CHIPS and use Storage Access APIs to enable our integrations to work seamlessly in a 3rd party context.

CHIPS: Cookies with Independent Partitioned States

What is CHIPS?

Cookies Having Independent Partitioned States (CHIPS) is a browser mechanism that isolates cookies for different top-level sites. Think of it like creating separate, secure compartments for cookies from different websites, preventing cross-site tracking while maintaining necessary functionality. This would ensure there are separate cookie jars for each origin and 3rd party domain combinations so that there is no cross site cookie access. The following is a simplified implementation of how to set CHIPS from the backend.

Core Implementation

public generatePartitionedCookie (response, params) {
  const { userId, domain, expiresIn = 86400 } = params;

  // Generate your session cookie value here
  const cookieValue = this.generateSecureToken(userId);

  // Create a CHIPS-compatible cookie configuration
  return response.cookie({
    name: 'quizizz_session_id',
    value: cookieValue,
    domain,
    maxAge: expiresIn,
    sameSite: 'none',  // Allows cross-site access
    secure: true,      // HTTPS only
    partitioned: true  // Enable CHIPS
  });
}

Advantages

• Prevents cross-site tracking

• Maintains session continuity for a given origin and 3rd party domain combination

• Provides granular control over which context the cookie is stored

• Enhances user privacy

Storage Access API: Explicit Cross-Site Permission

Understanding the Storage Access API

The Storage Access API provides a transparent mechanism for websites to request explicit permission to access storage across different domains. Unlike traditional third-party cookies, this approach gives users more control and visibility into how their data is accessed.

Implementation Strategy

class StorageAccessManager {
  public async requestStorageAccess(): Promise<boolean> {
    try {
      // Check API availability
      if (!document.hasStorageAccess) {
        return false;
      }

      // Request cross-site storage access
      const accessGranted = await document.requestStorageAccess({
        types: ['cookies', 'localStorage', 'sessionStorage']
      });

      return accessGranted;
    } catch (error) {
      // Handle access request failures
      return false;
    }
  }
}

This provides an explicit consent mechanism where the user can decide to choose to give access to cross site data. In most cases, critical information wouldn’t be kept in browser storage but accessing these stores when consent is not given is a problem as it will throw an error like below,

Access to localStorage failed: SecurityError: Failed to read the 'localStorage' property from 'Window': Access is denied for this document.

Alternative Approaches to Cross-Site Authentication and Storage access

While Quizizz has primarily implemented CHIPS, several alternative strategies exist for managing cross-site authentication and storage in the era of third-party cookie deprecation.

1. Token based authentication

Token-based authentication is a common way to overcome third-party cookie deprecation (3PCD) restrictions, particularly in web applications that rely on being able to access inside a 3rd party context. Here's a concise explanation of how it would work

• Token Issuance:

  • The authentication server validates the user provided credentials.

  • If valid, it generates a JSON Web Token (JWT) or other token types (e.g., OAuth access token) and sends it back to the client in the response.

• The client stores the token securely, typically in:

  • LocalStorage (persists across tabs but is prone to XSS).

  • SessionStorage (limited to the tab but less persistent).

  • In-Memory (best against XSS but cleared on refresh).

• For each subsequent request to a protected resource, the client includes the token in the Authorization header

import * as jwt from 'jsonwebtoken';
import * as crypto from 'crypto';

class SimpleJWT {
  private readonly SECRET_KEY;
  private readonly ISSUER: string = 'quizizz.com';

  constructor() {
    // Use a secure key for production instead
    this.SECRET_KEY = crypto.randomBytes(64).toString('hex');
  }

  public generateCrossDomainToken(options) {
    const { userId, userRole, domains } = options;

    return jwt.sign({
      sub: userId,           // Subject (user identifier)
      role: userRole,        // User role/permissions
      aud: domains,          // Intended audience (domains)
      iss: this.ISSUER,      // Token issuer
      exp: this.getExpiration(), // Expiration time
      iat: Date.now(),       // Issued at time
      jti: this.generateJTI() // Unique token identifier
    }, this.SECRET_KEY, {
      algorithm: 'HS256'
    });
  }
}

Along with a custom authentication implementation, we can also leverage OpenID Connect (OIDC) which provides a cookie-free, token-based approach to user authentication. It relies on OAuth 2.0 to handle secure token exchange, enabling applications to authenticate users without relying on third-party cookies.

Here’s how you can integrate your application with Google OIDC

const { Issuer } = require('openid-client');

// Discover the OIDC provider's configuration
Issuer.discover('https://accounts.google.com') // Replace with your provider's URL
  .then((googleIssuer) => {
    const client = new googleIssuer.Client({
      client_id: 'YOUR_CLIENT_ID',
      client_secret: 'YOUR_CLIENT_SECRET',
      redirect_uris: ['http://localhost:3000/callback'], // Adjust to your setup
      response_types: ['code'],
    });

    // Redirect users to the authorization URL
    const authUrl = client.authorizationUrl({
      scope: 'openid email profile',
    });

    // Exchange the authorization code for tokens (e.g., in a callback route)
    app.get('/callback', async (req, res) => {
      const params = client.callbackParams(req);
      const tokenSet = await client.callback('http://localhost:3000/callback', params);
      console.log('Tokens received:', tokenSet);

      const userInfo = await client.userinfo(tokenSet.access_token);
   // Handle user session with the information obtained using the open ID token
      res.send('Authentication successful');
    });
  })
  .catch((err) => console.error('Error with OIDC setup:', err));

Similar to google, we can use any OIDC identity providers like Microsoft, Okta, Auth0 etc., we can refer to all available certified OIDC implementations here.

On top of OIDC being a tested standard for authentication, once we’ve added support for authenticating users through OIDC in an application, it makes it trivial to support more SSOs going forward.

2. Federated Identity Management (FedCM)

FedCM is a privacy-preserving alternative to third-party cookies that enables federated identity login flows without relying on cross-site cookies. It facilitates authentication through identity providers (IdPs) while adhering to modern web privacy standards.

How FedCM Overcomes 3PCD Restrictions

• No third-party cookies: It uses browser-mediated communication between Relying Parties (RPs) and IdPs.

• User consent: The browser ensures that users are aware of and consent to sharing their identity.

• Reduced tracking: Only essential data is shared, minimizing potential user tracking across sites.

Basic Workflow

1. User initiates login on the Relying Party (RP) website.

2. RP requests identity from the browser using FedCM.

3. Browser contacts the IdP to fetch identity information.

4. Users consent to sharing their identity.

5. IdP returns a token (e.g., OpenID token).

6. RP verifies the token and logs in the user.

1. Client-Side: Request Identity

The RP (your web app) initiates the authentication flow using the FedCM API.

async function loginWithIdentityProvider() {
  try {
    const result = await navigator.identity.get({
      provider: 'https://idp.example.com',
      // Optional: Specify a nonce for additional security
      nonce: 'random-string-for-verification',
    });

    if (result) {
      // Send the token to your server for verification
      await fetch('/api/auth/callback', {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
        },
        body: JSON.stringify({ token: result.token }),
      });

      console.log('User successfully logged in.');
    }
  } catch (error) {
    console.error('Login failed:', error);
  }
}

2. IdP Implementation

The identity provider serves a configuration endpoint for the browser and handles authentication.

• /.well-known/fedcm.json (IdP metadata configuration):

{
  "id_assertion_endpoint": "https://idp.example.com/assertion",
  "accounts_endpoint": "https://idp.example.com/accounts",
  "branding": {
    "background_color": "#FFFFFF",
    "icon_url": "https://idp.example.com/logo.png"
  }
}

• an Endpoint for identity assertion

app.post('/assertion', (req, res) => {
  const { clientId, nonce } = req.body;

  // Validate client and nonce, then issue a token
  const token = generateOpenIDToken({
    sub: 'user123',
    aud: clientId,
    nonce,
  });

  res.json({ token });
});

3. Server-Side: Validate Token

The RP’s backend validates the token received from the client.

const jwt = require('jsonwebtoken');

app.post('/api/auth/callback', (req, res) => {
  const { token } = req.body;

  try {
    const decoded = jwt.verify(token, 'your-public-key', {
      audience: 'your-client-id',
      issuer: 'https://idp.example.com',
    });

    // User is authenticated
    req.session.user = decoded.sub;
    res.status(200).json({ message: 'Login successful' });
  } catch (error) {
    res.status(401).json({ error: 'Invalid token' });
  }
});

Key Benefits of FedCM

• User privacy: Reduces tracking by limiting shared information.

• Improved UX: Native browser consent flow simplifies login.

• No reliance on third-party cookies: Secure and privacy-focused federated login.

Limited Support

FedCM is still under development and adoption. It works in Chromium-based browsers with certain flags enabled (chrome://flags/#fedcm). But other major browsers like Firefox and Safari do not support it.

3. Related Website sets

Related Website Sets is a privacy-preserving mechanism introduced by Google to help websites that are part of the same organizational group maintain cross-site functionality after third-party cookies are phased out. Think of it like a family pass for websites owned by the same company.

Key Concepts:

• Allows a set of related websites to share limited cross-site access

• Helps maintain functionality for sites with related domains

• Requires explicit declaration and verification by site owners

Here’s a practical example to illustrate how to implement Related Website Sets:

// Registering a Related Website Set
async function registerRelatedWebsiteSet() {
  if ('relatedWebsites' in navigator) {
    try {
      // Declare primary site and associated sites
      const registrationResult = await navigator.relatedWebsites.register({
        primary: 'https://main-site.com',
        associatedSites: [
          'https://analytics.main-site.com',
          'https://payments.main-site.com',
        ]
      });

      // Check registration status
      if (registrationResult.status === 'success') {
        console.log('Related Website Set registered successfully');
      }
    } catch (error) {
      console.error('Related Website Set registration failed:', error);
    }
  }
}

// Check if a site is part of a Related Website Set
async function checkRelatedWebsiteSet() {
  if ('relatedWebsites' in navigator) {
    const isRelated = await navigator.relatedWebsites.isRelatedWebsite();
    console.log('Is this site part of a Related Website Set?', isRelated);
  }
}

Implementation Steps:

1. Site owners must declare their Related Website Set through Google Search Console by hosting a file at /.well-known/related-website-set.json similar to the following under your main domain,

{
  "primary": "https://main-site.com",
  "associatedSites": ["https://analytics.main-site.com", "https://payments.main-site.com"],
  "description": "Example Related Websites"
}

2. Websites must meet specific criteria (same organization, related purpose)

3. Use the navigator.relatedWebsites API to register and check set membership

Limitations:

• Limited to a maximum of 5 associated sites

• Requires explicit registration and verification

• Not a complete replacement for third-party cookies

Quick Compatibility Testing

Follow these steps to quickly see if your web application is compatible with the guidelines or not. Run these diagnostics when your web application loads inside an iFrame or is embedded within another website.

1. Simulate Third-Party Cookie Restrictions

Use Chrome’s built-in flags to simulate third-party cookie restrictions, you can also find the same flag if you visit chrome://flags

# Chrome launch command with third-party cookies blocked
google-chrome --disable-third-party-cookies

2. Basic Compatibility Diagnostic Script

function diagnoseThirdPartyCookieImpact() {
  try {
    document.hasStorageAccess().then((hasAccess) => {
      if (hasAccess) {
    	// storage access has been granted already.
      } else {
    	// storage access hasn't been granted already;
    	// you may want to call requestStorageAccess().
      }
    });
  } catch (error) {
    console.error('Cookie storage test failed', error);
  }
}

window.addEventListener('load', diagnoseThirdPartyCookieImpact);

Conclusion

Adhering to third-party cookie deprecation guidelines requires a thorough examination of cookie and storage usage of web applications. By implementing one of these authentication strategies, platforms can ensure they maintain a predictable user experience across different domains.

If you found this interesting and want to solve similar interesting, deep technical problems, we’re looking for you at Quizizz. We’re looking for folks across Frontend, Backend, DevOps roles. Apply below!

See Open Roles

References

• https://developers.google.com/privacy-sandbox/cookies

• https://developers.google.com/privacy-sandbox/cookies/prepare/audit-cookies

• https://developers.google.com/privacy-sandbox/cookies/prepare/test-for-breakage

• https://developers.google.com/privacy-sandbox/cookies/chips

• https://developers.google.com/privacy-sandbox/cookies/storage-access-api

• https://developer.mozilla.org/en-US/docs/Web/API/FedCM_API

• https://github.com/GoogleChrome/related-website-sets/blob/main/RWS-Submission_Guidelines.md#set-formation-requirements

As you read through, if you have any feedback for us, we’d love to hear from you in the comments below!

Data Sources

Let’s start off with our data sources at Quizizz. Primarily, there are two categories of data sources that we consume:

• Internal Data: This contains all the data generated by Quizizz, primarily derived from usage of the product

• External Data: This is the data we source from all the 3rd party tools we use

Internal Data

Our platform’s data backbone is MongoDB, with additional layers from:

• Cassandra: Stores in-product reporting data

• Redis: Temporary store for caching as well as other use cases

• ElasticSearch: Serves search results and recommendations

• Clickhouse: Serve real-time analytics to our end users

• Pinecone: Recent addition to power AI features

Most of our internal data is ETL'd from MongoDB. The system reads chunks of Mongo Operations log (oplog) once every 30 minutes and writes to BigQuery. The data is not modeled in any way, and lands in bigQuery as a JSON object in a single column.

External Data

Beyond our own platform, we leverage a powerhouse lineup of external tools that keep our data ecosystem dynamic and robust.  We use:

• Stripe

• Salesforce

• Hubspot

• Braze

• Zendesk, just to name a few

To bring all this data together, we rely on Fivetran for data integration. It’s our go-to for ETL, for pulling data into our warehouse from these various sources, though we have had some twists along the way like migrating from Stitch to address model completeness (for instance, we always struggled to get full Stripe subscription data).

One heads-up: Fivetran’s billing based on Monthly Active Rows (MARs) keeps us on our toes with its dynamic pricing, making it tough to nail down monthly costs. We’re learning as we go, and here’s a quick tip we picked up: 

Tip: Fivetran doesn’t charge for the first 2 weeks of a new sync, so explore every table during this grace period! Once you see what’s useful, refine your selections to avoid unnecessary costs — selecting all tables could mean a surprise bill if unchecked; a possible bankruptcy, too!

For certain elements which cannot be synced directly via Fivetran, we have our own custom scripts to fill in the gaps.

Transformations

Once data lands in our warehouse, the magic really begins!

Using dbt (Data Build Tool), we streamline and structure our data so it’s not only just useful but also impactful. With dbt, our team handles transformations in BigQuery without relying on engineering for every change. This flexibility means we can adjust and optimize data models quickly, ensuring the freshest insights are just a query away. 

As of late 2024, our pipeline boasts about 800 data models, organized into two core "layers":

- The Base Layer: This is where raw data gets its first polish. Think of it as our prep station for everything that enters BigQuery. It’s also where we tackle JSON files, ensuring naming consistency, data types, and streamlined formats to support smooth downstream analysis. 

- The Consumption Layer: This is where each team can access insights tailored to their specific needs. With datasets aligned to pods and team use cases, this layer empowers analysts across Quizizz to find the data they need and build reports that drive real results.

Both layers refresh within the same command in the same dbt job, and their refreshes are sequential.

The cost for refreshing both layers is about $150/day, a small investment for the clarity it brings across teams.

Self-Serve Analytics

Now, we can make more sense with the data we have and start doing analytics on top of it. We use a variety of tools to make it seamless for people from different functions in the company to be able to get answers to their questions.

To make this possible, we host a self-managed instance of Metabase, chosen for its intuitive query builder that anyone can use. After trialing Superset and Redash, we landed on Metabase based on its user-friendly design and internal user feedback. Today, teams create and curate dashboards, pull insights on demand, and answer many of their data questions without waiting on analysts. With enriched fields and ready-to-use aggregates in our clean dataset, we’re seeing more autonomy and more impactful decisions driven directly by data.

Fun fact: Self-hosting Metabase via AWS Elastic Beanstalk costs us just $70/month. With the average querying cost around $25/day, it’s been a budget-friendly way to democratize data across the org.

Custom Tools

For those projects needing a bit more power, we turn to Hex, a platform that combines the versatility of a Python environment with an interactive dashboard setup. This lets us create internal tools that respond to user input, like our: 

Funnel Builder

Built atop the modeled events data, this is an easy-to-use dashboard that helps Product Managers and Designers visualize product funnels. They can filter on various user/event attributes. This has greatly reduced the dependency of product stakeholders on product analysts to understand our user funnels.

Experimentation Dashboard

This dashboard lets product stakeholders easily access A/B test results. We’ve curated our L0 metrics and given the flexibility to perform A/B tests on conversion rates between any 2 events or page views. The dashboard also renders the underlying SQL queries for others to augment them for their special use cases.

Reverse ETL

With rich user data now in our warehouse, we can also now power data flowing back into the 3rd party tools we use, empowering multiple functions like GTM, sales, and success teams with powerful metrics to engage with our users more effectively. This way, our teams have up-to-date insights right where they need them, keeping communication targeted and relevant.

The logic for the enrichment is maintained in dbt, in our reverse_ETL dataset. While tools like Hightouch and Census offer reverse ETL solutions, we found their pricing models a bit priced for the value they provide. Instead, we handle this with Python and Golang scripts customized to our specific needs.

What’s next?

We’re just getting started. As we look ahead, our focus is on taking our data capabilities to the next level, adding both depth and efficiency to our processes:

More Product Analytics Tools

We’re working on releasing more interval analytics tools such as cohort builder, trend builder, and product flow builder to empower all functions to be able to do more.

Orchestration Framework

The warehouse is becoming more ubiquitous, with pipelines making a stop via it. Managing cron jobs is becoming more difficult and we want to move away from the current model.

With these projects in the pipeline, we're gearing up for an exciting close to 2024, strengthening Quizizz into a more data-powered platform for our teams, educators, and students worldwide!

We’re on the lookout for people who can help us build the next version of our Data Platform as well as our core product. If you are passionate about building great consumer products, have a knack for tinkering with newer technologies and you like to work with great minds, Quizizz is the place to be.

We’re hiring across all functions and levels within engineering and we want you to be part of this amazing journey ahead.

Our Careers page

In this blog, we'll discuss how we started building AI-based products at Quizizz. We will start with a basic understanding of how these models work followed by some simple and effective prompt engineering techniques.

Understanding Large Language Models (LLMs)

LLMs, such as GPT-4o or Claude 3.5 Sonnet, are language models trained on vast amounts of data. Their primary job is to predict the next word (token) in a sequence, much like an advanced autocomplete system.

LLMs analyze the context provided by previous words to predict what should come next. This predictive ability makes them effective at understanding language and following instructions or answering questions.

Most LLMs undergo fine-tuning on specific datasets, which enhances their abilities for various tasks. For example, models used by ChatGPT are further trained on conversation-based datasets to make them proficient in handling interactive dialogues.

Choosing the Right Model

Our primary use case at Quizizz was to reduce the amount of time and effort a teacher would have to spend in creating their assessment. We wanted to make sure that we use the right model that was lightweight, fast, inexpensive and is effective at generating the right content.

Here are some common options:

• OpenAI’s GPT Models

• Anthropic’s Claude Models

• Google’s Gemini Models

• Meta’s Llama Models

Smaller models are faster and cheaper than larger models but are generally less capable than larger models in complex tasks

We chose OpenAI’s GPT models while we were experimenting with integrating AI with Quizizz. GPT-3.5 was one of the first models we integrated with and gave us decent results. But as we started getting better models like GPT-4 and then 4o, we were quick to experiment and switch. We kept using GPT-3.5 for simpler tasks. We wanted to always optimize for cost without compromising on quality ensuring that teachers still had a great experience with AI.

Getting the Most from Your Model: Prompt Engineering

Prompt engineering is the art of designing effective prompts to get the best response from an LLM. A well-crafted prompt can significantly enhance the quality of the output.

Be Direct and Specific

At Quizizz, we started with a simple prompt to create questions based on the subject/topic entered by the teacher. This was one of the earliest versions we used which gave us great feedback from the teachers on how we could improve it.

It’s important to keep the prompt as simple as possible for the model to understand and then act upon. Few tips on how to be direct and specific:

• Avoid ambiguous words and phrases

• Specify the desired response format, the expected length, and the tone of the prompt.

• Be as objective as possible.

• Do not mention redundant details in the prompt.

• Break down the task into smaller and easier sub-tasks.

• Do not ask leading questions if you don’t want leading answers.

• Instead of saying what not to do, say what to do.

Few-shot Prompting

One of the things that teachers want their students to have is mastery over what they are learning. And they usually do that by providing answer explanations after a problem is attempted. We started generating answer explanations given a question based on the techniques mentioned above. While that worked well, the answer explanations were not very consistent and sometimes had mistakes.

That’s where Few-shot Prompting came in. It involves providing the model with input-output pairs that it can use to infer the structure and tone of the desired response.

For our specific use case, our in-house experts categorized different types of concepts and manually created examples for those. Whenever an answer explanation is supposed to be created, we retrieve the relevant examples and pass them to the model for generating an answer explanation of the new question.

One-shot prompting is a technique where one input-output pair is provided as an example.

Chain of Thought Prompting

At Quizizz, we deal with a lot of Math-related use-cases, and one of the biggest problems with that it requires reasoning. Asking an LLM to answer a Math question often results in incorrect answers. Chain of Thought Prompting is a technique that tries to induce reasoning in the output of the model thereby helping it break the complex task into multiple smaller tasks that it is good at.

Asking a model to “Think step-by-step” or “Explain the reason before answering” are a few ways to do Chain of Thought Prompting. We can also use Few-shot Chain of Thought Prompting where the examples help the model on how the thought process should look like.

Conclusion

The first step to building an AI-based product is building a basic understanding of the foundational technologies and learning how to utilize them effectively. This article should provide a great starting point, and with careful prompt engineering, you can leverage these models to create compelling AI features. Everything boils down to experimentation and figuring out the right model and prompt which works for your use case.

In the next part of this series, we will go beyond the basics of prompting to explore advanced techniques that make AI products more capable and reliable.

If you’re interested in leveraging AI to build products that improve the daily lives of educators and learners, we want you! We’re looking for backend and frontend engineers who have a high sense of ownership, are curious, and are willing to step into the world of a fast-paced startup. Apply to the open roles using the link below.

Join Us!

Originally published on July 10, 2023

Introduction

At Quizizz, our mission is to empower educators to engage learners to help improve their learning outcomes. A core part of our platform is the ability for teachers to conduct live gamified quizzes in classrooms. Since these activities are happening live, it is essential to have no disruptions and a smooth and frictionless user experience for every student.

One of the most pressing problems that we have been facing was DoS (Denial of Service) attacks. In this blog post, I’m going to talk about what we did in order to negate the consequences of DoS attacks and our learnings from this effort. 

Our current system

As we have been growing and amassing new users, the frequency of such attacks is also growing. In our initial days, we found quicker and cost-efficient solutions to manage these attacks. Our quick solutions were to use AWS WAF (Web Application Firewall) and custom application-level rate limiters in each service.

As the frequency and complexity of these attacks increased, and our infrastructure became more complex, we had to find a permanent solution. Our current solutions still let DoS attacks through, mostly due to WAF’s 5-minute window being too long for detecting DoS attacks before they propagate to our services. Additionally, WAF has started becoming prohibitively expensive as our traffic increases as well.

Problems

Failure to block all DoS attacks

We have been facing DoS attacks for some time now, AWS WAF (Web Application Firewall, a managed DoS protection service by AWS) wasn’t fully effective in stopping these attacks and quite a few of them would leak into our system, which would cause scaling up of our services, high resource/CPU usage on individual containers, and would have cascading effects into our entire system, generally resulting in instability to users. 

Application level rate limiters and quick fixes proved ineffective since attackers would change strategy and start attacking different endpoints/services. We needed an effective single service in front of all our services to handle DoS attacks by rate limiting based on multiple attributes, such as login state, IP address, HTTP method/path/headers, etc.

Duplication of logic in multiple services

In our current infrastructure, traffic is routed to different load balancers behind our CDN. 

While this is a simpler way of building our platform, this has a huge downside. We have little to no control over how requests are routed, and we have no place to write business logic, such as authentication, which is common to all services.

The result of this is that over time individual microservices started implementing their own versions of the same business logic. Changes to the common layers require similar changes in all services which can over time cause bugs and increase tech debt. These bugs can have severe consequences such as security vulnerabilities. 

The solution is to have a common entry point, the gateway service, to the system which serves as a middleware to all our requests.

Heavy reliance on AWS restricting customizability, ballooning costs

We use AWS Application Load Balancers for a number of interesting platform features that enhance developer experience and provide developers with ways to test and experiment with their code without releasing it to actual users. This strategy has worked out so far but as we continue to grow quickly, we have been facing issues with AWS services quota limits, and costs. 

We realized that to continue forward in our growth, providing speed to developers, and ensuring Quizizz remains a reliable and highly available platform while also ensuring cost efficiency, we’d need to move parts of our platform away from AWS services towards cloud-native solutions.

We decided to move forward with building a custom gateway service with our own implementation of a rate limiter, deploying it on Kubernetes while using a cloud-native tech stack.

Gateways, Kong: What are these?

An application gateway is a networking component that acts as an intermediary between users and web applications. It provides secure and scalable access to web applications by offering functionalities such as load balancing, SSL/TLS termination, and web application firewall (WAF) capabilities. It serves as a central entry point, routing incoming requests to the appropriate backend servers, and can also provide features like session affinity and URL-based routing. Overall, an application gateway helps improve performance, reliability, and security for web applications by managing traffic and enforcing policies at the application layer.

Kong Gateway is an open-source, cloud-native API gateway that acts as a central point of control for managing and securing API traffic. It enables organizations to efficiently manage their APIs by providing capabilities such as authentication, rate limiting, request/response transformations, and traffic control. Kong Gateway acts as a middleware layer between clients and backend services, ensuring reliable and secure communication. With its extensive plugin ecosystem, it offers flexibility and extensibility to customize and enhance API functionality. By simplifying API management tasks, Kong Gateway empowers developers to focus on building robust and scalable applications while ensuring high performance and security for their APIs.

There is a lot more to say about Kong than what I have summarized and if you’re interested, this is a great start.

Kong needs a persistent data store where it can store rate limits, while there are many options for these, one of the more popular options is Redis, which we used.

Why we built our own rate limiter

Kong provides a default rate limiter, but it didn’t fill our use case completely. There were many features we wanted that required custom implementation, such as:

Redis Cluster Support

Quizizz gets ~750k RPM peak traffic, and DoS attacks can go into 10s of millions of RPM. This meant we can’t rely on a single Redis node to guarantee high availability and performance. We wanted a large Redis cluster to ensure Redis didn’t crash during DoS attacks.

Configurable Time Windows

The default rate limiter provides only a small set of time windows on which rate limits can be configured. For our use case, we realized that a one-minute window is too long, and within that period of time, an attacker can cause harm to our system, and hamper other user’s experience. A one-second window, while important, would not be enough since we’d like to block attacks for a longer period of time. This meant we wanted configurability on n second windows. 

IP Blacklisting/whitelisting

Blacklisting and whitelisting IP addresses is an important part of our rate limiter that can power multiple features. For example, we need to whitelist our own custom IPs. We may also whitelist IPs for certain users if they have special cases that require rate limits to be relaxed for a small period of time. Furthermore, we are working on improving our system so that it provides a captcha instead of an error when a certain IP is rate limited. If the IP completes the captcha, we can then whitelist it for a short period of time. Another interesting feature is to blacklist certain IPs which we know are malicious.

Custom logic for different rate limits

While we want to block DoS attacks, we also want to ensure that this doesn’t hamper the experience of legitimate users. Legitimate user requests follow certain patterns that DoS attacks generally don’t, for example, DoS attacks generally use unauthenticated requests, hit the same endpoint, etc.

There is no definitive way to tell whether certain activity is legitimate or not, and we cannot write a rate limiter that would never block a legitimate user request, but we can understand how users use our platform, and build a rate limiter that minimizes the probability that a user request is returned with a 429 response.

Fault tolerance

One of our core emphasis while building this service was to ensure that requests don’t get blocked in the case of a failure in Gateway service. To put it simply, we understand that code can and will fail, but it shouldn’t impact how users experience on our platform.

This meant we had to ensure fallbacks and write custom code to handle anything and everything that can happen. So for example, if Redis crashes, requests shouldn’t fail and instead start routing without the rate-limiting logic. The default rate limiter wouldn’t handle all such cases and hence we had the need to build our own.

Architecture

We created staging deployments based on a custom header using Cloudfront’s continuous deployment feature. This allowed us to initially start sending our own traffic to Kong. Once we were confident, we started sending small amounts of user traffic based on percentage on a single route, gradually increasing traffic percentage. All this while, we started doing small load and simulation tests on the gateway service while monitoring it to see how it would react.

Cloudfront staging distribution only allows up to 15% of traffic to be routed to the staging deployment. To increase it beyond 15%, we used Lambda@Edge. We wrote a lambda function that would change the origin of the request using custom logic. Using this, we were able to configure request routing very granularly, constantly updating it based on our requirements and testing. After gradual increases over a long period of time, we finally released gateway service to 100% of traffic with the older CDN distribution 

All this time, we had enabled shadow mode on our custom rate limiter, where it would not block any requests, instead updating a metric when a certain IP crosses the rate limit. Using this, we were able to fine-tune our rate limiter logic and decide on time windows and rules. Eventually, we disabled shadow mode once we had finalized the rate limiter rules.

To scale pods, we used Horizontal Pod Autoscaler(HPA), and to scale nodes, we used Cluster Autoscaler(CAS)

Roadblocks

Out Of Memory (OOM) Restarts

During testing Kong, gateway service pods failed to restart after crashing due to OOM errors. This was mostly caused by a bug in Kong due to which containers wouldn’t delete a socket connection file after dying, and hence new containers couldn’t spin up since the ports wouldn’t be open. This bug was fixed in the later versions of Kong. 

Conclusion

In summary, our current system was vulnerable to DoS attacks and regular attacks causing degradation of user experience while increasing infrastructure costs. AWS WAF was proving to be ineffective against larger attacks due to its long 5-minute time window. We implemented a common gateway layer using Kong which could rate limit requests on configurable time windows. This also allowed us to move common business logic from multiple services to a single service, therefore cleaning up our architecture while reducing the occurrence of DoS attacks. While this has not completely solved the problem for us, it was a good first step in the right direction.

Come Join Us!

If you’re passionate about solving similar problems, have experience in building platform systems, reach out to us and help us the next challenging problem. Reach out to us, and let’s build the best learning platform in the world!

We're hiring

At Quizizz, we have been using MongoDB as our primary database and Google BigQuery as our analytics data warehouse. The ability to effectively sync data from MongoDB to BigQuery has played a vital role in driving product and business decisions. Our data ingestion pipeline has been syncing MongoDB data to BigQuery for the past few years. In this article, we will explore the challenges we faced with our previous ingestion pipeline, discuss the decisions we made while building our new data ingestion pipeline, and delve into the details of how this pipeline has empowered us to make data-driven decisions at Quizizz.

Old Data Ingestion Pipeline

Architecture

Our old data ingestion pipeline had the following architecture:

• Events were triggered from code after DB changes (Create, Update, and Delete).

• These events were added to SQS (Amazon’s managed queuing service) so that they could be asynchronously processed without affecting our live flow.

• Consumers would read events from SQS and populate temporary BigQuery (BQ) tables. There are temporary tables for each entity, with one table for each 6-hour block.

• Every 24 hours, a cron job would merge the data from the temporary BQ tables for each entity into the main BQ table for that entity. This batch-processing method aimed to optimize costs by reducing the frequency of table updates.

Challenges with Old Data Ingestion Pipeline

tl;dr: Data Loss and Unreliable Data

Code-triggered Events

In our old pipeline, events were manually triggered from code during create, update, and delete operations. However, this approach had limitations, primarily, the risk of missed events due to code bugs.

Lack of fault tolerance

Our old pipeline was based on SQS. The system was fairly simple without a DLQ resulting in the absence of a robust retry mechanism in case of failures. This lack of fault tolerance made us vulnerable to data loss during transient failures and/or intermittent network issues. To enhance the reliability of our pipeline, we prioritized the implementation of a comprehensive retry mechanism (covered later).

Manual Schema Updates

Another challenge we encountered was the manual synchronization of schema changes between our primary database and BigQuery. This process required human intervention and was prone to errors, leading to data loss. We aimed to streamline this process and ensure data consistency by automating schema updates or removing the need for schema updates.

Backfill and Evolving Data Requirements

We do not index documents in any of the collections on the created date in MongoDB. Accommodating evolving data requirements and performing backfill operations proved to be challenging in our previous pipeline as querying based on dates would be very expensive. Due to changing business needs, we had to change the data type of certain fields in the past. Having strict data types for fields in BQ would result in errors when backfilling older events. We needed a more flexible solution to adapt to changing data needs without disrupting workflows.

Loss of Historical Data

In the previous architecture, we only stored the current state of each entity, disregarding the change events. This lack of historical data limited our ability to perform comprehensive analysis and gain insights into the evolution of our data over time.

Introducing the New Data Ingestion Pipeline

tl;dr: Change Data Capture-based Kafka Connect pipeline

Change Data Capture

To address the limitations of our old pipeline, we decided to use Change Data Capture (CDC) for data ingestion. CDC is a widely adopted standard for syncing data from a primary data source to a data lake, data warehouse, or any other secondary data source.

Debezium (an open-source CDC framework) captures change events (creates, updates, deletes) from various databases and adds the change events to Kafka. Consumers can read from Kafka and react to those change events. In our case, Debezium captures change events from MongoDB in real time, allowing us to keep our data warehouse, BigQuery up to date. We used Debezium Kafka Connector for this.

Using CDC helped us solve the problems related to triggering events from code.

Fault tolerance using Kafka

Because of Debezium, we started using Kafka instead of SQS. Because of Kafka, we got more control in terms of error handling. Consumers could now implement retries with exponential backoff. If a message could not be processed due to non-retriable exceptions, Kafka Connect tasks would fail and we get alerts. This helped us make the pipeline fault-tolerant.

Every component in the pipeline is decoupled and can be scaled independently. Offsets are maintained for every component (including the change stream offset) in Kafka making it easy to restart any of the components in case of issues.

Agile Data Storage with JSON State

As part of our new pipeline, we made the strategic decision to store the state data as a JSON string within a field called “state” for each event row. This approach allows us to store the entire state of each entity instead of specific fields at a given time. By doing so, we eliminate the need for backfilling operations as the data is always complete. Furthermore, this approach helps us avoid schema updates, enhancing our agility and reducing disruptions.

Streaming Change Events to BigQuery

In our new data ingestion pipeline, we have introduced a more efficient approach. Each collection in MongoDB now has a dedicated events table in BigQuery. The change events are streamed directly to BigQuery and stored as rows in the corresponding table. The new tables follow an append-only model, ensuring an efficient and reliable data flow. This also helped us store historical data and make the analytics data available in real time. We used BigQuery Kafka Connector for this.

Easier onboarding of new tables

In the new pipeline, onboarding a new table requires just adding the name of the collection in the configuration and data would start flowing ASAP. Backfilling data for any table would require adding an entry in a signal table and everything gets handled.

Other Use Cases

Given the reliable and fault-tolerant nature of Kafka and Kafka Connect as well as the usefulness of CDC events, this change opens up possibilities for expanding use cases beyond syncing data to BigQuery. Some potential use cases that can benefit from a similar pipeline include:

• Syncing Data to Elasticsearch: With our pipeline’s scalability and reliability, we can seamlessly sync data from MongoDB to Elasticsearch by adding a new sink connector for ES. This eliminates the sync issues often encountered with code-based triggers in our current Elasticsearch ingestion pipeline.

• Triggering Real-time Campaigns: The flexibility of our data ingestion pipeline allows us to sync data to platforms like Braze or other similar systems. By capturing and streaming change events, we can trigger real-time emails or campaigns based on user interactions and events, delivering personalized experiences to our users.

• Building a Data Lake with S3: Our new pipeline sets the foundation for building a data lake by syncing data to Amazon S3. Storing the complete state as a JSON string enables seamless integration of MongoDB change events into our data lake. This empowers our machine learning pipelines, leveraging the scalability and processing capabilities of S3.

Conclusion

Building a scalable and agile data ingestion pipeline is crucial for organizations aiming to make data-driven decisions effectively. At Quizizz, we embarked on a journey to transform our pipeline, overcome challenges, and leverage industry-standard solutions.

Our new data ingestion pipeline empowers us to capture and sync change events from MongoDB in real-time, ensuring data accuracy and eliminating missed events. This new pipeline is playing a pivotal role in empowering Quizizz to make data-driven decisions that drive our product and business growth.

If you’re interested in solving similar problems at high scale, reach out to us and help us build a more robust pipeline.

We're hiring

Interviewing is a two-way street. We want to provide the best candidate experience to every person who walks through our doors. We want to be consistent and thorough in evaluating our candidates. And we want to make candidates comfortable, so they can ask us questions too. As much as we evaluate candidates, candidates are evaluating us with every interaction. Interviews form everlasting memories of such interactions and come up while talking to their friends; friends who could be potential candidates in the future. As someone famous once said, “People will forget what you said, people will forget what you did, but people will never forget how you made them feel!”  Remember the times when you were a candidate and had a pleasant interview experience? Remember to consciously pay it forward! 

Expectations

Whether you are a candidate, interviewer or recruiter, interviews are stressful. Our goal is to make it as stress-free as possible, so we can get the candidate to showcase the best version of themselves and for interviewers to make the best decisions, all within a short time frame.

What are our expectations from you as an interviewer?

• Treat candidates with respect. Be on time. 100% attention, nothing less. Don’t code, don’t respond to Slack messages, or WhatsApp. Don’t multitask. People can tell when you get distracted and feel let down. Ensure to turn off your notifications. 

• Make candidates feel comfortable: You will be able to bring the best out of them. The goal is not to create pressure situations and stump them with our questions, the goal is to encourage them to feel supported –  treat an interview as a working session with a co-worker. 

• Let candidates impress you.  Talk less and listen more. Listen and ask insightful questions with genuine curiosity and ensure to not come across as judgmental. A good rule of thumb for a great interview is when a candidate talks > 70% of the time. 

• Answer questions candidly. Leave time at the end of an interview to answer questions. Interviewing is a two-way street. Candidates are evaluating us as much as we are. Do not provide answers if you are not sure and do not overcommit in order to come across as knowing something.  

• Provide next steps to candidates before wrapping up the call. Do not give out the result of the interview. Leave that to the Recruiting team. 

• Make clear decisions. A candidate is one of 3 – Strong Yes, Yes, No. There are no Maybes. Gather enough information and signals to make your decision. If you are in doubt, talk to your peers or leads, and convert the maybe to either Yes or No.

• Leave thorough feedback in Workable (our ATS). There’s no expectation to write essays. Write down key talking points that are essential to discuss during the debrief. 

• Attend Debriefs. Notes are sometimes not sufficient. Providing context can help us better understand feedback and make good decisions collectively. 

What should you expect as an interviewer?

• A clear description of role and level. Candidates will be considered for a certain level based on past experience. Treat this as a starting point, not the final one.

• Competencies and skills to evaluate. There is no expectation on you to evaluate all aspects. Every member in the panel will have primary areas to cover during the interview. 

What should you expect from candidates?

These will evolve over time and that’s natural. At this time, here’s what matters:

• Coding and Problem-Solving. Are candidates strong with reasoning skills about their solutions and are they competent in converting their ideas to code? 

• System Design / Domain Knowledge. Are candidates able to articulate high-level designs and are they able to deep-dive in explaining trade-offs?

• Effective Communication.  Are candidates able to share ideas, explain technical topics effectively, listen, answer and ask questions? 

• Growth Mindset. Given that technologies are evolving at a fast rate and problems can be solved in more than one way, are candidates able to consistently learn, relearn and unlearn? 

• No “brilliant” jerks. Regardless of how skilled they are, we do not want jerks.

• Integrity. Last but not least – Do what you say and say what you do. This will help us build trust.

What should you expect from our recruiting team?

• Integrity. We say things the way they are, whether it is with candidates or interviewers. No false promises.

• Communicate decisions with respect. We let candidates know about our decision, regardless of what it is. Rejections are as important as acceptances in our books.

• Responsive. Candidates are anxious. We respond to all candidate communications as quickly as possible.

• Set clear expectations with candidates and interviewers. About the candidates’ role, level, and other relevant information. 

• Track performance, analyze, and report numbers. Sourcing and Conversions are our friends. We continue to improve when we find out what works and what does not.  Data is king.

Feedback

The last 5 mins after every interview should be used to write feedback in Workable. This not only helps us go fast, but also allows you to capture thorough feedback as the conversation will be fresh in your mind. 

• Capture important points about your conversation. You won’t remember the conversation in a few days when we do the debrief. Do not overestimate what you can remember.

• Paste the code/link in Workable. This is important for 2 reasons a) we don’t repeat questions b) helpful during debriefs for others to see. 

• Leave notes for subsequent interviews. We don’t want everyone asking candidates the same questions. It is boring and a loss of signal.

Becoming an Interviewer

As an interviewer, that seems like a lot of effort. I’d rather build products. Why do it?

The biggest reason → You are learning! Interviews teach you a lot about yourself. Some concrete ones:   

• Interviews help improve communication skills. Communication is not only about speaking, but also listening.

• Interviews help you understand the business, product, and technology better. Answering candidate questions requires a deeper understanding. 

• You will get help when the candidate joins! They’ll remember how you made them feel, and pay it back. 

• You are likely to spend more time with your co-workers than with your friends and family. You might as well have a say in the decision. 🙂 

How do I become one?

1. Shadow interviews. Shadow experienced interviewers to learn the art of interviewing. At the end of the interview, spend 5 mins with the interviewer to discuss the candidate’s performance. 

2. Reverse Shadow Interviews. Take the lead after you’ve shadowed 3 interviews. The interview will still have the experienced interviewer, but they will shadow you. After the interview, discuss the observations of a candidate’s performance with the experienced interviewer. They will provide feedback and approve you to lead interviews independently. 

3. Go solo! 

Take ownership when there is a conflict of schedules. Be proactive about asking for feedback, and discussing candidates. Stalk people’s calendars for interviews and ask to shadow. Be creative! 

FAQs

When do I have to submit feedback? 

Hiring is competitive. We need to move fast so we can land the best candidates. In most cases, you are able to submit feedback within 5 mins after the interview. Stuff happens and we are overbooked sometimes. In such cases, ensure to submit feedback within 24 hours of the completion of the interview, at the latest. 

Do I need to attend Interview Debriefs?

Yes, it is mandatory for every interviewer to attend the debrief session. We are all interviewing different facets and it is important to provide all the information/context so we can arrive at a good decision with high confidence. 

How many interviews should I do in a week? 

In an ideal situation, an interviewer would cap the number of interviews at 5 per week. There will be weeks when we will do more given our ambitious goals. Please pay attention to how you feel about taking more interviews and let us know – Interview burnout is real. 

Some interviewers prefer batching all of them in 1 day, others prefer 1-2 a day. Priorities will be captured and scheduling will take into account interviewer preferences. But remember, the schedule has too many variables. As we know, scheduling is an NP-hard problem :-). 

Will it look bad on me as an interviewer if I said yes, and everyone else says no, or vice versa? 

No. We are all evaluating different skills for candidates as a panel. It is reasonable for candidates to be strong in some areas and weak in others. Every interviewer can probe different facets. 

Can I give a strong YES when I have not evaluated all skills of a candidate?

Absolutely. Your strong yes is for areas that you have evaluated. For example, you will find a candidate who is strong at system design and you should give a strong yes despite not knowing their strength in coding. Bring all interviewers together in a debrief and discuss all aspects. 

Should we have candidates work on home assignments / multi-day projects?

Projects can be a good idea for interviews. They provide a window into the thought process, execution, and communication of a candidate. Home assignments are more realistic when compared to operating under the gun in an hour-long conversation. However, candidates may not be able to afford the time to complete projects when juggling jobs, families, etc.,  

Too many interviews will burn me out. What can we do?

In addition to capping the number of interviews/week, as an organization, we practice NO-interview days on Wednesdays.  

Conclusion

Interviewing is both an art and science. It is important to be yourself, throughout. It may be challenging to trust your process as you start and it will be tempting to follow these steps to the T. However, over time, you will rely a lot more on your own than the guidelines provided. And that’s a natural evolution! 

Originally published on February 12, 2022. Copied from from the previous blog

I joined Quizizz a couple of weeks ago and wrote about it here. I have also been working closely with the team over the last few months. In this post, I want to talk about our vision and mission, plans for building the tech organization, and a sneak peek into what to expect if you choose to work with us.

What’s Quizizz?

We are on a mission to motivate every learner on the planet.

Teachers in the United States get an hour a day outside classroom time for preparation. You can imagine it’s probably worse in India given the difference in population and class sizes. Teachers barely have any time to consider new teaching methodologies to rethink their instructions and learn modern practices (ex: incorporating games, flipped classrooms, etc.,) to make classroom learning more effective. Students are bored in their classes and end up supplementing with tuition / after-school classes. While other EdTech companies are trying to solve this problem using async learning, tuition, and sharing extra content, we at Quizizz are on a mission to empower our teachers to use their classroom time more effectively using technology and modern pedagogy involving gameplay flipped classrooms and interactive content.

Quizizz is a platform that has been empowering teachers to motivate and engage learners around the globe. We started out by building products to gamify and make different types of assessments engaging and fun. We have grown to 75M monthly active users and 2B responses recorded every month in the last year. These games take up the first and last 10 mins of a typical class. We have the remaining 40 mins in a classroom that is ripe for innovation. Building on this foundation, we are on the journey to becoming the preferred OS for every teacher to create and deliver engaging content, irrespective of the mode of learning.

We always talk about students being at the center of a triangle of stakeholders — teachers, parents, and peers. So far, we have had success in exploring the link between the teacher and the student via classroom content, gameplay, assessments, and homework. Not only are we planning to go deeper in exploring the student-teacher link, but we also intend to go broader in exploring the other 2 links.

So what will it take to achieve this mission?

Building a World-Class Technology Team

We have gotten to this scale with a nimble team of 20. As we continue to grow, we will preserve the small team culture. Having a direct channel with users and speed will be our biggest moat. We don’t want the bureaucracy of big companies. We want to be able to delight our teachers and students at speed. To fulfill our ambitions for growing the business, we will hire not only some of the best engineers across the board from Infrastructure, Data, ML, Distributed Systems, Frontend, and Mobile but product managers and designers.

At the end of the day, we can guarantee this — there will always be more problems to solve than people. The idea is to relentlessly prioritize building the most impactful products and platforms, eliminate bureaucracy, and create opportunities for everyone to learn, lead, and experience exponential growth in their careers. To that end, we organize ourselves as small, autonomous, decentralized, and mission-oriented teams with high levels of ownership and accountability. Reducing the number of decision-makers is the only way we can move fast. Read more about how we organize ourselves here.

What challenges can you expect as an engineer?

We have the scale of users that every engineer craves. Our real-time experiences can support 3K concurrent users in a session and we have 500K active connections across all sessions on our servers. You can imagine the number of broadcasts per second, and the optimizations required in packaging data and rendering. A flavor of some of the challenges you can expect are –

• Geographically distributed scale that will require us to build multi-datacenter infrastructure, storage technologies optimized for different workloads, and streaming technologies to power real-time experiences

• We have collected massive amounts of data over the last five years, which enables us to now launch products that require solving deep technical problems in Machine Learning (ML) (discovery, recommendations, personalization, etc.,).

• Modern Frontend platforms will require scaling the number of concurrent users for our real-time interactive experiences. For comparison, most frontends scale horizontally (ex: Google search, Facebook newsfeed, etc.,), but at Quizizz, we need to scale both horizontally and vertically.

This is a non-exhaustive list of technologies used to build our products. Most engineers on our team used these technologies for the first time in their careers at Quizizz. More than having a knowledge of these technologies, willingness to learn has helped us build with an eye toward the future.

What is Our Culture like?

Tim Ferris once defined culture as — “what happens when people are left to their own devices.” As part of my onboarding over the last few weeks, I’ve done ~35 1:1s and thought I’d ask people what they like about working at Quizizz. It was heartwarming to hear people talk about what they inherently feel rather than perks like freebies, free food, snacks, etc., (which by the way, we have!). I’ll list a few responses verbatim –

We will continue to stay true to these values. We want to bring people that are excited by working alongside peers like the above. Staying true to our product, our focus is on creating an environment where everyone has the opportunity to learn and grow in their careers. To that end, we expect our team members to learn and adapt themselves. People who thrive here don’t expect to be micromanaged. As someone famously said, “It doesn’t make sense to hire smart people and then tell them what to do; we hire smart people so they can tell us what to do”.

If you love taking higher levels of ownership, are comfortable with things not being defined to the T, and have an endless curiosity to learn, relearn and unlearn, then this is the place for you. We will provide you with problems, a bit of chaos, and lots of opportunities, (Chaos creates opportunity!) and empower you to make some truly magical strides in defining the future of Edtech.

Does this sound like you? If so, apply directly here. If you don’t see yourself fit the descriptions, but are excited to contribute to our mission, hit me up at sandeep [at] quizizz [dot] com. We are hiring for frontend, backend, mobile, and data roles.

We are building a global platform from India. More than one in two schools in the US use Quizizz. We’re rapidly growing in South East Asia and India. It is a rare opportunity to build a global product from the ground up with a brand as loved as ours.

Thanks to Haripriya for editing and Vijaya for the beautiful illustrations!

We're hiring!

Originally published on June 9, 2023

Introduction

Here at Quizizz, we’ve been continuously improving our product and scaling our operations, serving an ever-growing number of teachers and schools globally. As our platform expanded, so did the complexity of our authorization needs, especially with the addition of more collaborative features and the management of diverse user roles and permissions.

This led us to SpiceDB, an authorization solution inspired by Google’s Zanzibar. SpiceDB provides a robust, scalable, and flexible framework for fine-grained, relationship-based access control, which perfectly suited our requirements.

In this blog post, we’re going to discuss our adoption of SpiceDB, including our reasons for choosing it, how we went about the integration, and the improvements we’ve observed since.

Authentication vs Authorization

Authentication and authorization are layers of an application that serve as barriers of defense to stop actors from performing actions that they shouldn’t be able to.

Authentication is the process of verifying who a user is and establishing their identity. This usually plays out in a “login” experience where the user enters their username/password or uses an SSO (single sign-on) provider in order to prove their identity to you.

On the other hand, authorization kicks in after a user is authenticated. It’s about permissions and determining what an authenticated user is allowed to do. It answers the question “Can this user perform a particular action (like edit or delete) on this resource?”.

In a nutshell, authentication is about confirming identity, while authorization is about granting access based on that identity.

Authorization at Quizizz

Authorization at Quizizz is critical to our collaborative features. Let’s look at how it’s utilized in different scenarios:

• Sharing Quizzes: Teachers often need to share quizzes with colleagues. Our authorization system manages who can access these quizzes and what they can do with them – view, edit, or share.

• Co-Teaching: Classrooms often have more than one teacher instructing students. We want to facilitate collaboration among these teachers, and allow them to work together on creating assignments, assessing their students and making data informed adjustments to their lesson plans. 

• Sharing Reports: Student reports are valuable for collaboration between teachers. Teachers can make data-driven decisions about the pace of instruction and assessment that their students require. Authorization ensures these sensitive data can only be accessed by authorized individuals.

In short, authorization plays a crucial role in enabling collaboration securely and effectively.

The need for change

Our existing system had chugged along quite well for the last few years, but we saw that it was becoming a limiting factor in the kind of capabilities we wanted to build into Quizizz.

A couple of key areas that were proving to be difficult were:

• Many-to-many relationships: Collections are logical groupings (like a folder) of quizzes. A quiz can be part of many collections, and a collection can have many quizzes. Managing permissions between entities which don’t have a strict one-to-one relationship like quizzes and collections was becoming complicated at our scale of data and latency requirements

• Transitive permissions: A Class in Quizizz is a logical grouping of students. Teachers can create and assign assessments directly to a class instead of having to do that to each individual student. We struggled with transitive permissions, such as granting access from a class to a co-teacher and then to other entities that were part of the class. Bringing these together in an efficient and maintainable manner was proving to be difficult.

In addition, as we built out our B2B product, we required a way to model hierarchical structures where every level would have permissions for the level below it (district-school, enterprise-branch etc.)

We started re-thinking our current architecture, debating between other out-of-the-box solutions and building it all ourselves, when we stumbled upon Zanzibar, including a couple of blogs by Airbnb and Carta detailing their experience with in-house implementations of Zanzibar.

Zanzibar

Zanzibar is based on “relation tuples” to define access rules, which it uses to create an entity graph. It then uses set operations to resolve permission checks by traversing the graph of relations. 

These tuples, which could represent entities or other tuples, allow transitive relations. Permissions are defined through userset_rewrite clauses in a tuple, determining access based on whether the entity falls within the permitted tuple set. 

Continuing our research on Zanzibar, we found SpiceDB by AuthZed, an open-source Zanzibar-inspired relation-based access-control service that tries to stay true to the principles outlined in the paper.

On further evaluation, we felt like this was a good fit for us

• The community and the developers behind it were active

  • The discord server has a lot of people talking about how they’re working with SpiceDB in their organizations

  • Their developers gave us a custom patch to help with backfilling relations into SpiceDB at scale, which was incredibly helpful in getting us started

• They had a solid roadmap of features, and fairly frequent release cycles. One thing that was of quite a lot of interest to us was the Tiger cache proposal, which would help solve for ACL-aware filtering (think full-text search across resources that a user has access to) in a performant manner.

• The tooling was pretty mature (SDKs in multiple languages, the playground etc.). 

• We were able to model our current and expected features in the SpiceDB schema DSL in a very simple and straightforward way, making it extremely easy to grok and maintain for people new to the system.

With this, we started tinkering with it and working on a proof-of-concept to see if it would be able to support the scale and latency requirements at Quizizz, which has hundreds of millions of responses every day.

SpiceDB at Quizizz

Authzed has a good series of blogs that break down different components of SpiceDB in quite a lot of detail, which I would encourage you to read! But in this blog, I’ll go through our process of adopting SpiceDB into our stack.

SpiceDB keeps a record of relation tuples in a selected datastore (we used Postgres on RDS). It processes permission checks by dividing the query into subproblems and distributing them across multiple instances. SpiceDB achieves this via kuberesolver, which monitors K8s endpoints for other instances. To streamline this and other SpiceDB tasks, we utilized the SpiceDB Operator to manage our cluster.

We built out a new service that would own the domain of Identity and Access management (IAM) in Quizizz, hooked it up to SpiceDB and started running some tests. After being satisfied with the kind of performance and flexibility it offered us in our controlled load tests, we decided to put this into production in “shadow” mode, i.e, all requests going to our existing authorization system would also be duplicated to the new service, and we would compare the results between the two to verify correctness and performance.

Other services interact with the IAM service to check whether an entity has the permissions to perform some action on a resource that is specific to their domain. 

Results

• We saw a tiny latency increase (as one would expect with adding a couple of network hops), but very little of that was attributable to SpiceDB itself

• We experienced <20-30ms p99 latencies for the CheckPermission API (which tells us whether an accessor has the required permission for a particular resource)

• Due to some optimizations that we had done in our internal layers, not every call to access a quiz goes through to SpiceDB. So the most frequent call was actually to list all the relationships for a particular resource, which had a p99 of <10ms which was excellent.

• When we checked for correctness comparisons across a couple of weeks, we found very few instances where there was a mismatch between our existing system and SpiceDB. In fact, a couple of times SpiceDB was correct when our internal system wasn’t!

• When we tried to build out something new which wasn’t part of our existing authorization model, it took less than a couple of days to have that layer up and running in SpiceDB.

However, we did run into a few issues that arose due to the way some of our existing flows worked and how SpiceDB works internally. We were using the TOUCH operation in SpiceDB, which ensures idempotency and does not throw an error if a relation already exists, to create relations. The implementation of TOUCH (at the time), would delete the older relation (if it already existed), and then create a new one. It would create a transaction, and then perform these operations within that transaction. 

This worked fine under normal load, but in a particular case, when we were on-boarding one of our larger clients (the State of Paraná, Brazil), we were sending one WriteRelationships call for each teacher that we onboarded. This resulted in extremely high contention in these transactions, resulting in a massive spike in our Postgres CPU utilization and SpiceDB API latency (graphs below).

The fix in this case was fairly simple: we modified our flow to batch relationships (which SpiceDB allows as part of a single API call) across the onboarding process and made a single call instead of many. 

Conclusion

We realized we needed to upgrade our authorization system to keep up with Quizizz’s growth and the complexity of our platform. 

The impact of introducing SpiceDB into our systems has been quite significant. 

• In less than a day, we were able to build and deliver a feature that our enterprise client (State of Paraná) requested, which had a strong dependency on authorization.

• We continued building highly collaborative capabilities (like the co-teaching experience) into Quizizz with minimal time spent on modeling the authorization layer, and instead were able to focus all our time on building the best possible experience for our teachers.

Come Join Us!

If you’re interested in working on building an identity management platform, dealing with high scale authorizations, and tinkering with databases —Quizizz offers a fertile ground for engineering talent to thrive and make an impact. Reach out to us, and let’s build the future of learning together!

We're hiring!