Secure, Real-Time Data Streaming for Glassdoor
Engineering a high-throughput, secure data streaming solution to process encrypted, sensitive data in real-time.
The Challenge
Glassdoor needed to process a high volume of sensitive, encrypted user data in real-time. The solution required both extreme performance to handle the data throughput and robust security to ensure data privacy and compliance. The challenge was to build a low-latency pipeline that could ingest, decrypt, process, and re-encrypt data on the fly without creating vulnerabilities.
Our Solution
We architected a secure, high-performance data streaming solution using a combination of leading technologies.
Secure Kafka Streaming:
We implemented Apache Kafka as the backbone of the streaming platform, configuring it with end-to-end encryption (TLS/SSL) and authentication (SASL) to secure data in transit.
Real-Time Processing:
A custom processing engine was developed to consume data from Kafka topics, perform in-memory decryption of sensitive fields, apply business logic, and re-encrypt the results before publishing to a new topic.
Secure Data Sink:
The processed data was securely streamed into SQL Server using Kafka Connect, with Transparent Data Encryption (TDE) enabled on the database to protect data at rest.
The Impact
The resulting solution provided Glassdoor with a highly scalable and secure pipeline capable of processing sensitive data in real-time. It met their stringent security and compliance requirements while delivering the low-latency performance needed for their application. The architecture ensured that data was protected at every stage, both in transit and at rest.
Project Overview
Key details about the engagement.
Client
Glassdoor
Services
Data Engineering, Real-Time Streaming, Data Security
Technologies
Apache Kafka, SQL Server, Custom Encryption Libraries