I first had a hiring manager interview.

Following this, I had a senior manager interview.

Both of these were largely behavioral. The technical questions just revolved around asking what projects I have done. There were a few questions about how I account for people from different backgrounds on my team, etc.

I worked on a project to develop a real-time data processing pipeline for a financial institution.

The goal was to ingest, transform, and analyze streaming financial data from various sources, such as market feeds and trading platforms, with minimal latency.

Here's a technical breakdown of the project:

• Data Ingestion: We utilized Apache Kafka as the central message broker. Multiple producers, built using Java and the Kafka client library, pushed data streams into distinct Kafka topics. Each topic was configured with appropriate partitioning and replication factors to ensure fault tolerance and scalability.

• Stream Processing: Apache Flink was chosen for real-time stream processing. Flink jobs were developed in Scala to consume data from Kafka topics. These jobs performed transformations, such as data validation, enrichment with historical data from a time-series database (InfluxDB), and feature engineering.

• State Management: Flink's managed state capabilities were leveraged for tasks like calculating moving averages and detecting anomalies. Checkpointing was configured to save the operator state periodically to distributed storage (Amazon S3) for fault recovery.

• Data Storage: Processed data was pushed to two primary destinations. For immediate querying and dashboarding, results were written to a PostgreSQL database. For long-term storage and batch analytics, data was landed in a data lake on Amazon S3 in Parquet format.

• Monitoring and Alerting: Prometheus was integrated for collecting metrics from Kafka, Flink, and application components. Grafana was used for building real-time dashboards to visualize pipeline performance, latency, and error rates. Alertmanager was configured to trigger alerts based on predefined thresholds.

• Deployment: The pipeline was deployed on a Kubernetes cluster, allowing for elastic scaling and simplified management of Kafka and Flink clusters. Docker containers were used to package the Flink jobs and Kafka producers.

This architecture enabled us to process millions of events per second with an end-to-end latency of under 500 milliseconds.