Principal Architect – Data Platform

We are looking for a hands-on Principal Data Platform Architect to own the design, development, and evolution of the client data platform. The role focuses on building a secure, scalable, and governed lakehouse architecture based on Apache Iceberg and Amazon S3, covering data storage, ingestion, cataloging, governance, analytics, and document management. The successful candidate will establish a robust data foundation for regulated banking environments, ensuring strong data isolation, security, compliance, and operational reliability across multiple banks. This is a highly technical leadership role requiring both architectural expertise and hands-on engineering capabilities.

Key Responsibilities
• Lakehouse Architecture: Design and own the Apache Iceberg and Amazon S3 lakehouse architecture, including table design, schema and partition evolution, snapshots, compaction, and retention policies.
• Data Catalog & Metadata: Architect and implement a scalable data catalog and metadata management layer, ensuring effective data discovery, organization, and governance.
• Data Storage Architecture: Establish bronze, silver, and gold data layers, implementing appropriate lifecycle management, encryption, storage optimization, and cost controls.
• Multi-Tenant Data Isolation: Design secure, isolated data environments for individual banks using dedicated AWS accounts, namespaces, catalogs, encryption keys, and cost attribution mechanisms.
• Data Ingestion & Integration: Develop scalable ingestion frameworks supporting batch processing, real-time streaming, Change Data Capture (CDC), transactional outbox patterns, file-based ingestion, and API integrations.
• Data Governance & Compliance: Define and implement standards for data contracts, lineage, classification, retention, data quality, access controls, and auditability in regulated banking environments.
• Analytics & BI Enablement: Build the foundations for analytics and business intelligence through SQL query engines, semantic layers, and reusable data products.
• Document Management: Architect document storage and management capabilities using Amazon S3, including metadata indexing, full-text search, retention policies, and integration with case management systems.
• Hands-on Engineering: Develop production-grade solutions in Java or Python, including reusable frameworks, SDKs, and reference implementations. Review and validate AI-generated code to ensure quality, security, and maintainability.
• Technical Leadership & Operations: Establish operational standards for data freshness, quality, observability, and SLOs. Mentor engineers, drive architectural consistency, and improve data engineering practices across distributed teams. Technical Requirements
• Data Lakehouse Architecture: Strong hands-on expertise with Apache Iceberg, Amazon S3, Parquet, and Medallion Architecture (Bronze, Silver, Gold).
• Data Catalog & Metadata Management: Experience with AWS Glue Data Catalog, Iceberg REST Catalog, Apache Polaris, Nessie, or equivalent solutions.
• Data Processing & Analytics: Proficiency with Apache Spark, Flink, Trino, Athena, or similar distributed processing and SQL query engines.
• Data Ingestion & Streaming: Experience building batch and real-time pipelines using Kafka, Debezium, AWS DMS, CDC, and API-based integrations.
• Data Governance: Strong understanding of data lineage, data contracts, classification, retention policies, and data quality frameworks.
• Cloud Architecture & Security: Advanced AWS expertise, including IAM, KMS, Lake Formation, multi-account architecture, encryption, and data isolation.
• Data Quality & Orchestration: Experience with Airflow, Dagster, or Step Functions, along with automated data validation and monitoring.
• Document Management & Search: Experience with S3-based document storage, metadata indexing, and OpenSearch / Elasticsearch.
• Software Engineering & DevOps: Strong production-level Java or Python skills, experience building reusable frameworks and SDKs, and proficiency with Terraform and CI/CD.
• Production Operations & Technical Leadership: Experience with observability, SLOs, performance optimization, and mentoring engineers in distributed teams.