senior data engineer

ABOUT THE CLIENT

Our client is a leading professional services firm providing audit, tax, strategy, transaction, and consulting services to clients in the financial services sector.

The Digital Factory team drives technological innovation and digital transformation initiatives across service lines, supporting critical business operations while maintaining the highest regulatory compliance and security standards. Undergoing digital transformation. They specialize in implementing advanced AI solutions across their business operations and are committed to leveraging cutting-edge technology to drive innovation and efficiency.

PROJECT DETAILS

Our client is seeking a talented and motivated Senior Data Engineer to join our dynamic Digital Factory team at Luxembourg. In this role, you will analyze, develop, support, and maintain applications.

YOUR TEAM

Our Digital Factory team provides access to the latest technology, data capabilities and infrastructure to accelerate innovation. Digital Factory Luxembourg is a diverse and international group of digital experts, including designers, developers, AI specialists, and business professionals, focused on creating valuable digital solutions for our clients and stakeholders.

WHAT'S IN IT FOR YOU

• Interview process that respects people and their time
• Professional and open IT community
• Internal meet-ups and resources for knowledge sharing
• Time for recovery and relaxation
• Bright online and offline events
• Opportunity to become part of our internal volunteer community

RESPONSIBILITIES

• Collaborate with stakeholders to understand data needs and translate them into scalable, maintainable data engineering solutions.
• Design, build, and maintain batch and streaming data pipelines in Databricks using Apache Spark, Python, and SQL.
• Apply industry standards and engineering best practices (code reviews, version control, CI/CD, testing, documentation) throughout the data lifecycle.
• Implement robust ETL/ELT frameworks, including ingestion, transformation, and curation layers (e.g., bronze/silver/gold) leveraging Delta Lake.
• Optimize Spark jobs for performance and cost efficiency, including partitioning, caching, AQE, Z-Ordering, and efficient file management.
• Ensure data quality, integrity, and observability with validation, monitoring, data lineage, and alerting (e.g., expectations, SLAs/SLIs).
• Contribute to data modeling (ER, dimensional/star schema) to enable analytics, BI, and downstream consumption.
• Orchestrate and schedule pipelines using Databricks Workflows and/or external tools (e.g., ADF, Airflow), integrating secrets management and environment configuration.
• Implement security and governance controls, including access policies, role-based permissions, and cataloging (e.g., Unity Catalog).
• Document architecture, data flows, runbooks, and standards; promote reusable patterns and componentization across projects.
• Stay current with the Databricks/Spark ecosystem and tooling; drive continuous improvement in platform capabilities and engineering practices.
• Nice to have: contribute to Azure-based architecture design (e.g., ADLS Gen2, ADF, Databricks, Event Hubs, Synapse, Key Vault) and environment setup.

SKILLS

Technology Requirements:

• Strong proficiency in Apache Spark (Structured Streaming, DataFrames), Databricks, Python (PySpark), and SQL.
• Proven experience building and operating scalable ETL/ELT pipelines and data products in Databricks.
• Hands-on with Delta Lake, optimization techniques (partitioning, file sizing, Z-Ordering), and performance tuning.
• Experience with data modeling (ER and dimensional), metadata management, and data quality frameworks.
• Solid software engineering practices: Git-based workflows, code reviews, unit/integration testing, and CI/CD (e.g., Azure DevOps, GitHub Actions).
• Familiarity with orchestration tools (Databricks Workflows, ADF, or Airflow).
• Nice to have: Azure cloud services and architecture (ADLS Gen2, ADF, Azure Databricks, Synapse, Event Hubs, Key Vault), infrastructure-as-code, cost optimization, and security hardening.

General Qualifications:
- Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or a related field.
- 3+ years (Mid) to 5+ years (Senior) of hands-on data engineering experience building production-grade pipelines.
- Experience working with large, complex datasets and heterogeneous data sources (APIs, files, relational/NoSQL, streaming).
- Strong analytical and problem-solving skills with attention to detail and a focus on reliability and maintainability.
- Ability to communicate technical concepts clearly to both technical and non-technical stakeholders.
- Familiarity with data governance, privacy, and compliance standards.
- Experience working in Agile teams and using tools like Jira or Trello is a plus.
- Nice to have: experience in regulated or data-intensive industries (e.g., financial services), and exposure to solution/enterprise architecture on Azure.

Your personal recruiter