Mission

Design, build, and operate reliable and scalable data pipelines and infrastructure that power internal analytics and business intelligence at Optable. This includes developing Spark-based ETLs and maintaining a high-quality data warehouse used by Engineering, Product, Sales, and Customer Success teams. Maybe others as well.

🛠️ Core Responsibilities

1. Development

• Build and maintain batch Spark jobs on k8s to ingest and transform data from internal services and external integrations.
• Implement resilient, testable, and performant Spark code (Scala).
• Write high-quality data contracts and schemas with clear ownership and versioning.
• Help design and evolve the core data warehouse schema optimized for BI and analytics.
• Collect feedback from the clients of this product (Solutions, Sales, Product, etc) to understand reporting needs and help prioritize a backlog.
• Use DAG orchestration (with Astro, a Managed Airflow service) for productizing pipelines.

2. Maintenance

• Set up robust monitoring and alerting for data freshness, pipeline health, and SLA adherence (Prometheus metrics, Grafana Cloud for vizualisations, Grafana IRM for incidents).
• Debug and resolve data quality issues proactively.
• Be ready to participate actively in incident response related to data infrastructure.
• Document pipelines and datasets to support self-serve access and discovery.
• Contribute to internal tooling and CI/CD workflows to improve developer velocity and platform stability.