Senior Data Engineer

Job Overview:
The Data Engineer will be responsible for suggesting and implementing efficiencies in internal processes, optimizing data delivery, and enhancing the scalability of technology. The role involves implementing new features, improving data systems, managing ETL processes, and maintaining data quality and governance. This position requires strong technical skills in data engineering, a solid understanding of data management principles, and experience with various data technologies.

Key Responsibilities:

• Process Improvement:
  • Suggest and execute internal process improvements to automate manual processes and optimize data delivery and technology scalability.
  • Implement enhancements and new features across data systems.
• Data Systems Management:
  • Improve and streamline processes within data systems with support from the Senior Data Engineer.
  • Test CI/CD processes to ensure optimal data pipelines.
  • Assemble and manage large, complex data sets to meet functional and non-functional business requirements.
  • Develop and maintain automated monitoring solutions.
• ETL & Data Quality:
  • Highly efficient in ETL processes and ensuring data consistency.
  • Develop and conduct unit tests on data pipelines.
  • Maintain data quality, data governance, and overall maintenance of data infrastructure systems, including AWS/database security.
• Data Infrastructure:
  • Maintain data warehouse and data lake metadata, data catalogues, and user documentation.
  • Ensure best practices are implemented and maintained for database collations, engines, indices, etc.
  • Support reporting and analytics infrastructure.

Qualifications:

• Educational Background:
  • BSc degree in Computer Science, Information Systems, Engineering, or a related technical field, or equivalent work experience.
• Experience:
  • 3+ years of related work experience.
  • Minimum of 2 years’ experience in building and optimizing ‘big data’ data pipelines and maintaining data sets.
  • Experienced in Python and SQL (PostgreSQL, MS SQL).
  • Experienced in using cloud services: AWS, Azure, or GCP.
  • Proficiency in version control, CI/CD, and GitHub.
  • Understanding/experience with Glue and PySpark is highly desirable.
  • Experience in managing the data lifecycle.
• Skills & Competencies:
  • Proficiency in manipulating, processing, and architecting large disconnected data sets for analytical requirements.
  • Ability to maintain and optimize processes supporting data transformation, data structures, metadata, dependency, and workload management.
  • Strong understanding of data management principles, including data quality assurance and governance.
  • Strong analytical skills with experience working with unstructured datasets.
  • Understanding of message queuing, stream processing, and scalable ‘big data’ data stores.
  • Strong attention to detail and good communication and interpersonal skills.

Additional Information:

• Work Environment:
  • Fully remote work with the option to work from co-working spaces in Cape Town, Mauritius, Kenya, and Nigeria.
  • Access to a cutting-edge development stack and a team of over 80 highly skilled technical professionals.
  • Backing from the international media powerhouse Ringier AG, providing access to world-class development resources and international business insights.
  • Opportunities for autonomy, flexibility, and skill mastery.