Data Engineer

Job Description

We are looking for a dedicated Data Engineer who is passionate about building robust data architectures and transforming raw data into valuable insights. This role is ideal for someone who can design, develop, and maintain scalable data solutions, working closely with cross-functional teams to ensure efficient and secure access to critical data. If you’re driven by data optimization, ETL processes, and the latest in data technology, join us to empower our organization with the insights it needs.

Core Responsibilities

• Architect and Develop Scalable Data Solutions: Design and implement advanced data pipelines and processing solutions to support high-performance, scalable data architectures, ensuring the infrastructure can handle growing data demands.
• Collaborate with Development Teams: Partner with software and product development teams to gather requirements and translate them into efficient, data-driven solutions, fostering collaboration and understanding across departments.
• Implement and Manage ETL Processes: Design, build, and monitor ETL processes that ensure smooth data integration from various sources. Leverage company-specific technologies to create seamless data flows while adhering to best practices for consistency and reliability.
• Database Performance Optimization and Maintenance: Monitor and fine-tune database performance, balancing factors like cost and performance. Implement indexing, caching, and query optimization strategies to improve data retrieval times.
• Data Security and Privacy Compliance: Develop and enforce security protocols to protect sensitive data, ensuring compliance with industry standards and data privacy regulations.
• Collaborate with Data Analysts and Scientists: Work closely with data professionals to ensure data accessibility, quality, and availability, supporting the organization’s needs for insights and decision-making.

Qualifications and Experience

• Proven experience in building and managing large-scale data solutions, preferably within a cloud environment like AWS.
• Strong skills in database technologies, with hands-on experience in query optimization, data modeling, and storage solutions.
• Proficiency in ETL tools and data transformation techniques, ensuring data accuracy and integrity throughout the pipeline.
• In-depth knowledge of data security and regulatory requirements to protect and secure sensitive information.
• Effective collaboration skills with a history of working cross-functionally with developers, analysts, and data scientists.

Tech Stack

This role requires familiarity with the following tools and technologies:

• Airflow: For scheduling, managing, and monitoring complex data workflows and ETL processes.
• AWS (S3, Redshift, EMR): For scalable data storage, processing, and warehousing.
• Kafka: To enable real-time data streaming and support event-driven architectures.
• Qlik and Spark: Used for data visualization and distributed data processing, essential for handling big data.
• Python and SQL: For data transformation, querying, and pipeline development.