Data Engineer Job Description Template
Our company is looking for a Data Engineer to join our team.
Responsibilities:
- Work with data scientists to integrate their machine learning models in production environment;
- Research new ways for data acquisition;
- Automate and monitor ETL jobs created by self and others;
- Lead the data projects by understanding the requirements, suggesting solutions and timely delivery;
- Apply industry best practices for data capture and storage;
- Leveraging data from a variety of sources to build and maintain our data lake and datawarehouse;
- Work independently and as a team to deliver complex projects involving various technological and product teams.
Requirements:
- Collecting and integrating data sets;
- Solid experience in building RESTful APIs and microservices, e.g. with Flask;
- understanding of SQL/APIs;
- Cleaning and organising data;
- Experience in analysing, modelling and interpreting large and complex data, with the ability to integrate data from multiple sources and technologies;
- distributed computer frameworks on Hadoop, Spark, distributed SQL, and/or noSQL query engines;
- At least 2 years’ business experience in BI / Data Engineering;
- Advanced knowledge and skills in Python, Apache Airflow, R, ETL and integration;
- experience with AWS or equivalent other cloud environments – any or all of EC2, S3, RDS, Dynamo DB, EMR, Redshift, Glue, Athena, Apache Parquet;
- Expertise in SQL, SQL tuning, schema design, Python, Kubernetes;
- Proven experience in a fast-moving environment;
- Strong understanding of data structures and how to optimise data delivery;
- Background in Mathematics, Physics or Computer Science;
- working knowledge of Big Data querying tools like Hive or Presto;
- Strong knowledge of SQL, relational databases, NoSQL, column store and schema design.