In my role as a Senior Data Engineer, I worked on a project deployed on the AWS cloud as part of a large team focused on building a robust data engineering platform. My responsibilities included developing ETL and ELT processes using Apache Spark to ensure efficient data processing. Additionally, I orchestrated and monitored complex data workflows using Airflow for scheduling and orchestration purposes.

• Developing scalable and high-performance ETL/ELT pipelines with Apache Spark.
• Orchestrating and monitoring intricate data workflows using Airflow.
• Collaborating effectively within a large team of data engineers.

Technology Stack: AWS, Apache Spark, Airflow

In my role as a Senior Data Engineer, I spearheaded the development of a migration tool aimed at transferring data from Oracle to Azure. This project involved creating a comprehensive architecture, managing a backlog, assigning tasks, building data pipelines, database modeling, documentation writing, testing, and auditing. I utilized Azure Data Factory and Microsoft SQL Server to ensure a smooth and efficient migration process while maintaining data integrity throughout the project.

• Designing a scalable and robust architecture for the migration tool.
• Managing a complex backlog and task allocation.
• Implementing efficient data pipelines and database modeling.
• Ensuring data integrity and conducting thorough testing and auditing.

Technology Stack: Azure Data Factory, Microsoft SQL Server, Oracle, Data Modeling, Project Management

As a Data Engineer, I contributed to a Proof of Concept (PoC) project focused on migrating data from an SFTP server to an Azure Microsoft SQL Server. My responsibilities included data modeling, creating database objects, and developing stored procedures. We utilized Azure Data Factory as the primary migration tool, ensuring efficient and reliable data transfer from the source to the target environment.

• Designing an effective data model for the target SQL Server database.
• Creating and managing database objects and stored procedures.
• Implementing Azure Data Factory pipelines for seamless data migration.

Technology Stack: Azure Data Factory, SFTP, Microsoft SQL Server, Data Modeling

In my role as a Data Engineer, I contributed to a migration project aimed at transitioning data objects from Teradata to Snowflake. Our team utilized Python scripts to perform thorough audits, ensuring data consistency throughout the migration process. To streamline the migration, we employed dbt (Data Build Tool), which automated the transfer of data and objects between the two platforms, enhancing efficiency and reducing manual effort.

• Maintaining data integrity during the migration process.
• Automating the migration of numerous data objects.
• Implementing robust auditing mechanisms for data validation.

Technology Stack: Python, dbt (Data Build Tool), Teradata, Snowflake

In my role as a Data Engineer and Backend Developer, I contributed to the development of a large-scale data storage system for research files and data masking. The project was deployed on Azure, utilizing Python and Flask for backend development. Data was stored in SQL Server and Snowflake databases, with ELT and ETL processes managed through Azure Data Factory. I worked on handling data from various sources, including FTP, SFTP servers, APIs, and databases, ensuring efficient data ingestion and processing.

• Ingesting and processing large volumes of research data.
• Implementing data masking techniques to protect sensitive information.
• Developing a scalable and secure backend application.

Technology Stack: Azure, Python, Flask, SQL Server, Snowflake, Data Factory

In my role as a Data Engineer and Backend Developer, I worked on a project designed to collect data from multiple Kubernetes clusters, aggregate the data, and create a web service for frontend visualization. The project was deployed on Azure, using Event Hub and Spark Streaming (Databricks) for real-time data collection, data lake gen2 for storage, and TimescaleDB for data aggregation and insertion. I developed the backend using Go and managed Spark batch processing to aggregate and insert data into TimescaleDB.

• Real-time data collection from Kubernetes clusters.
• Efficient data aggregation and storage.
• Backend development for data visualization.

Technology Stack: Azure, Event Hub, Spark Streaming (Databricks), Data Lake Gen2, TimescaleDB, Go

In my role as a Data Engineer, I developed a platform for real-time data collection from IoT devices on various vessels. This project aimed to operate offline and synchronize data when connected to the internet. My responsibilities included data collection from over 40 IoT devices, data storage in PostgreSQL, data aggregation using Python and FastAPI, and data visualization on frontend dashboards. I used Node-RED for orchestration and deployed the project on AWS.

• Implementing offline data collection and synchronization.
• Processing and aggregating data from multiple IoT devices.
• Creating effective data visualizations on dashboards.

Technology Stack: PostgreSQL, Python (FastAPI), Node-RED, AWS

In my role as a Data Engineer, I contributed to the development of a POC project aimed at message analysis and real-time geolocation data visualization. I was responsible for creating and designing a data lakehouse using MinIO and Hive, processing messages with PySpark, setting up APIs in Python, and visualizing data with Superset. I also managed data orchestration using Airflow.

• Building and designing a robust data lakehouse.
• Analyzing and processing real-time messages.
• Creating effective visualizations for geolocation data.

Technology Stack: MinIO, Hive, PySpark, Python, Superset, Airflow

In my role as a Junior Data Engineer, I contributed to a large team focused on creating an MVP platform for real-time data collection from scooters and bicycles. My responsibilities included data aggregation and visualization using Spark Streaming and Kafka for ELT and ETL processes, MinIO and Cassandra for storage, and Kibana for visualization within the ELK stack. Additionally, I used Scala for development, Airflow for orchestration, and Docker Compose for project management.

• Developing a system for real-time data collection and aggregation.
• Maintaining reliable and efficient ELT and ETL processes.
• Utilizing the ELK stack for effective data visualization.

Technology Stack: Spark Streaming, Kafka, MinIO, Cassandra, Kibana (ELK stack), Scala, Airflow, Docker Compose

In my internal Big Data courses, I focused on both the theoretical and practical learning of RDBMS and NoSQL databases. I worked extensively with the Hadoop ecosystem, including components such as HDFS, MapReduce, and Yarn, as well as tools like Hive, Spark, Cassandra, and MongoDB. Additionally, I learned to set up and operate clusters on Google Cloud Platform using Dataproc.

• Mastering RDBMS and NoSQL database concepts.
• Hands-on experience with Hadoop ecosystem tools.
• Setting up and managing clusters on GCP.

Technology Stack: Hadoop (HDFS, MapReduce, Yarn), Hive, Spark, Cassandra, MongoDB, Google Cloud Platform (Dataproc)

Throughout my Java external courses, I focused on learning the core concepts of Java programming, algorithms, frameworks, and database theory. I developed web applications using Java Servlets, JSP, and MySQL, which strengthened my programming skills and provided hands-on experience with Java technologies.

• Learning Java programming fundamentals and algorithms.
• Building web applications with Java Servlets and JSP.
• Gaining a deep understanding of database theory.

Technology Stack: Java, Java Servlets, JSP, MySQL

As a Python Developer at Ukrainian Railways, I was responsible for configuring servers, developing, supporting, refactoring, and testing Python applications utilizing FastAPI. This role demanded strong Python programming skills, expertise in API development, and the ability to troubleshoot and optimize applications to enhance performance and reliability.

• Developing robust and scalable Python applications.
• Refactoring and optimizing code for enhanced performance.
• Conducting thorough testing to ensure application reliability.

Technology Stack: Python, FastAPI, Server configuration

In my role at Ukrainian Railways, I handled the maintenance and administration of on-premise servers. My key responsibilities were configuring and managing MySQL databases, as well as tuning and optimizing queries and tables to boost performance. This position required advanced analytical skills, precision, and extensive expertise in database management.

• Ensuring stable and high-performance on-premise servers.
• Optimizing MySQL queries and tables for better efficiency.

Technology Stack: MySQL, On-premise server management