Data Engineer, Greenfield Global (Sep 2023 - Present)
- Engineered an advanced departmental IT dashboard using Power BI, SQL, Databricks, and Azure Data Factory (ADF), seamlessly aggregating data from over 20 sources. Skillfully refining a daily influx of 20,000+ data points, this dynamic solution empowers upper management to effectively track and forecast the team’s key performance indicators (KPIs) and overall department performance trends.
- Collaborated with business analysts and data scientists to develop and deliver a diverse set of reports utilizing various data types. Demonstrated expertise in SQL, Power BI, Databricks, and Azure Data Factory. Applied advanced data processing and transformation techniques across a range of datasets, enhancing accuracy and reliability.
- Implemented a streamlined solution by leveraging Power Apps and Azure APIs to automate an alerting service. This service promptly notifies the relevant teams whenever there are failures or errors in an ETL pipeline. This proactive approach not only significantly saved time for the company but also ensured swift response and resolution to any issues in the data processing workflows.
- Enhanced data governance through the development of an automated data dictionary report. This initiative streamlined understanding across business teams, mitigated duplicate data set creation, and resulted in substantial resource savings.
Senior Software Developer, RBC (Dec 2022 - Sep 2023)
- Architected a robust C file watcher application, adept at monitoring and tracking 1000+ daily files, seamlessly producing tailored reports integrated with Power BI and SQL. This innovation not only slashed the support team’s workload by over 50% but also streamlined the debugging of sensitive data, effectively addressing complexity and compliance challenges.
- Developed and implemented an automated data quality framework, integrating validation scripts into the ETL pipeline for continuous monitoring. This initiative resulted in a 25% reduction in data-related errors, ensuring a high-quality, reliable data set for improved reporting and analysis.
- Designed and developed highly efficient automation software using C#, resulting in a saving of over 100 hours of manual code deployment and eliminating all human errors.
- Efficiently updated, documented, troubleshot, and maintained middleware software for various API applications in coordination with development and client deadlines.
Data Engineer, ETL Developer,Statistics Canada(Feb 2022 - Dec 2022)
- Developed data transformation code in ETL pipelines to integrate data into Azure data warehouse for consumption by the data analyst clients through Power BI.
- Improved database extraction performance by over 40% by researching and implementing efficient database extraction methods.
- Designed and Developed Azure data warehouse architecture to streamline and simplify the data transformation and data loading process, which decreased the amount of storage used.
- Automated the ETL process using C#, Scala, and PySpark scripts, reducing the processing time from 5 hours to just 10 minutes.
Programmer Analyst, Application Development,Statistics Canada (Jan 2020 - Feb 2022)
- Developed, implemented, and maintained over 20 web scraping applications for different clients collecting over 2TB of data weekly using C#, Selenium, and Restful API’s implementing advanced knowledge of Data‑Structures and Algorithms to improve code efficiency and runtime for various projects, greatly increasing performance.
- Developed, implemented, and maintained a data validation software that was used by clients to validate over 400M rows of data on a weekly basis to ensure data quality is up to a given standard.
- Successfully managed multiple automation projects through their SDLC life cycles using CA Automic, Windows Task Scheduler, and Bash scripts.