Abstract
ISSN:
Volume 2 Issue 1
Digital Journal of Engineering Science and Technology (DJEST)
Augmenting Data Integrity: Leveraging AI & ML Algorithms to Enhance Data Quality at Scale
Staff Data Engineer, Oportun, USA
*Corresponding Author: Abhijit Joshi, Staff Data Engineer, Oportun, USA
DOI:
Abstract
In this paper, we propose a comprehensive framework for enhancing data quality by leveraging advanced Artificial Intelligence (AI) and Machine Learning (ML) algorithms. As data volumes increase exponentially, ensuring data quality becomes an ever more critical challenge. Traditional methods are not scalable, leading to bottlenecks and a significant rise in costs associated with manual data cleansing, validation, and anomaly detection processes. This paper introduces AI-driven methodologies for anomaly detection, data cleansing, and consistency validation that can be embedded directly into data pipelines, ensuring real-time quality checks and automated corrections. We present algorithms such as Random Forest and Principal Component Analysis (PCA) for outlier detection and anomaly identification, provide pseudocode, mermaid diagrams for each methodology, and explain the integration of these AI models into existing data workflows. Additionally, we provide visual data through graphs and charts to demonstrate the effectiveness of AI/ML techniques on data integrity improvement.
Testimonials
Collaborating with WestLand Publishers has been a great experience. Their dedication to quality and integrity sets them apart in the industry. From manuscript submission to publication, their team offers exceptional support, ensuring a smooth journey.
WestLand Publishers is committed to fostering innovation and collaboration in scholarly publishing. It's an honor to have my research featured in their journals, and I appreciate their support.
Publishing with WestLand Publishers has been a positive experience. Their professionalism and rigorous peer review process contribute to the quality of their publications.
WestLand Publishers' efforts in advancing scientific research are commendable. It's been a privilege to partner with them.
I recommend WestLand Publishers to researchers seeking a reputable platform. Their attention to detail makes them stand out.