Mastering Regular Expressions for Data Cleaning in Python
Data cleaning is a critical step in any data science project. A clean dataset is crucial for generating accurate and meaningful insights. Regular expressions (regex) are incredibly powerful tools for pattern matching and data cleaning tasks. In this blog, you will learn how to harness the power of regular expressions in Python to clean and preprocess datasets efficiently.
1. Introduction to Regular Expressions
Regular expressions are a syntax used to describe search patterns. They provide a powerful and flexible way to search strings, replace substrings, and extract information from text. In Python, the ‘re’ module enables regex operations. Let’s start with a simple example:
import re
text = 'Error 404: Page not found. Try later.'
pattern = re.compile(r"\berror\s\d+\b", re.I)
match = pattern.search(text)
if match:
print('Match found:', match.group())
In this snippet, we are searching for the word “error” followed by a number. The pattern ‘r”\berror\s\d+\b”‘ uses \ for word boundaries, \s for whitespace, and \d+ for one or more digits. The re.I flag makes the search case-insensitive.
2. Removing Unwanted Characters
Dealing with unwanted characters is a common data cleaning task. Let’s see how you can use regex to remove all non-alphanumeric characters from a dataset:
import pandas as pd
data = {'dirty_column': ['Data@Science!', 'Python$%2021', 'Clean%Data123']}
df = pd.DataFrame(data)
df['cleaned'] = df['dirty_column'].apply(lambda x: re.sub(r'[^\w\s]', '', x))
print(df)
In this example, we use the regex pattern ‘[^\w\s]’ to remove any character that is not an alphanumeric character (word character) or space. The ‘re.sub()’ function replaces these unwanted characters with an empty string.
3. Extracting Patterns from Text
Suppose you are working with a dataset that contains mixed text, such as a list of contacts, and you need to extract phone numbers. Here’s how you can achieve this:
text_data = "Contact Jack at +1-202-555-0173 or via mail: jack@example.com. Alex is at (202) 555-0198 or alex@work.org."
phone_numbers = re.findall(r"\+?\d{1,3}[\s\-\(]?\d{1,3}[\)\s\-]?\d{3}[-\s]?\d{4}", text_data)
print(phone_numbers)
Using the regex pattern ‘\+?\d{1,3}[\s\-\(]?\d{1,3}[\)\s\-]?\d{3}[-\s]?\d{4}’, we successfully extract phone numbers in various formats by specifying optional country codes and handling different delimiters like spaces, dashes, and parentheses.
4. Data Validation with Regular Expressions
Input validation is vital to ensure data consistency and accuracy. To validate an email address using regex, you can use the following approach:
def is_valid_email(email):
pattern = r"^[\w\.-]+@[a-zA-Z0-9-]+\.[a-zA-Z]{2,6}$"
return bool(re.match(pattern, email))
emails = ['valid.email@example.com', 'invalid-email@.com']
valid_emails = [email for email in emails if is_valid_email(email)]
print(valid_emails)
This pattern checks that an email has a proper username, an @ symbol, a domain name, and a valid top-level domain.
5. Optimizing Regex Patterns for Performance
Some regex patterns can become inefficient on large datasets. Minimizing the complexity of your regex is key to enhancing performance. For instance:
dates = ['2023/10/05', '2023.10.05', '10-05-2023']
simple_pattern = r"\d{4}[\-/\.]\d{2}[\-/\.]\d{2}"
matches = [d for d in dates if re.match(simple_pattern, d)]
print(matches)
In this code, I used a simpler pattern ‘\d{4}[\-/\.]\d{2}[\-/\.]\d{2}’ to handle multiple delimiters without overcomplicating the expression. Always test your regex on sample data and refine it to balance complexity and performance.
Useful links:
