Data Cleaning: Duplicates, Data Types, Text aur Outliers

Lesson 12 of 26

Missing values ke alawa bhi data me kai problems hoti hain. Is lesson me ek jaan-boojh ke 'ganda' dataset saaf karenge:

Setup

Python 
import pandas as pd

raw = pd.DataFrame({
    "customer": [" aman sharma", "RIYA VERMA ", "Kabir Singh", "aman sharma", "Sara Khan"],
    "city": ["delhi", "Mumbai ", "PUNE", "delhi", "Delhi"],
    "amount": ["1,200", "850", "abc", "1,200", "15000"],
    "order_date": ["2026-01-05", "2026-01-06", "2026-01-07", "2026-01-05", "not known"],
})
print(raw)
print()
print(raw.dtypes)

Output

Plain Text 
       customer     city amount  order_date
0   aman sharma    delhi  1,200  2026-01-05
1   RIYA VERMA   Mumbai     850  2026-01-06
2   Kabir Singh     PUNE    abc  2026-01-07
3   aman sharma    delhi  1,200  2026-01-05
4     Sara Khan    Delhi  15000   not known

customer      object
city          object
amount        object
order_date    object
dtype: object

Problems: naamon me extra spaces aur alag-alag case, amount text hai (comma aur "abc"), date bhi text hai, aur ek customer do baar hai.

Step 1: Text cleanup

.str se har string pe ek saath kaam hota hai:

Example

Python 
df = raw.copy()
df["customer"] = df["customer"].str.strip().str.title()
df["city"] = df["city"].str.strip().str.title()
print(df[["customer", "city"]])

Output

Plain Text 
      customer    city
0  Aman Sharma   Delhi
1   Riya Verma  Mumbai
2  Kabir Singh    Pune
3  Aman Sharma   Delhi
4    Sara Khan   Delhi

Step 2: Duplicates hataana

Text saaf hone ke baad hi duplicates sahi pakde jaate hain — pehle " aman sharma" aur "aman sharma" alag dikhte:

Example

Python 
print("Duplicate rows:", df.duplicated().sum())
df = df.drop_duplicates().reset_index(drop=True)
print(df)

Output

Plain Text 
Duplicate rows: 1
      customer    city amount  order_date
0  Aman Sharma   Delhi  1,200  2026-01-05
1   Riya Verma  Mumbai    850  2026-01-06
2  Kabir Singh    Pune    abc  2026-01-07
3    Sara Khan   Delhi  15000   not known

Sirf kuch columns ke basis pe duplicate check karna ho to df.drop_duplicates(subset=["customer"], keep="first") likhiye.

Step 3: Data types theek karna

errors="coerce" ka matlab — jo value convert na ho sake, usse NaN bana do (crash mat karo):

Example

Python 
df["amount"] = pd.to_numeric(df["amount"].str.replace(",", ""), errors="coerce")
df["order_date"] = pd.to_datetime(df["order_date"], errors="coerce")
print(df)
print()
print(df.dtypes)

Output

Plain Text 
      customer    city   amount order_date
0  Aman Sharma   Delhi   1200.0 2026-01-05
1   Riya Verma  Mumbai    850.0 2026-01-06
2  Kabir Singh    Pune      NaN 2026-01-07
3    Sara Khan   Delhi  15000.0        NaT

customer              object
city                  object
amount               float64
order_date    datetime64[ns]
dtype: object

Ab "abc" aur "not known" dono NaN/NaT ban gaye, jinhe pichhle lesson ke tareekon se handle kar sakte hain.

Step 4: Outliers pakadna (IQR method)

Outlier wo value hai jo baaki data se bahut door ho. Ye data entry ki galti bhi ho sakti hai, ya sach me ek rare case. Sabse popular tareeka hai IQR (Interquartile Range):

  • Q1 = 25th percentile, Q3 = 75th percentile
  • IQR = Q3 − Q1
  • Neeche ki limit = Q1 − 1.5 × IQR, upar ki limit = Q3 + 1.5 × IQR
  • In limits ke bahar wali values outliers hain

Example

Python 
salaries = pd.Series([32000, 35000, 38000, 40000, 42000, 45000, 47000, 50000, 250000])

q1 = salaries.quantile(0.25)
q3 = salaries.quantile(0.75)
iqr = q3 - q1
lower = q1 - 1.5 * iqr
upper = q3 + 1.5 * iqr

print(f"Q1={q1}, Q3={q3}, IQR={iqr}")
print(f"Limits: {lower} se {upper}")
print("Outliers:", salaries[(salaries < lower) | (salaries > upper)].tolist())

Output

Plain Text 
Q1=38000.0, Q3=47000.0, IQR=9000.0
Limits: 24500.0 se 60500.0
Outliers: [250000]

Outliers ka kya karein?

Example

Python 
capped = salaries.clip(lower=lower, upper=upper)   # limit pe rok do
print("Mean pehle:", round(salaries.mean()))
print("Mean capping ke baad:", round(capped.mean()))

Output

Plain Text 
Mean pehle: 64333
Mean capping ke baad: 43278

Ek 2.5 lakh wali value ne poora average bigaad diya tha. Outliers ke saath teen raaste hain: hata do (agar pakka galti hai), cap kar do (clip), ya rehne do (agar wo sach me important hain — jaise fraud detection me outliers hi to dhoondhne hain).

Cleaning checklist

Problem

Pandas solution

Extra spaces / case

.str.strip(), .str.lower(), .str.title()

Galat text

.str.replace("old", "new")

Duplicates

duplicated(), drop_duplicates()

Number text me hai

pd.to_numeric(errors="coerce")

Date text me hai

pd.to_datetime(errors="coerce")

Column ke naam

df.rename(columns={...})

Outliers

IQR method, clip()

On This Page