Data Analysis

Data Collection, Cleaning, and Preprocessing Terms

Learn the source, pipeline, schema, missing-data, quality, scaling, encoding, and validation terms used to prepare reliable data.

12 matching terms

Collection and pipelines

Data source

Meaning

The system, file, sensor, survey, or service from which data originates.

When to use it

Record ownership, update frequency, access method, and known limitations.

Practical example

Source: production orders database, refreshed every hour.

Collection and pipelines

ETL

Meaning

Extract, transform, and load: a pipeline that moves and prepares data for a target system.

When to use it

Use it for repeatable ingestion into warehouses, marts, or analytics systems.

Practical example

Extract orders -> standardize currency -> load analytics.orders

Structure

Schema

Meaning

A definition of data fields, types, relationships, and constraints.

When to use it

Use it to validate incoming data and coordinate producers and consumers.

Practical example

order_id: string, ordered_at: timestamp, amount: decimal

Structure

Data type

Meaning

A classification that determines how a value is stored and interpreted.

When to use it

Set correct types before sorting, arithmetic, filtering, and date operations.

Practical example

Convert ordered_at from text to timestamp.

Data quality

Missing value

Meaning

A value that is absent, unknown, or not applicable.

When to use it

Measure missingness by column and investigate why it occurs before treatment.

Practical example

missing_rate = isna(customer_age).mean()

Data quality

Imputation

Meaning

Replacing missing values with estimated or rule-based values.

When to use it

Use it when deletion would waste data and the replacement assumptions are defensible.

Caution

Imputation can distort distributions and uncertainty; document the method and preserve a missingness indicator when useful.

Practical example

Fill missing store_size with the median within each store_type.

Data quality

Duplicate record

Meaning

A repeated row or entity that represents the same underlying event or object.

When to use it

Define a business key and retention rule before removing duplicates.

Practical example

Keep the latest row for each order_id after sorting by updated_at.

Data quality

Outlier

Meaning

An observation that is unusually distant from the main data pattern.

When to use it

Investigate whether it is an error, a valid rare event, or a signal needing separate treatment.

Practical example

Review transactions above the 99.9th percentile instead of deleting them automatically.

Transformation

Normalization

Meaning

Rescaling values to a bounded range, commonly 0 to 1.

When to use it

Use it when algorithms or comparisons are sensitive to feature scale.

Practical example

x_scaled = (x - min(x)) / (max(x) - min(x))

Transformation

Standardization

Meaning

Centering and scaling values, commonly to mean 0 and standard deviation 1.

When to use it

Use it for models that compare or combine features with different units.

Practical example

z = (x - mean(x)) / std(x)

Transformation

Categorical encoding

Meaning

Converting categorical labels into a representation an algorithm can process.

When to use it

Choose one-hot, ordinal, target, or learned encoding based on meaning and model behavior.

Practical example

One-hot encode channel into channel_web, channel_store, channel_partner.

Data quality

Data validation

Meaning

Automated or manual checks that data satisfies expected rules and constraints.

When to use it

Run it at pipeline boundaries to detect quality regressions early.

Practical example

Assert order_id is unique, amount >= 0, and currency is allowed.