Introduction

Data transformation is a basic process in data analysis and machine learning. It involves converting raw data into a more suitable format for analysis.

We started learning about this in B1700.

By implementing these transformations, we can ensure data quality, improve our models’ performance, and enable meaningful comparisons between different features (or variables) in our data.

In machine learning, it’s common to refer to what we might consider variables as ‘features’. The two terms are used interchangeably in this module.

Transformation techniques like scaling, standardisation, and normalisation help address common data challenges such as:

Inconsistent scales between different features;
Non-normal distributions that may affect statistical analyses;
Outliers that could skew results; and
Missing or incomplete data that needs pre-processing.

Scaling, standardisation, and normalisation

graph TD;
    A["Raw Data"] --> B["Data Transformation"];
    B --> C["Scaling"];
    B --> D["Standardisation"];
    B --> E["Normalisation"];
    C --> F["Fixed Range"];
    D --> G["Zero Mean &<br>Unit Variance"];
    E --> H["Specific Distribution"];
    
    style A fill:#f9f,stroke:#333;
    style B fill:#bbf,stroke:#333;
    style C fill:#dfd,stroke:#333;
    style D fill:#dfd,stroke:#333;
    style E fill:#dfd,stroke:#333;
    style F fill:#ffe,stroke:#333;
    style G fill:#ffe,stroke:#333;
    style H fill:#ffe,stroke:#333;

Scaling, standardisation, and normalisation are data pre-processing techniques we can use to adjust feature values for consistency and comparability.

Scaling ensures features fit within a specified range.
Standardisation transforms the data to have a mean of 0 and a standard deviation of 1.
Normalisation also transforms data to fit a specific range or magnitude.

Content map

Review each of the following elements (in order):

Scaling

Standardisation

Normalisation

Revision questions

To help consolidate your learning, review the revisions questions here:

Revision Questions - Data Transformation

Links

Presentation as PDF

Demonstration

Practical

Next module topic