> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mage.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Data validation
> Every data loader and transformer block has data validation capabilities built-in.
You can define 1 or more test functions in a single block. Each test function
accepts a data object as an argument.
Within the body of the function, you can write any type of test you want to
validate the input data.
After the block’s main code is executed, the output data is passed into each
test function for validation. If any tests fail, then the block run will also
fail.
***
### Example
Here is an example of a transformer block with 2 tests:
```python theme={"system"}
from pandas import DataFrame
if 'transformer' not in globals():
from mage_ai.data_preparation.decorators import transformer
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
COLUMNS_TO_USE = ['name']
@transformer
def transform_df(df: DataFrame, *args, **kwargs) -> DataFrame:
return df.iloc[:1][COLUMNS_TO_USE]
@test
def test_number_of_rows(df) -> None:
assert len(df.index) >= 2, 'The output has more than 1 row.'
@test
def test_columns(df) -> None:
assert df.columns[0] != COLUMNS_TO_USE[0], 'The output columns don’t match.'
```
You can combine all your data validations into 1 test function or you can
split them up into multiple test functions. The benefit of splitting them up
is that they can run in parallel, speeding up the data validation.
***
### Log output
Each test run is recorded and can be viewed in the logs. Here is an example:
```
Start executing block.
--------------------------------------------------------------
2/2 tests passed.
Finish executing block.
```
***
# Data Quality with Great Expectations
## Setup
1. Before adding expectations to your pipeline, please make sure you have at least 1
data loader, transformer, or data exporter block.
They must be Python blocks (SQL block support coming soon).
If you don’t have any blocks, add a data loader block and paste the following code:
```python theme={"system"}
import io
import pandas as pd
import requests
@data_loader
def load_data_from_api(*args, **kwargs):
url = 'https://raw.githubusercontent.com/mage-ai/datasets/master/restaurant_user_transactions.csv'
response = requests.get(url)
return pd.read_csv(io.StringIO(response.text), sep=',')
```
For more expectations, read Great Expectation’s [documentation](https://greatexpectations.io/expectations/).
2. [Add Great Expectations power up to your pipeline](/design/blocks/extension#how-to-add-extensions-to-your-pipeline).
3. In the list of available power ups, click the option for Great Expectations.
***
## Adding expectations to your pipeline
1. Once you’re on the Great Expectations power up detail page, you can add extension blocks to
the current pipeline by clicking the button labeled \[+ Extension block].
2. In dropdown menu, click the template option labeled Empty template.
3. A popup dialog may appear asking you to enter a name for the new extension block.
If it appears, fill in `test number of rows` as the name and click the button labeled \[Save and add block].
4. Paste the following code in the extension block named `test number of rows`:
```python theme={"system"}
@extension('great_expectations')
def validate(validator, *args, **kwargs):
validator.expect_table_row_count_to_be_between(
min_value=1000,
max_value=10000,
)
```
You can add expectations using code or from a JSON object.
See section [Defining expectations](#defining-expectations) below for more details.
5. In the extension block near the bottom, click the input field that says
“Select blocks to run expectations on”.
6. Once you click that input field, a list of blocks from your pipeline will appear.
7. Check the checkbox on the right side of the dropdown to associate that block to this extension block.
8. Click the button labeled Save selected blocks.
9. After you save, a button that is labeled after the name of the block you just selected will appear.
For example, if your block is named `load_api_data_demo`,
then a button labeled load\_api\_data\_demo will appear.
10. Click that button to run your extension block for the block `load_api_data_demo`.
11. The output should look something like this:
```
Calculating Metrics: 0%| | 0/1 [00:00