Data Validation

Dataset validation module for SpaCy training data quality assurance.

This module provides functionality to validate SpaCy NER and text classification datasets, generate quality reports with statistics, and create visualizations of dataset characteristics.

class spacylize.validator.DataValidator(dataset_path, output_folder, task=None)

Bases: object

Validator for SpaCy datasets that generates quality reports and visualizations.

Analyzes datasets to compute statistics about document lengths, entity/category distributions, and other quality metrics. Supports both NER and text classification tasks. Outputs both JSON reports and visualization plots.

Parameters:

• dataset_path (str)

• output_folder (str)

• task (str)

dataset_path

Path to the SpaCy dataset file (.spacy).

output_folder

Folder where reports will be saved.

task

The SpaCy task type (‘ner’ or ‘textcat’).

nlp

Blank SpaCy language model for processing.

json_path

Path where JSON report will be saved.

png_path

Path where visualization plots will be saved.

__init__(dataset_path, output_folder, task=None)

Initialize the DataValidator.

Parameters:

• dataset_path (str) – Path to the SpaCy dataset file (.spacy) to validate.

• output_folder (str) – Directory where validation reports will be saved.

• task (str) – Optional task type (‘ner’ or ‘textcat’). Auto-detects if not specified.

Raises:

ValueError – If the task is invalid or cannot be detected.

run()

Run the validation process and generate reports.

Analyzes the dataset using task-specific validation logic, computes statistics, and generates both a JSON report and visualization plots.

Raises:

ValueError – If the task type is not supported.