TCPDashboard/docs/decisions/ADR-002-common-package-refactor.md

ADR-002: Common Package Refactoring

Status

Accepted and Implemented

Context

The common package contained several large, monolithic files that were becoming difficult to maintain and extend. The files included:

• aggregation.py
• indicators.py
• validation.py
• transformation.py
• data_types.py

These files handled critical functionality but were growing in complexity and responsibility, making it harder to:

• Understand and maintain individual components
• Test specific functionality
• Add new features without affecting existing code
• Ensure proper separation of concerns

Decision

We decided to refactor the common package into a more modular structure by:

1. Splitting Large Files into Sub-packages:

   • Created aggregation/ package with specialized modules
   • Created indicators/ package with focused components
   • Maintained core data types in a single, well-structured file

2. Improving Validation:

   • Enhanced modularity of validation system
   • Added clearer validation rules and messages
   • Maintained backward compatibility

3. Enhancing Transformation:

   • Added safety limits system
   • Improved error handling
   • Better separation of transformation concerns

4. Preserving Data Types:

   • Reviewed and verified data_types.py structure
   • Maintained as single file due to good organization
   • Documented existing patterns

Consequences

Positive

• Better code organization and maintainability
• Clearer separation of concerns
• Easier to test individual components
• More focused and cohesive modules
• Better safety with new limits system
• Improved documentation and examples
• Easier to extend with new features

Negative

• Slightly more complex import paths
• Need to update existing documentation
• Initial learning curve for new structure

Neutral

• Need to maintain more files
• More granular version control
• Different organization pattern from original

Alternatives Considered

Keep Monolithic Structure

Rejected because:

• Growing complexity
• Difficult maintenance
• Hard to test
• Poor separation of concerns

Complete Microservices Split

Rejected because:

• Too complex for current needs
• Would introduce unnecessary overhead
• Not justified by current scale

Hybrid Approach (Selected)

Selected because:

• Balances modularity and simplicity
• Maintains good performance
• Easy to understand and maintain
• Allows for future growth

Implementation Notes

Phase 1: Aggregation and Indicators

• Split into focused sub-packages
• Added proper interfaces
• Maintained backward compatibility
• Added comprehensive tests

Phase 2: Validation and Transformation

• Enhanced validation system
• Added safety limits
• Improved error handling
• Updated documentation

Phase 3: Verification

• Reviewed data types
• Ran comprehensive tests
• Updated documentation
• Verified no regressions

Migration Guide

For Developers

1. Update imports to use new package structure
2. Review new safety limits in transformation
3. Check validation error handling
4. Update any custom extensions

For Maintainers

1. Familiarize with new package structure
2. Review new testing patterns
3. Understand safety limit system
4. Follow modular development pattern

References

• Original task list: tasks/refactor-common-package.md
• Documentation standards: docs/documentation.mdc
• Test coverage reports
• Code review feedback