TCPDashboard/tasks/refactor-common-package.md

## Relevant Files

- `data/common/aggregation.py` - To be broken into a sub-package.
- `data/common/indicators.py` - To be broken into a sub-package and have a bug fixed.
- `data/common/validation.py` - To be refactored for better modularity.
- `data/common/transformation.py` - ✅ Refactored into transformation package with safety limits.
- `data/common/data_types.py` - To be updated with new types from other modules.
- `data/common/__init__.py` - ✅ Updated to reflect the new package structure.
- `tests/` - Existing tests will need to be run after each step to ensure no regressions.

### Notes

- This refactoring focuses on improving modularity by splitting large files into smaller, more focused modules, as outlined in the `refactoring.mdc` guide.
- Each major step will be followed by a verification phase to ensure the application remains stable.

## Tasks

- [x] 1.0 Refactor `aggregation.py` into a dedicated sub-package.
  - [x] 1.1 Create safety net tests to ensure the aggregation logic still works as expected.
  - [x] 1.2 Create a new directory `data/common/aggregation`.
  - [x] 1.3 Create `data/common/aggregation/__init__.py` to mark it as a package.
  - [x] 1.4 Move the `TimeframeBucket` class to `data/common/aggregation/bucket.py`.
  - [x] 1.5 Move the `RealTimeCandleProcessor` class to `data/common/aggregation/realtime.py`.
  - [x] 1.6 Move the `BatchCandleProcessor` class to `data/common/aggregation/batch.py`.
  - [x] 1.7 Move the utility functions to `data/common/aggregation/utils.py`.
  - [x] 1.8 Update `data/common/aggregation/__init__.py` to expose all public classes and functions.
  - [x] 1.9 Delete the original `data/common/aggregation.py` file.
  - [x] 1.10 Run tests to verify the aggregation logic still works as expected.

- [x] 2.0 Refactor `indicators.py` into a dedicated sub-package.
  - [x] 2.1 Create safety net tests for indicators module.
  - [x] 2.2 Create a new directory `data/common/indicators`.
  - [x] 2.3 Create `data/common/indicators/__init__.py` to mark it as a package.
  - [x] 2.4 Move the `TechnicalIndicators` class to `data/common/indicators/technical.py`.
  - [x] 2.5 Move the `IndicatorResult` class to `data/common/indicators/result.py`.
  - [x] 2.6 Move the utility functions to `data/common/indicators/utils.py`.
  - [x] 2.7 Update `data/common/indicators/__init__.py` to expose all public classes and functions.
  - [x] 2.8 Delete the original `data/common/indicators.py` file.
  - [x] 2.9 Run tests to verify the indicators logic still works as expected.

- [x] 3.0 Refactor `validation.py` for better modularity.
  - [x] 3.1 Create safety net tests for validation module.
  - [x] 3.2 Extract common validation logic into separate functions.
  - [x] 3.3 Improve error handling and validation messages.
  - [x] 3.4 Run tests to verify validation still works as expected.

- [x] 4.0 Refactor `transformation.py` for better modularity.
  - [x] 4.1 Create safety net tests for transformation module.
  - [x] 4.2 Extract common transformation logic into separate functions.
  - [x] 4.3 Improve error handling and transformation messages.
  - [x] 4.4 Run tests to verify transformation still works as expected.
  - [x] 4.5 Create comprehensive safety limits system.
  - [x] 4.6 Add documentation for the transformation module.
  - [x] 4.7 Delete redundant transformation.py file.

- [x] 5.0 Update `data_types.py` with new types.
  - [x] 5.1 Review and document all data types.
  - [x] 5.2 Add any missing type hints.
  - [x] 5.3 Add validation for data types.
  - [x] 5.4 Run tests to verify data types still work as expected.

- [ ] 6.0 Final verification and cleanup.
  - [x] 6.1 Run all tests to ensure no regressions.
  - [x] 6.2 Update documentation to reflect new structure.
  - [x] 6.3 Review and clean up any remaining TODOs.
  - [ ] 6.4 Create PR with changes.