49 lines
1.7 KiB
Markdown
49 lines
1.7 KiB
Markdown
|
# System Utilities
|
||
|
|
|
||
|
|
This document describes the system utility functions found in `cycles/utils/system.py`.
|
||
|
|
|
||
|
|
## Overview
|
||
|
|
|
||
|
|
The `system.py` module provides utility functions related to system information and resource management. It currently includes a class `SystemUtils` for determining optimal configurations based on system resources.
|
||
|
|
|
||
|
|
## Classes and Methods
|
||
|
|
|
||
|
|
### `SystemUtils`
|
||
|
|
|
||
|
|
A class to provide system-related utility methods.
|
||
|
|
|
||
|
|
#### `__init__(self, logging=None)`
|
||
|
|
|
||
|
|
- **Description**: Initializes the `SystemUtils` class.
|
||
|
|
- **Parameters**:
|
||
|
|
- `logging` (optional): A logging instance to output information. Defaults to `None`.
|
||
|
|
|
||
|
|
#### `get_optimal_workers(self)`
|
||
|
|
|
||
|
|
- **Description**: Determines the optimal number of worker processes based on available CPU cores and memory.
|
||
|
|
The heuristic aims to use 75% of CPU cores, with a cap based on available memory (assuming each worker might need ~2GB for large datasets). It returns the minimum of the workers calculated by CPU and memory.
|
||
|
|
- **Parameters**: None.
|
||
|
|
- **Returns**: `int` - The recommended number of worker processes.
|
||
|
|
|
||
|
|
## Usage Examples
|
||
|
|
|
||
|
|
```python
|
||
|
|
from cycles.utils.system import SystemUtils
|
||
|
|
|
||
|
|
# Initialize (optionally with a logger)
|
||
|
|
# import logging
|
||
|
|
# logging.basicConfig(level=logging.INFO)
|
||
|
|
# logger = logging.getLogger(__name__)
|
||
|
|
# sys_utils = SystemUtils(logging=logger)
|
||
|
|
sys_utils = SystemUtils()
|
||
|
|
|
||
|
|
|
||
|
|
optimal_workers = sys_utils.get_optimal_workers()
|
||
|
|
print(f"Optimal number of workers: {optimal_workers}")
|
||
|
|
|
||
|
|
# This value can then be used, for example, when setting up a ThreadPoolExecutor
|
||
|
|
# from concurrent.futures import ThreadPoolExecutor
|
||
|
|
# with ThreadPoolExecutor(max_workers=optimal_workers) as executor:
|
||
|
|
# # ... submit tasks ...
|
||
|
|
# pass
|
||
|
|
```
|