Cycles/docs/utils_system.md

System Utilities

This document describes the system utility functions found in cycles/utils/system.py.

Overview

The system.py module provides utility functions related to system information and resource management. It currently includes a class SystemUtils for determining optimal configurations based on system resources.

Classes and Methods

SystemUtils

A class to provide system-related utility methods.

__init__(self, logging=None)

• Description: Initializes the SystemUtils class.
• Parameters:
  • logging (optional): A logging instance to output information. Defaults to None.

get_optimal_workers(self)

• Description: Determines the optimal number of worker processes based on available CPU cores and memory. The heuristic aims to use 75% of CPU cores, with a cap based on available memory (assuming each worker might need ~2GB for large datasets). It returns the minimum of the workers calculated by CPU and memory.
• Parameters: None.
• Returns: int - The recommended number of worker processes.

Usage Examples

from cycles.utils.system import SystemUtils

# Initialize (optionally with a logger)
# import logging
# logging.basicConfig(level=logging.INFO)
# logger = logging.getLogger(__name__)
# sys_utils = SystemUtils(logging=logger)
sys_utils = SystemUtils()


optimal_workers = sys_utils.get_optimal_workers()
print(f"Optimal number of workers: {optimal_workers}")

# This value can then be used, for example, when setting up a ThreadPoolExecutor
# from concurrent.futures import ThreadPoolExecutor
# with ThreadPoolExecutor(max_workers=optimal_workers) as executor:
#     # ... submit tasks ...
#     pass