Capture measurements in CSV based on time intervals

[seriab]

• CodeCarbon version: 2.2.2
• Python version: 3.11.5
• Operating System: Linux-6.2.0-35-generic-x86_64-with-glibc2.35

Description

I have implemented a method that enables the capture and storage of consumption logs (CSV format) in parallel every second during the execution of my algorithm. Nevertheless, I would be interested in exploring whether there are more efficient alternatives from a technical or academic standpoint to accomplish this process. Below, I am providing the code of my current solution for review and suggestions.

What I Did

from codecarbon import EmissionsTracker
import threading

pthread=1
def fthread():
    while pthread:
        tracker.flush()
        time.sleep(1)

# Creates an instance of a thread and passes the function is executed.
eng_measure = threading.Thread(target=fthread)

# Training 
tracker = EmissionsTracker(project_name="Test",tracking_mode="machine")
tracker.start()
eng_measure.start()
try:
    # code
finally:
    pthread = 0
    eng_measure.join()
    emissions = tracker.stop()

[benoit-cty]

Hello,
Doing it every second could be quite disk intensive, and useless as CodeCarbon only update measure every 15 seconds.

For a better solution we could update CodeCarbon to write to a csv file in the same place we call the API and Prometheus : https://github.com/mlco2/codecarbon/blob/master/codecarbon/emissions_tracker.py#L703

So you will just need to do:
tracker = EmissionsTracker(project_name="Test",tracking_mode="machine", csv_run_name="emissions_test54.csv")
And CodeCarbon will update the CSV every 8 x 15 seconds (api_call_interval * measure_power_secs).

What do you think of this ?
Maybe CodeCarbon could generate a run_id to prevent the need of giving the file name.

[seriab]

Thank you very much for your response. The proposed update would be very useful. I will be looking forward to your reply.

[NerfedAce]

Hi! I noticed the previous PRs related to this issue were closed.

I’d be interested in working on this feature if it’s still needed. I’ll first review the previous implementations and maintainer feedback before opening a PR.

[harsith1410]

Hi @benoit-cty,

I'd love to take a shot at implementing this feature if it's still open!

I've reviewed your feedback regarding the disk-intensive nature of 1-second flushes and your blueprint for linking the CSV write to the existing API/Prometheus data loop inside codecarbon/emissions_tracker.py (running every api_call_interval * measure_power_secs). I also plan to look into incorporating the auto-generated run_id for file naming as you suggested.

Since the previous PRs were closed, I'm happy to dive in, set it up locally, and work on a clean implementation. Could you please assign this to me? Thanks!

[sankalpsthakur]

I'd like to work on this — planning a minimal change so CSV measurement export can capture rows on a configurable time interval, with tests.

[sankalpsthakur]

Opened PR with the maintainer-suggested approach (interval CSV write on the API/Prometheus live cadence via opt-in csv_run_name): #1270