How to use tqdm to log progress bars in cloudwatch logs?

[rklarpit]

I want to log progress of my API in cloudwatch. Is it possible to do the same?

[austinmw]

Hey, I'm also curious how to do this. I get the following ugly printout when using cloudwatch:

It looks like it doesn't support carriage return?

[0x26res]

I think it's because of the way tqdm erases the progress bar and redraw it in place. But cloudwaatch can not go back in time.

You could use position=-1 so it only appends to the log, and then I think it should display each progress bar log in one cloudwatch entry (rather than one big blob).

from time import sleep

from tqdm import tqdm

if __name__ == "__main__":
    for i in tqdm(range(9), position=-1):
        sleep(0.1)

Then you'd get:

  0%|          | 0/9 [00:00<?, ?it/s]
 11%|█         | 1/9 [00:00<00:00,  9.81it/s]
 22%|██▏       | 2/9 [00:00<00:00,  9.76it/s]
 33%|███▎      | 3/9 [00:00<00:00,  9.73it/s]
 44%|████▍     | 4/9 [00:00<00:00,  9.65it/s]
 56%|█████▌    | 5/9 [00:00<00:00,  9.73it/s]
 67%|██████▋   | 6/9 [00:00<00:00,  9.70it/s]
 78%|███████▊  | 7/9 [00:00<00:00,  9.61it/s]
 89%|████████▉ | 8/9 [00:00<00:00,  9.57it/s]
100%|██████████| 9/9 [00:00<00:00,  9.63it/s]

[0x26res]

@casperdcl I saw you recently added this feature: #1491

For context, when running stuff in AWS, I'm trying to have each tqdm statement on a different line.
Otherwise it looks like a giant blob in aws cloudwatch, and we get warnings all over the place.

Previously I had to do something like this:

import os

IS_AWS_BATCH: bool = "AWS_EXECUTION_ENV" in os.environ
TQDM_POSITION: Optional[int] = -1 if IS_AWS_BATCH else None

# ....

for _ in tqdm(data, position=TQDM_POSITION):
  pass

It works but I need to remember to import TQDM_POSITION and plug the position for every tqdm call.

I was hopping with your recent change I could use export TQDM_POSITION=-1. But it doesn't work.

TQDM_POSITION='-1' python -c "
import tqdm
from time import sleep

for _ in tqdm.tqdm(range(5)):
    sleep(0.1)
"

Gives me:

  File "<string>", line 5, in <module>
  File "/lib/python3.10/site-packages/tqdm/std.py", line 1091, in __init__
    self.pos = self._get_free_pos(self) if position is None else -position
TypeError: bad operand type for unary -: 'str'
Exception ignored in: <function tqdm.__del__ at 0x104fdd090>
Traceback (most recent call last):
  File "/lib/python3.10/site-packages/tqdm/std.py", line 1147, in __del__
    self.close()
  File "/lib/python3.10/site-packages/tqdm/std.py", line 1273, in close
    pos = abs(self.pos)
AttributeError: 'tqdm' object has no attribute 'pos'

Do you know what I'm doing wrong? And would you be able to suggest another way to achieve this?

[casperdcl]

thanks @0x26res - that's a bug. The position argument is default-None so type inference doesn't work, i.e. TQDM_POSITION='-1' results in tqdm(position='-1') not tqdm(position=-1).

Will fix shortly :)

btw I think your work-around will fix #1319 too.

[N-Demir]

Why doesn't

from time import sleep

from tqdm import tqdm

if __name__ == "__main__":
    for i in tqdm(range(9), position=-1):
        sleep(1)

print each update to the progress bar on a new line? According to the earlier posts it seems like it should?

[0x26res]

Why doesn't

from time import sleep

from tqdm import tqdm

if __name__ == "__main__":
    for i in tqdm(range(9), position=-1):
        sleep(1)

print each update to the progress bar on a new line? According to the earlier posts it seems like it should?

@N-Demir it depends on where you run your python program.

For example if I use the terminal, they all go on the same line.

But if I run in pycharm they all go on a different line. And if I run in AWS with cloudwatch, again, they all go on a different line

[miohtama]

I have released a package called tqdm-loggable that automatically converts progress bars to time-interval logged messages.

When the application is running in a headless mode, progress bar are not displayed, but instead you get a log message at a regular interval during the progress bar running:

tqdm_logging.py     :139  2022-09-21 12:13:56,307 Progress on:Sample progress 21.0kit/60.0kit rate:1,982.9it/s remaining:00:19 elapsed:00:10 postfix:Current time=2022-09-21 10:13:55.801787
tqdm_logging.py     :139  2022-09-21 12:14:06,392 Progress on:Sample progress 41.0kit/60.0kit rate:1,984.1it/s remaining:00:09 elapsed:00:20 postfix:Current time=2022-09-21 10:14:05.890220