tqdm.auto.tqdm problem detecting IPython (Spyder and Jupyter QtConsole)

[AJJLagerweij]

Frontmatter

• I have marked all applicable categories:
  • exception-raising bug
  • visual output bug
  • documentation request (i.e. "X is missing from the documentation." If instead I want to ask "how to use X?" I understand [StackOverflow#tqdm] is more appropriate)
  • new feature request
• I have visited the [source website], and in particular
  read the [known issues]
• I have searched through the [issue tracker] for duplicates
• I have mentioned version numbers, operating system and
  environment, where applicable:

  import tqdm, sys
  print(tqdm.__version__, sys.version, sys.platform)
  
  >>> 4.54.1 3.8.5 (default, Sep  4 2020, 07:30:14) 
  >>> [GCC 7.3.0] linux

Problem, Example

The tqdm.auto.tqdm tries to serve the nicely looking tqdm.notebook.tqdm class to my IPython console. The behaviour was tested using:

• IPython 7.19.0
• Spyder 4.2.0
• Jupyter Console 6.2.0
• Python 3.8.5
  all obtained through a conda installation.

The shortest code to recreate the problem is with the following script

# Import packages.
from tqdm.auto import tqdm
from time import sleep

# Keep track of sleep.
for _ in tqdm(range(6, 13)):
    sleep(1)

Which works from the terminal, using basic class tqdm.tqdm:

Works from Jupyter Notebook using the decorated class tqdm.notebook.tqdm:

Fails in IPython, via Jupyter Console, Jupyter QtConsole and Spyder. It tries to serve the tqdm.notebook.tqdm class while this cannot be displayed properly. Resulting in the following message HBox(children=(HTML(value=''), FloatProgress(value=0.0, max=7.0), HTML(value=''))).

Background

I happened upon this issue when using the tqdm.contrib.concurrent.process_map function. I created an issuejupyter/qtconsole#461 on the Jupyter QtConsole git, which after a short discussion pointed out the solution for my specific case. I was also pointed out the difficulties of detecting the frontend and possible ways to solve that.

Because the solution was including the optional key tqdm_class,
see the documentation, it became apparent that there is an issue in the tqdm.auto.tqdm class.

Related Issues

The following issues are related, but far older and closed: #747, #645, #394.

[casperdcl]

As pointed out in jupyter/qtconsole#461 (comment) and in the FAQs (try using tqdm.autonotebook instead of tqdm.auto to see the warning):

By design, in the Jupyter architecture it's not possible to tell what frontend (Notebook or Qtconsole) sent something to run to a kernel. So it's not possible for tqdm to reliably send back to the frontend only the html or terminal representations. About this, it's also important to understand that multiple frontends can be connected to a single kernel at the same time.

Yup this annoyance. Will see about sending both representations...

[AJJLagerweij]

Yes, when running tqdm.autonotebook I do obtain the following clear warning:

TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)

Whereas the documentation of tqdm.auto indicates that it does suppress this warning if it selects tqdm.autonotebook. Could anyone explain why it is suppressed, that must have a reason?

It is quite clear what happens when calling these two functions, (tqdm.auto vs tqdm.autonotebook), but when these functions are called indirectly it is difficult to understand the behaviour. Do you have any idea how to communicate this behaviour more clearly?

Lastly I want to propose a quick and dirty fix, but I'm unsure whether this is possible in python. I would like a warning to be printed with an explanation regarding tqdm.tqdm vs tqdm.notebook.tqdm when the console displays HBox(children=(HTML(value=''), FloatProgress(value=0.0, max= ), HTML(value=''))). However, this would require the code to verify what was actually printed in the console/terminal. Is this possible?

[casperdcl]

actually I think this issue is also essentially a duplicate of #937, #1035 (i.e. same solution - support both HTML & text rendering).

I never managed to fix it because of jupyter-widgets/ipywidgets#2850 but maybe things have changed and someone can open a PR?

[casperdcl]

Could anyone explain why it is suppressed, that must have a reason

because 99.9% of users wanting to auto-detect Notebook & CLI (tqdm.autonotebook) don't want a warning since they never use jupyter console. They didn't want the warning so much that I relented and implemented tqdm.auto

[casperdcl]

can you try tqdm.auto from #1099 (pip install -U "git+https://github.com/tqdm/tqdm.git@devel#egg=tqdm")?

[AJJLagerweij]

I've tried the dev version, and see the appropriate progress bar appear using tqdm.auto.tqdm. However, a TqdmDeprecationWarning causes it to error out. This is not the case for tqdm.tqdm.

import tqdm
print(tqdm.__version__)

>>> 4.54.2.dev6+g5b9961f

# Import and run basic tqdm version
from tqdm import tqdm
from time import sleep

# Track sleep.
for _ in tqdm(range(6, 13)):
    sleep(1)

>>> 100%|██████████| 7/7 [00:07<00:00,  1.00s/it]

# Import and run automatic tqdm version
from tqdm.auto import tqdm
from time import sleep

# Track sleep.
for _ in tqdm(range(6, 13)):
    sleep(1)
    
>>> 0%|          | 0/7 [00:00<?, ?it/s]
TqdmDeprecationWarning: Please use `tqdm.gui.tqdm(...)` instead of `tqdm(..., gui=True)`
---------------------------------------------------------------------------
TqdmDeprecationWarning                    Traceback (most recent call last)
<ipython-input-3-6d59c09eeffd> in <module>
      3 
      4 # Track sleep.
----> 5 for _ in tqdm(range(6, 13)):
      6     sleep(1)
      7 

~/.anaconda3/envs/tqdm/lib/python3.8/site-packages/tqdm/notebook.py in __iter__(self, *args, **kwargs)
    258     def __iter__(self, *args, **kwargs):
    259         try:
--> 260             for obj in super(tqdm_notebook, self).__iter__(*args, **kwargs):
    261                 # return super(tqdm...) will not catch exception
    262                 yield obj

~/.anaconda3/envs/tqdm/lib/python3.8/site-packages/tqdm/std.py in __iter__(self)
   1147 
   1148         if getattr(self, 'sp', None) is None:
-> 1149             raise TqdmDeprecationWarning(
   1150                 "Please use `tqdm.gui.tqdm(...)` instead of"
   1151                 " `tqdm(..., gui=True)`\n",

TqdmDeprecationWarning: Please use `tqdm.gui.tqdm(...)` instead of `tqdm(..., gui=True)`

While the tqdm.auto is used I see the jupyter qtconsolse print the following errors and warnings in my terminal window, although I'm not sure if that is useful.

[JupyterQtConsoleApp] ERROR | No such comm target registered: jupyter.widget
[JupyterQtConsoleApp] ERROR | No such comm target registered: jupyter.widget
[JupyterQtConsoleApp] ERROR | No such comm target registered: jupyter.widget
[JupyterQtConsoleApp] ERROR | No such comm target registered: jupyter.widget
[JupyterQtConsoleApp] ERROR | No such comm target registered: jupyter.widget
[JupyterQtConsoleApp] ERROR | No such comm target registered: jupyter.widget
[JupyterQtConsoleApp] ERROR | No such comm target registered: jupyter.widget
[JupyterQtConsoleApp] ERROR | No such comm target registered: jupyter.widget
[JupyterQtConsoleApp] ERROR | No such comm target registered: jupyter.widget
[JupyterQtConsoleApp] ERROR | No such comm target registered: jupyter.widget
[JupyterQtConsoleApp] ERROR | No such comm target registered: jupyter.widget
[JupyterQtConsoleApp] WARNING | No such comm: b6b387215da8491a8714e0a86308bdfb
[JupyterQtConsoleApp] WARNING | No such comm: 065220269acc455a81e526facb2fa525
[JupyterQtConsoleApp] WARNING | No such comm: 43f90b4fae734a3990b4669969cc0557

[casperdcl]

@AJJLagerweij ah yes ofc, can you try again now?

In jupyter console + tqdm.auto, you should get a bar which doesn't update (afaik there's no way to force an update of a widget in jupyter console - but at least it doesn't crash).

[AJJLagerweij]

Wow, thanks, it seems to work as you expect.

# Import packages.
import tqdm
print(tqdm.__version__)

>>> 4.54.2.dev8+ge225d9a

# Import tqdm.auto.
from tqdm.auto import tqdm
from time import sleep

# Track sleep.
for _ in tqdm(range(6, 13)):
    sleep(1)

>>> 0%|          | 0/7 [00:00<?, ?it/s]

No crashes or errors and as you mention it stays at 0% till the end. You mention that there is no way to force updating in a Jupyter console. I'll ask them if we can do something about that, but before that I would like to understand the issue. After all the normal tqdm progress bar updates as I'm expecting it to do.

[casperdcl]

there is no way to force updating in a Jupyter console

Specifically, no way to force update a widget afaik. You can always use plain tqdm.tqdm though.

The issue is tqdm.notebook displays a widget which has an HTML and a text representation. It will update both representations but it's up to the frontend (i.e. browser/notebook or console) to actually update the display...

[AJJLagerweij]

Thanks that it is clear, I'll communicate that with the Jupyter community. Let's see if they can implement that.

[ianhi]

Should this be closed? With latest versions I see the same stuck at zero percent. I also still get these errors in the console

I think the way the comms work is that there is a per-frontend extension that manages setting them up. widgetnbextension for classic notebook, jupyterlab-widgets for jupyterlab, something else for voila and I guess there's no equivalent for qtconsole.

So you can send still send updates but they just run into a wall on the frontend. Probably the best way to will be to detect if the comm is actually set up properly and then remove the widget from the mimebundle and update the display however you do in ipython. Though I don't see an obvious way to detect this :/ (I will ask around about this)

[ianhi]

(I will ask around about this)

Part of that asking is here: https://gitter.im/jupyter-widgets/Lobby?at=602c8e2c93539e23437de478

[casperdcl]

@ianhi this is closed simply because it seems to be an upstream issue which tqdm can't do anything about