Handle new naming scheme in newer Jupyter notebook

[mjziebarth]

As mentioned in #1210, this adds handling of a different cell code naming scheme in the newest Jupyter notebook / IPython version (at least as packaged on Arch Linux).

Let me know if anything else can / needs to be done.

Cheers!

[codecov]

Codecov Report

Merging #1214 (c6f7a29) into master (0fa2cb9) will decrease coverage by 5.82%.
The diff coverage is 50.00%.

@@            Coverage Diff             @@
##           master    #1214      +/-   ##
==========================================
- Coverage   94.51%   88.68%   -5.83%     
==========================================
  Files          47       47              
  Lines        7034     7052      +18     
==========================================
- Hits         6648     6254     -394     
- Misses        386      798     +412     

Impacted Files	Coverage Δ
joblib/func_inspect.py	92.34% <50.00%> (-0.47%)	⬇️
joblib/test/test_dask.py	3.73% <0.00%> (-95.15%)	⬇️
joblib/_dask.py	24.13% <0.00%> (-69.96%)	⬇️
joblib/hashing.py	89.47% <0.00%> (-1.76%)	⬇️
joblib/test/test_parallel.py	95.76% <0.00%> (-1.34%)	⬇️
joblib/memory.py	95.37% <0.00%> (-0.19%)	⬇️
joblib/test/test_memory.py	98.62% <0.00%> (+0.01%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0fa2cb9...c6f7a29. Read the comment docs.

[ogrisel]

Thanks @mjziebarth, I pushed a small improvement to your PR. I tested the fix locally and I confirm that it works as expected. There is no easy way to write a test for this so... so be it.

BTW next time, please create your PR from a dedicated branch instead of using master. Otherwise it will be very confusing for your to sync your local joblib repo with the main joblib repo.

[ogrisel]

Thanks @mjziebarth!

[timcoote]

Just checking..... I upgraded to 1.1.0 and found that this invalidated caches created by the earlier version of the module used within a Jupyter notebook. check_call_in_cache fails to find the earlier versions of the function, which, presumably were in different filenames.

Is this as expected: should earlier cached function/args/kwargs sets fail? I'd hoped that this wasn't the case, but I can understand if it is.

[ogrisel]

I believe this is expected but what I would not expect is that the cache would have worked without invalidation using a recent version of jupyter and the older version of joblib.

[NikolayXHD]

Hi, the fix did not work for me, IPython.__version__ '8.4.0'

parts array in func_inspect.py#L151 looks like this:

['', 'tmp', 'ipykernel_25910', '649976777.py']

UPDATE: the Memory cache IS preserved between IPykernel sessions, however, when the cell with defined function is updated, the last part '649976777.py' is changed, effectively invalidating the cache.

I believe joblib.Memory is supposed to invalidate the cache whenever the function code changes, using joblib.func_inspect.get_func_code to detect code change. If so, then variation in last path segment '649976777.py' should be ignored, since it does not imply the change in cached function code.

def get_func_name():
    # ...
    parts[-2] = 'ipykernel'
    # add this line to prevent invalidating cahe on arbitrary change
    # in jupyter cell code
    del parts[-1]