Formatting Python adding extra characters in code.

[TheWizardofGauze]

Type: Bug

Tried testing with both Black and autopep8 with similar results, Extension Bisect pointed at ms-python.python.

I was editing a file for my bot and noticed that it was adding characters when I saved, with both auto format and manual, and only towards the end where my new code was. Not really sure if there's any steps to outline but I can give an example.

Before:

@bot.command()
@commands.is_owner()
async def example(ee,aa,ff,gg,amsdk,adsd,sadasd,sad,sd,asd,a,ad,s):
    test=1+3+ 2
@bot.command()
@commands.is_owner()
async def shutdown(ctx):
    await ctx.message.add_reaction("👋🏼")
    await bot.close()

After:

@bobot.command()
@commands.is_owner()
async def example(ee,aa,,ff,,gg,,amssdk,adssd,saddasd,sadd,sd,,asdd,a,a d,,s)::
    test=1+13++3 +22
@

@bot.command()
@commands.is_owner()
async def shutdown(ctx):
    await ctx.message.add_reaction("👋🏼")
    await bot.close()

Extension version: 2023.20.0
VS Code version: Code 1.84.2 (1a5daa3a0231a0fbba4f14db7ec463cf99d7768e, 2023-11-09T10:51:52.184Z)
OS version: Windows_NT x64 10.0.19045
Modes:

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz (12 x 3696)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled
Load (avg)	undefined
Memory (System)	31.92GB (22.12GB free)
Process Argv	--crash-reporter-id 98d07474-2978-4a7b-808e-bb2602f5f283
Screen Reader	no
VM	0%

A/B Experiments

vsliv368:30146709
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
vstes627:30244334
vslsvsres303:30308271
vserr242:30382549
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscorecescf:30445987
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593:30376534
pythonvs932:30410667
py29gd2263:30880072
vsclangdc:30486549
c4g48928:30535728
dsvsc012:30540252
pynewext54:30695312
azure-dev_surveyone:30548225
2e4cg342:30602488
89544117:30613380
showlangstatbar:30737416
pythonfmttext:30731395
fixshowwlkth:30771522
showindicator:30805244
pythongtdpath:30769146
i26e3531:30792625
pythonnosmt12:30797651
pythonidxpt:30866567
pythonnoceb:30805159
asynctok:30898717
dsvsc013:30795093
dsvsc014:30804076
dsvsc015:30845448
pythontestfixtcf:30871695
pythonregdiag2:30871582
pyreplss1:30897532
pythonmypyd1:30879173
pythoncet0:30885854
pythontbext0:30879054
accentitlementst:30887150
dsvsc016cf:30886111
dsvsc017:30886112
dsvsc018cf:30886115
aa_t_chat:30882232

[karthiknadig]

@TheWizardofGauze can you add the following two settings to User settings.json?

"python.trace.server" : "verbose",
"black-formatter.trace.server": "verbose",

These are hidden settings, you won't find them in the UI. After adding these settings, Go to Output > Python Language Server, and clear the logs, do the same with Output > Black Formatter. Then reproduce the bug, and share the logs from both Python Language Server and Black Formatter.

[TheWizardofGauze]

@karthiknadig
Black Formatter.txt
Python Language Server.txt

[karthiknadig]

@TheWizardofGauze Thanks for the logs. Just one more bit, from black-formatter can you share the request textDocument/didOpen part from the logs.

[TheWizardofGauze]

@karthiknadig
didOpen.txt

[karthiknadig]

With the exact code and extensions, I am not able to repo this. Can you share the contents of your environment?

> python -m pip list

I am suspecting something is interfering with diff calculation.

[TheWizardofGauze]

@karthiknadig

Package                    Version
-------------------------- ------------
absl-py                    0.12.0
aiodns                     3.0.0
aiohttp                    3.8.5
aiohttp-json-rpc           0.13.3
aiosignal                  1.3.1
aiosqlite                  0.17.0
altgraph                   0.17
aniso8601                  9.0.1
appdirs                    1.4.4
apsw                       3.43.1.0
astroid                    2.4.2
astunparse                 1.6.3
async-timeout              4.0.3
attrs                      23.1.0
autopep8                   1.5.4
Babel                      2.12.1
backoff                    2.2.1
backports.zoneinfo         0.2.1
beautifulsoup4             4.9.3
big-sleep                  0.9.1
black                      21.10b0
bleach                     3.2.1
boto3                      1.17.40
botocore                   1.20.40
braceexpand                0.1.7
Brotli                     1.1.0
bs4                        0.0.1
cachetools                 4.2.0
censys                     2.1.9
certifi                    2020.12.5
cffi                       1.15.1
chardet                    4.0.0
charset-normalizer         2.1.1
chromedriver-autoinstall   1.0
chromedriver-autoinstaller 0.6.2
click                      8.1.7
click-plugins              1.1.1
clip-anytorch              2.4.0
CoCa-pytorch               0.0.6
colorama                   0.4.6
commonmark                 0.9.1
contextlib2                21.6.0
cssselect                  1.1.0
cycler                     0.10.0
dalle-mini                 0.1.0
dalle2-pytorch             0.6.16
decorator                  4.4.2
deep-daze                  0.11.1
dicttoxml                  1.7.4
difPy                      3.0.8
discord                    2.0.0
discord.py                 2.3.2
docker-pycreds             0.4.0
docutils                   0.16
einops                     0.4.1
einops-exts                0.0.3
embedding-reader           1.4.2
emoji                      1.7.0
filelock                   3.7.1
fire                       0.4.0
Flask                      2.1.0
Flask-Cors                 3.0.9
Flask-RESTful              0.3.8
flatbuffers                1.12
flax                       0.3.3
frozenlist                 1.4.0
fsspec                     2022.5.0
ftfy                       5.9
future                     0.18.2
fuzzywuzzy                 0.18.0
gast                       0.3.3
geocoder                   1.38.1
geoip2                     4.1.0
gifmaker                   1.5
gitdb                      4.0.9
GitPython                  3.1.27
google-auth                1.28.0
google-auth-oauthlib       0.4.4
google-pasta               0.2.0
grpcio                     1.32.0
gTTS                       2.3.1
h5py                       2.10.0
html2text                  2020.1.16
huggingface-hub            0.7.0
idna                       3.4
imageio                    2.9.0
imgcat                     0.5.0
importlib-metadata         6.8.0
ip2geotools                0.1.5
IP2Location                8.5.1
isort                      5.6.4
itsdangerous               2.1.2
iv                         1.0.5
jax                        0.2.10
Jinja2                     3.1.2
jmespath                   0.10.0
Js2Py                      0.71
Keras-Preprocessing        1.1.2
keyring                    21.8.0
kiwisolver                 1.1.0
kornia                     0.6.5
lazy-object-proxy          1.4.3
lpips                      0.1.4
lxml                       4.6.2
macholib                   1.14
Markdown                   3.4.4
markdown-it-py             3.0.0
MarkupSafe                 2.1.1
matplotlib                 3.2.0
maxminddb                  2.0.3
mccabe                     0.6.1
mdurl                      0.1.2
msgpack                    1.0.4
multidict                  6.0.4
mypy-extensions            0.4.3
nmap                       0.0.1
numpy                      1.19.5
oauthlib                   3.1.0
opt-einsum                 3.3.0
orjson                     3.9.7
packaging                  23.1
pandas                     1.4.2
pathspec                   0.9.0
pathtools                  0.1.2
pefile                     2019.4.18
phonetics                  1.0.5
Pillow                     7.0.0
pip                        23.1
pip-review                 1.1.0
pkginfo                    1.6.1
platformdirs               3.10.0
playsound                  1.3.0
promise                    2.3
protobuf                   3.15.6
psutil                     5.9.5
py2exe                     0.10.1.0
pyarrow                    7.0.0
pyasn1                     0.4.8
pyasn1-modules             0.2.8
pycares                    4.3.0
pycodestyle                2.6.0
pycparser                  2.21
pycryptodome               3.15.0
pycurl                     7.43.0.5
pydantic                   1.9.1
pyfiglet                   0.8.post1
Pygments                   2.16.1
pyinstaller                5.0
pyinstaller-hooks-contrib  2022.4
pyjsparser                 2.7.1
pylint                     2.6.0
PyNaCl                     1.4.0
pyparsing                  2.4.6
pypresence                 4.2.1
pyquery                    1.4.3
python-dateutil            2.8.2
python-dotenv              0.19.0
python-Levenshtein-wheels  0.13.2
pytorch-pretrained-biggan  0.1.1
pytorch-ranger             0.1.1
pytz                       2023.3.post1
pytz-deprecation-shim      0.1.0.post0
pywin32-ctypes             0.2.0
PyYAML                     6.0.1
rapidfuzz                  3.3.0
ratelim                    0.1.6
readme-renderer            28.0
Red-Commons                1.0.0
Red-DiscordBot             3.5.5
Red-Lavalink               0.11.0
regex                      2021.3.17
requests                   2.28.1
requests-oauthlib          1.3.0
requests-toolbelt          0.9.1
resize-right               0.0.2
rfc3986                    1.4.0
rich                       13.5.2
rotary-embedding-torch     0.1.5
rsa                        4.7.2
s3transfer                 0.3.6
scapy                      2.4.4
schema                     0.7.5
scipy                      1.7.3
selenium                   3.141.0
sentry-sdk                 1.5.12
setproctitle               1.2.3
setuptools                 41.2.0
shodan                     1.28.0
shortuuid                  1.0.9
siren-pytorch              0.0.9
six                        1.16.0
smmap                      5.0.0
soupsieve                  2.2.1
tensorboard                2.4.1
tensorboard-plugin-wit     1.8.0
tensorflow-estimator       2.4.0
tensorflow-gpu             2.4.0
termcolor                  1.1.0
tokenizers                 0.12.1
toml                       0.10.2
tomli                      1.2.2
torch                      1.10.0+cu113
torch-fidelity             0.3.0
torch-optimizer            0.1.0
torchaudio                 0.10.0+cu113
torchmetrics               0.9.1
torchvision                0.11.1+cu113
tqdm                       4.64.0
transformers               4.20.0.dev0
twine                      3.3.0
typed-ast                  1.4.2
typing_extensions          4.7.1
tzdata                     2021.5
tzlocal                    4.1
Unidecode                  1.3.4
urllib3                    1.26.2
vector-quantize-pytorch    0.6.0
vqgan-jax                  0.0.1
wandb                      0.12.17
wcwidth                    0.2.5
webdataset                 0.2.5
webencodings               0.5.1
websockets                 8.1
Werkzeug                   2.1.2
wheel                      0.38.4
Wikidata                   0.7.0
wikipedia                  1.4.0
wonderwords                2.2.0
wptools                    0.4.17
wrapt                      1.12.1
x-clip                     0.6.1
XlsxWriter                 3.0.3
yarl                       1.9.2
zipp                       3.16.2

[karthiknadig]

What is your python version and what is your file encoding?

[TheWizardofGauze]

Python 3.8.10 and UTF-8

[karthiknadig]

I have been trying to repo this locally. For some reason I am not able to :( .

[TheWizardofGauze]

Disappointing to hear, but I appreciate your effort to help. It seems like it's only effecting my main file so I can still use it with everything else in the meantime. I'm in the process of building a new PC anyway so I'll see if it goes away with a fresh install.

[karthiknadig]

A user figured out the cause for this.

Closing in favor of: #395 as that has more details.
The emojis in your code is what is causing it to trip up.