Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem with Python 3.8+ on Windows when network drive is used #3417

Closed
Hofei90 opened this issue Mar 12, 2020 · 75 comments
Closed

Problem with Python 3.8+ on Windows when network drive is used #3417

Hofei90 opened this issue Mar 12, 2020 · 75 comments
Assignees
Milestone

Comments

@Hofei90
Copy link

Hofei90 commented Mar 12, 2020

The cause of the issue: https://bugs.python.org/issue37993

Solution 1

  1. Open Windows Registry regedit.exe
  2. Create a new key named Command Processor at HKEY_CURRENT_USER\Software\Microsoft
    Screen Shot 2021-03-31 at 18 14 29
  3. Create a new DWORD (32-bit) Value named DisableUNCCheck
    Screen Shot 2021-03-31 at 18 16 50
  4. Set the value of DisableUNCCheck to 1 (Hex)
    Screen Shot 2021-03-31 at 18 21 32
  5. Restart your Windows machine.

Solution 2

  1. Download our portable Python 3.7
  2. Extract and replace contents in %HOMEPATH%/.platformio/python3.
  3. Remove %HOMEPATH%/.platformio/penv folder.
  4. Restart VSCode.

Configuration

Operating system:
Windows_NT x64 10.0.17763

PlatformIO Version (platformio --version):
4.2.2b1
and
4.2.1

Python Version
3.8 32bit

Description of problem

During compilation I get the Python Traceback UnicodeDecodeError: 'charmap' codec can't decode byte.

Full Message:

> Executing task in folder IR_Sender_WB: C:\Users\***\.platformio\penv\Scripts\platformio.exe run <

Processing nodemcuv2 (platform: espressif8266; board: nodemcuv2; framework: arduino)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/nodemcuv2.html
PLATFORM: Espressif 8266 2.3.3 > NodeMCU 1.0 (ESP-12E Module)
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES:
- framework-arduinoespressif8266 3.20603.200130 (2.6.3)
- tool-esptool 1.413.0 (4.13)
- tool-esptoolpy 1.20800.0 (2.8.0)
- toolchain-xtensa 2.40802.191122 (4.8.2)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 31 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <IRremoteESP8266> 2.7.4
Building in release mode
Compiling .pio\build\nodemcuv2\src\main.cpp.o
Generating LD script .pio\build\nodemcuv2\ld\local.eagle.app.v6.common.ld
Exception in thread Thread-3:
Traceback (most recent call last):
  File "C:\Users\***\AppData\Local\Programs\Python\Python38-32\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Users\***\AppData\Local\Programs\Python\Python38-32\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "c:\users\***\.platformio\penv\lib\site-packages\platformio\proc.py", line 46, in run
    self.do_reading()
  File "c:\users\***\.platformio\penv\lib\site-packages\platformio\proc.py", line 69, in do_reading
    for byte in iter(lambda: self._pipe_reader.read(1), ""):
  File "c:\users\***\.platformio\penv\lib\site-packages\platformio\proc.py", line 69, in <lambda>
    for byte in iter(lambda: self._pipe_reader.read(1), ""):
  File "C:\Users\***\AppData\Local\Programs\Python\Python38-32\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 167: character maps to <undefined>
Compiling .pio\build\nodemcuv2\lib70a\IRremoteESP8266\IRac.cpp.o
Compiling .pio\build\nodemcuv2\lib70a\IRremoteESP8266\IRrecv.cpp.o
Compiling .pio\build\nodemcuv2\lib70a\IRremoteESP8266\IRsend.cpp.o
Compiling .pio\build\nodemcuv2\lib70a\IRremoteESP8266\IRtext.cpp.o
Compiling .pio\build\nodemcuv2\lib70a\IRremoteESP8266\IRtimer.cpp.o
Compiling .pio\build\nodemcuv2\lib70a\IRremoteESP8266\IRutils.cpp.o
======================================================================================== [FAILED] Took 5.42 seconds ========================================================================================
The terminal process terminated with exit code: 1

Terminal will be reused by tasks, press any key to close it.

If problems with PlatformIO Build System:

The content of platformio.ini:

; PlatformIO Project Configuration File
;
;   Build options: build flags, source filter
;   Upload options: custom upload port, speed and extra flags
;   Library options: dependencies, extra library storages
;   Advanced options: extra scripting
;
; Please visit documentation for the other options and examples
; https://docs.platformio.org/page/projectconf.html

[env:nodemcuv2]
platform = espressif8266
board = nodemcuv2
framework = arduino
monitor_speed = 115200

Additional info

On another PC with Python version 3.7 there are no problems compiling.
Here are the data of the PC where there is no problem:

Version: 1.43.0 (user setup)
Commit: 78a4c91400152c0f27ba4d363eb56d2835f9903a
Date: 2020-03-09T19:47:57.235Z
Electron: 7.1.11
Chrome: 78.0.3904.130
Node.js: 12.8.1
V8: 7.8.279.23-electron.0
OS: Windows_NT x64 10.0.17763

PlatformIO: Core 4.2.1, Home 3.1.1

@ivankravets
Copy link
Member

  1. Does both machines use the same user name?
  2. Do you have non-ASCII chars in the user name on a machine that fails?
  3. Do you use the same locale?

@Hofei90
Copy link
Author

Hofei90 commented Mar 13, 2020

1. Does both machines use the same user name?

no the machines have different user names

2. Do you have non-ASCII chars in the user name on a machine that fails?

no, the names contain only characters from [A-Z] [a-z] and char \ from the domain Name. But Domainname is same on both maschines.

3. Do you use the same locale?

Do you mean the character encoding setting like UTF-8? In Visual Studio Code is it UTF-8 or you mean on the OS?

@ivankravets
Copy link
Member

Where did you get this Python 3.8? Could you try to uninstall it and install again the latest Python 3? Also, please remove .platformio/penv folder form user home folder. Restart VSCode.

@Hofei90
Copy link
Author

Hofei90 commented Mar 14, 2020

Hello,
all Python versions were downloaded from the official website. (https://www.python.org/downloads/)

What did I do:

  1. uninstalled Python 3.8 and installed the last release with version 3.8.2 and deleted the penv folder and restarted VS Code
    Result: Problem still exists
  2. platformIO completely deleted from VS Code and reinstalled (Version 4.2.1)
    Result Problem still exists
  3. uninstall Python 3.8.2 and install Python 3.7.3, so that the versions are exactly the same as the working PC. delete the penv folder and restart VS Code.
    Result: Compilation works!

Conclusion:
Python 3.8 causes the error posted above
In Python 3.7 these problems do not yet exist

Translated with www.DeepL.com/Translator (free version)

@ivankravets
Copy link
Member

@Hofei90 could you provide a simple project ot reproduce this issue?

@Hofei90
Copy link
Author

Hofei90 commented Mar 14, 2020

This is the project where the problems were.
Just removed the docs folder, .vscode, .git and .pio.
IR_Sender_WB.zip

@ivankravets ivankravets transferred this issue from platformio/platformio-core Mar 14, 2020
@ivankravets
Copy link
Member

I moved this issue to ESP8266 dev-platform. I tried in my VM and cannot reproduce it.

Screen Shot 2020-03-14 at 12 29 42


@Hofei90 , could you open PlatformIO IDE Terminal and change directory to this project and run:

pio run --jobs 1

Could you provide a new output? It will show where it fails.

@Hofei90
Copy link
Author

Hofei90 commented Mar 14, 2020

I tried in my VM and cannot reproduce it.

Did you use Python 3.8.2 32bit too?
I just installed Python 3.8.2 on the other PC where 3.7 was previously installed and it worked.
After upgrading to Python 3.8.2, compiling no longer works.

Here is the output with
pio run --jobs 1

PS U:\gitea\IR_Sender_WB> pio run --jobs 1
Processing nodemcuv2 (platform: espressif8266; board: nodemcuv2; framework: arduino)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/nodemcuv2.html
PLATFORM: Espressif 8266 2.3.3 > NodeMCU 1.0 (ESP-12E Module)
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES:
 - framework-arduinoespressif8266 3.20603.200130 (2.6.3)
 - tool-esptool 1.413.0 (4.13)
 - tool-esptoolpy 1.20800.0 (2.8.0)
 - toolchain-xtensa 2.40802.191122 (4.8.2)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 30 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <IRremoteESP8266> 2.7.4
Building in release mode
Compiling .pio\build\nodemcuv2\src\main.cpp.o
Exception in thread Thread-3:
Traceback (most recent call last):
  File "C:\Users\***\AppData\Local\Programs\Python\Python38-32\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Users\***\AppData\Local\Programs\Python\Python38-32\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "c:\users\***\.platformio\penv\lib\site-packages\platformio\proc.py", line 46, in run
    self.do_reading()
  File "c:\users\***\.platformio\penv\lib\site-packages\platformio\proc.py", line 69, in do_reading
    for byte in iter(lambda: self._pipe_reader.read(1), ""):
  File "c:\users\***\.platformio\penv\lib\site-packages\platformio\proc.py", line 69, in <lambda>
    for byte in iter(lambda: self._pipe_reader.read(1), ""):
  File "C:\Users\***\AppData\Local\Programs\Python\Python38-32\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 166: character maps to <undefined>
======================================================================================== [FAILED] Took 5.49 seconds ========================================================================================
PS U:\gitea\IR_Sender_WB>

EDIT, add Output with verbose Option:

PS U:\gitea\IR_Sender_WB> pio run --jobs 1 --verbose
Processing nodemcuv2 (platform: espressif8266; board: nodemcuv2; framework: arduino; monitor_speed: 115200)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/nodemcuv2.html
PLATFORM: Espressif 8266 2.3.3 > NodeMCU 1.0 (ESP-12E Module)
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES:
 - framework-arduinoespressif8266 3.20603.200130 (2.6.3)
 - tool-esptool 1.413.0 (4.13)
 - tool-esptoolpy 1.20800.0 (2.8.0)
 - toolchain-xtensa 2.40802.191122 (4.8.2)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 30 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <IRremoteESP8266> 2.7.4 (\\***\***\***\gitea\IR_Sender_WB\lib\IRremoteESP8266)
Building in release mode
xtensa-lx106-elf-g++ -o .pio\build\nodemcuv2\src\main.cpp.o -c -fno-rtti -std=c++11 -Os -mlongcalls -mtext-section-literals -falign-functions=4 -U__STRICT_ANSI__ -ffunction-sections -fdata-sections -fno-exceptions -Wall -DPLATFORMIO=40201 -DESP8266 -DARDUINO_ARCH_ESP8266 -DARDUINO_ESP8266_NODEMCU -DF_CPU=80000000L -D__ets__ -DICACHE_FLASH -DARDUINO=10805 -DARDUINO_BOARD=\"PLATFORMIO_NODEMCUV2\" -DFLASHMODE_DIO -DLWIP_OPEN_SRC -DNONOSDK22x_190703=1 -DTCP_MSS=536 -DLWIP_FEATURES=1 -DLWIP_IPV6=0 -DVTABLES_IN_FLASH -Iinclude -Isrc -Ilib\IRremoteESP8266\src -IC:\Users\***\.platformio\Exception in thread Thread-3:
pTraceback (most recent call last):
a  File "C:\Users\***\AppData\Local\Programs\Python\Python38-32\lib\threading.py", line 932, in _bootstrap_inner
ck    self.run()
a  File "C:\Users\***\AppData\Local\Programs\Python\Python38-32\lib\threading.py", line 870, in run
ges\fra    self._target(*self._args, **self._kwargs)
m  File "c:\users\***\.platformio\penv\lib\site-packages\platformio\proc.py", line 46, in run
ewo    self.do_reading()
r  File "c:\users\***\.platformio\penv\lib\site-packages\platformio\proc.py", line 69, in do_reading
k-    for byte in iter(lambda: self._pipe_reader.read(1), ""):
a  File "c:\users\***\.platformio\penv\lib\site-packages\platformio\proc.py", line 69, in <lambda>
rdui    for byte in iter(lambda: self._pipe_reader.read(1), ""):
n  File "C:\Users\***\AppData\Local\Programs\Python\Python38-32\lib\encodings\cp1252.py", line 23, in decode
oes    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
pUnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 166: character maps to <undefined>
ressif8266\tools\sdk\include -IC:\Users\***\.platformio\packages\framework-arduinoespressif8266\tools\sdk\libc\xtensa-lx106-elf\include -IC:\Users\***\.platformio\packages\framework-arduinoespressif8266\cores\esp8266 -IC:\Users\***\.platformio\packages\framework-arduinoespressif8266\tools\sdk\lwip2\include -IC:\Users\***\.platformio\packages\framework-arduinoespressif8266\variants\nodemcu
src\main.cpp
======================================================================================== [FAILED] Took 5.44 seconds ========================================================================================

@ivankravets
Copy link
Member

ivankravets commented Mar 14, 2020

\****\userdaten

Could you try to move this project to C:\test-project. Just for test.

@Hofei90
Copy link
Author

Hofei90 commented Mar 14, 2020

Good eye, can you please make the path in your post unrecognizable and also remove it from the history

If the folder is on C: it works.
Funny though that under Python 3.7 it also works on the network folder

@ivankravets ivankravets transferred this issue from platformio/platform-espressif8266 Mar 14, 2020
@ivankravets
Copy link
Member

can you please make the path in your post unrecognizable

Sorry, done!


So, I moved back this issue to PIO Core because it looks like a bug. What is a type of this drive? How can I create it?

@Hofei90
Copy link
Author

Hofei90 commented Mar 15, 2020

Hello,
this is a personal network share within the local network. Further details, which hardware and/or software makes this available, I cannot say at the moment. I will try to find out more details

@Hofei90
Copy link
Author

Hofei90 commented Mar 16, 2020

Hello, the network drive is provided by a QNAP NAS. Unfortunately I cannot provide any further information about this

@ivankravets ivankravets changed the title UnicodeDecodeError: 'charmap' codec can't decode byte UnicodeDecodeError on Windows when network drive is used Mar 17, 2020
@draeron
Copy link

draeron commented Mar 20, 2020

i had this issue on windows (local disk). I solved this by changing the following line:

https://github.com/platformio/platformio-core/blob/develop/platformio/proc.py#L33

from :

self._pipe_reader = os.fdopen(self._fd_read)

to :

self._pipe_reader = os.fdopen(self._fd_read, encoding='utf-8')

there are a couple of stackoverflow on this.

@ivankravets ivankravets self-assigned this Mar 21, 2020
@ivankravets ivankravets added this to the 4.3.2 milestone Mar 21, 2020
@ivankravets
Copy link
Member

ivankravets commented Mar 21, 2020

@Hofei90 @draeron could you re-test with this PIO Core dev? Please run pio upgrade --dev

Does it work now?

@draeron
Copy link

draeron commented Mar 21, 2020

@ivankravets look like it solved the issue i had

@ivankravets
Copy link
Member

@draeron thanks!

@Hofei90 could you confirm?

@ivankravets
Copy link
Member

I merged into the development branch. Let see what people report.

@lmagyar
Copy link

lmagyar commented Jan 17, 2022

Commit 4687665 caused another issue: #4155

@richms
Copy link

richms commented Apr 1, 2022

Neither fix worked for me. The first one made it end up making the path within the mapped network drive, Seems it thinks that the space in the path that is mapped to the drive letter is a seperator because everything after that was created in the projects folder and had some things put into it as it was failing.

the second one seemed to have all my work undone by downloading python over what I replaced it with when I pressed build.

Begrudgingly moved my project onto the local drive just so I could build it. This is a showstopper for the use of platformio for me as I rely on my files being on a nas so they are accessible on all computers

@MarcusWolschon
Copy link

MarcusWolschon commented Apr 1, 2022

If it has problems with spaces in pathes, then clearly some parts of PlatformIO use inproper escaping for pathes.
An issue that must always be found and fixed.

@kbfifi
Copy link

kbfifi commented Jul 25, 2022

I think this is a wonderful project with lots of potential, but as long there's no fix for this issue I won't use it.

@ivankravets
Copy link
Member

Hi there,

Sorry for the historical issue. It looks like we have finally fixed it. Please re-test with pio upgrade --dev.

Does it work now without any issues?

P.S.: Any Python 3.6 or above is OK now!

@MarcusWolschon
Copy link

MarcusWolschon commented Apr 20, 2023

You have already closed it. So apparently testing is already complete.

@ivankravets
Copy link
Member

Yes, it was tested internally and published to the develop branch. Please re-test on your machine after upgrading PIO Core to the latest development version using pio upgrade --dev command.

@MarcusWolschon
Copy link

MarcusWolschon commented Apr 20, 2023

Doesn't work in the "Terminal" tab in PlatformIO/Visual Studio Code.
The "Debugging Console" tab doesn't accept input too.
Also this is a production environment. I don't actually want a development version of the IDE to keep maintaining an old production codebase that must compile and upload at any time.
(I started that projet using the old PlatformIO IDE when it outgrew ArduinoIDE. Later VS-Code was forced upon me when PlatformIO moved.)

PlatformIO has automatically updated already on start but doesn't tell me the new version number or a changelog.
Manually opening the "PIO Home" tab it shows core=6.1.6 and Home=3.4.3
with no obvious way to switch between dev and stable releases.

pio : Die Benennung "pio" wurde nicht als Name eines Cmdlet, einer Funktion, einer Skriptdatei oder eines ausführbaren Programms erkannt. Überprüfen Sie die Schreibweise des Namens, oder ob der Pfad korrekt ist (sofern enthalten), und wiederholen Sie den Vorgang.
In Zeile:1 Zeichen:1
+ pio upgrade --dev
+ ~~~
    + CategoryInfo          : ObjectNotFound: (pio:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

@MarcusWolschon
Copy link

MarcusWolschon commented Apr 20, 2023

Found it
grafik
grafik
grafik

I must say that is VERY well hidden.
But it doesn't offer any updates after switching to the dev-version and Help->Check For Updates doesn't find any too.

@ivankravets
Copy link
Member

You can open PlatformIO Core CLI in VSCode and type

pio upgrade --dev

Restart VSCode.

@MarcusWolschon
Copy link

MarcusWolschon commented Apr 20, 2023

Using that thin, blue toolbar at the bottom I can compile a copy of the project stored on a mapped network drive from a path that contains spaces and a period.

Looks like changing the checkbox "[X] use a development version" forgot to tell me that it needs me to exit and restart VS-code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests