time.strftime() and Unicode characters on Windows

[AndiDogold]

BPO	8304
Nosy	@terryjreedy, @pfmoore, @abalkin, @vstinner, @ericvsmith, @tjguk, @ezio-melotti, @shimizukawa, @zware, @eryksun, @zooba

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2010-04-03.15:08:42.673>
labels = ['3.8', '3.9', 'extension-modules', 'expert-unicode', 'type-bug', '3.10', 'library', 'OS-windows']
title = 'time.strftime() and Unicode characters on Windows'
updated_at = <Date 2021-03-08.19:17:29.084>
user = 'https://bugs.python.org/AndiDogold'

bugs.python.org fields:

activity = <Date 2021-03-08.19:17:29.084>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules', 'Library (Lib)', 'Unicode', 'Windows']
creation = <Date 2010-04-03.15:08:42.673>
creator = 'AndiDog_old'
dependencies = []
files = []
hgrepos = []
issue_num = 8304
keywords = []
message_count = 16.0
messages = ['102269', '102298', '102310', '102332', '102335', '159341', '222667', '226114', '251554', '251558', '251560', '255043', '255133', '388241', '388277', '388286']
nosy_count = 12.0
nosy_names = ['terry.reedy', 'paul.moore', 'belopolsky', 'vstinner', 'eric.smith', 'tim.golden', 'ezio.melotti', 'AndiDog_old', 'shimizukawa', 'zach.ware', 'eryksun', 'steve.dower']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue8304'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

Linked PRs

• gh-52551: Fix encoding issues in strftime() #125193
• [3.13] gh-52551: Fix encoding issues in strftime() (GH-125193) #125657
• gh-52551: Use wcsftime() to implement time.strftime() on Windows #125658
• [3.12] [3.13] gh-52551: Fix encoding issues in strftime() (GH-125193) (GH-125657) #125661

[AndiDogold]

There is inconsistent behavior in time.strftime, comparing Python 2.6 and 3.1. In 3.1, non-ASCII Unicode characters seem to get dropped whereas in 2.6 you can keep them using the necessary Unicode-to-UTF8 workaround.

This should be fixed if it isn't intended behavior.

Python 2.6

>>> time.strftime(u"%d\u200F%A".encode("utf-8"), time.gmtime()).decode("utf-8")
u'03\u200fSaturday'
>>> time.strftime(u"%d\u0041%A".encode("utf-8"), time.gmtime()).decode("utf-8")
u'03ASaturday'

Python 3.1

>>> time.strftime("%d\u200F%A", time.gmtime())
''
>>> len(time.strftime("%d\u200F%A", time.gmtime()))
0
>>> time.strftime("%d\u0041%A", time.gmtime())
'03ASaturday'

[ezio-melotti]

This seems to be fixed now, on both 3.1 and 3.2.
Can you try with 3.1.2 and see if it works?
What operating system are you using?

[ezio-melotti]

Actually the bug seems related to Windows.

[AndiDogold]

Just installed Python 3.1.2, same problem. I'm using Windows XP SP2 with two Python installations (2.6.4 and now 3.1.2).

[AndiDogold]

Definitely a Windows problem. I did this on Visual Studio 2008:

wchar_t out[1000];
time_t currentTime;
time(&currentTime);
tm *timeStruct = gmtime(&currentTime);

    size_t ret = wcsftime(out, 1000, L"%d%A", timeStruct);
    wprintf(L"ret = %d, out = (%s)\n", ret, out);

    ret = wcsftime(out, 1000, L"%d\u200f%A", timeStruct);
    wprintf(L"ret = %d, out = (%s)\n", ret, out);

and the output was

    ret = 8, out = (04Sunday)
    ret = 0, out = ()

Python really shouldn't use any so-called standard functions on Windows. They never work as expected ^^...

[vstinner]

Actually the bug seems related to Windows.

See also the issue bpo-10653: wcsftime() doesn't format correctly time zones, so Python 3 uses strftime() instead.

[BreamoreBoy]

Using 3.4.1 and 3.5.0 I get:-

time.strftime("%d\u200F%A", time.gmtime())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'locale' codec can't encode character '\u200f' in position 2: Illegal byte sequence

[terryjreedy]

I verified Marks 3.4.1 result with Idle.

It strikes me as a bug that a function that maps a unicode format string to a unicode string with interpolations added should ever encode the format to bytes, lets alone using using an encoding that fails or loses information. It is especially weird given that % formatting does not even work (at present) for bytes.

It seems to me that strftime should never encode the non-special parts of the format text. Instead, it could split the format (re.split) into a list of alternatine '%x' pairs and running text segments, replace the '%x' entries with the proper entries, and return the list joined back into a string. Some replacements would be locale dependent, other not.

(Just wondering, are the locate names of days and months bytes restricted to ascii or unrestricted unicode using native characters?)

[BreamoreBoy]

@alexander what is you take on this please? I can confirm that it is still a problem on Windows in 3.5.0.

[abalkin]

Mark, I am no expert on Windows. I believe Victor is most knowledgable in this area.