tree 71c39e08679c7dc5c530c6f31d7780401a99c0ab
parent e2704cba8119851a6b0d80ac94ace8d306efba45
author Nico Weber <thakis@chromium.org> 1701309296 +0000
committer Pdfium LUCI CQ <pdfium-scoped@luci-project-accounts.iam.gserviceaccount.com> 1701309296 +0000

Make WideString's FromUTF16BE / FromUTF16LE do surrogate fusing

...and use them in PDF_DecodeText().

No behavior change for PDF_DecodeText().

Otherwise, these functions are mostly used on Windows (where this
is a no-op since wchar_t is 2 bytes there) and in fuzzers, so this
generally shouldn't have a big effect.

This is a bit weird since WideString::ToUTF16LE() doesn't create
surrogates yet, so things don't round-trip cleanly through
FromUTF16LE().ToUTF16LE(). But that's not a new problem since
PDF_DecodeText() created fused surrogates before this as well,
and FX_UTF8Decode() also creates WideStrings with characters outside
the BMP. (ToUTF16LE() adds a zero-terminator, which is also a problem
round-trip wise.)

It's also strange that fx_string.h contains UTF8 conversion methods,
in addition to WideString::FromUTF8(). They operate on WideStringViews.
FX_UTF8Decode() should arguably be replaced by WideString::FromUTF8()
everywhere, but FX_UTF8Encode() takes a WideStringView -- we probably
want to create FX_UTF16LEEncode() and make WideString::ToUTF16LE() call
that possibly. Or maybe FX_UTF8Encode() should become a method on
ByteString -- ByteString::UTF8FromWide() or something.

But that's all for other CLs. This here seems like a small
self-contained progression.

Change-Id: I13afb5f66df6f553358cc03ccfd56a78e8e624c3
Reviewed-on: https://pdfium-review.googlesource.com/c/pdfium/+/114030
Commit-Queue: Nico Weber <thakis@chromium.org>
Reviewed-by: Lei Zhang <thestig@chromium.org>
Auto-Submit: Nico Weber <thakis@chromium.org>
Commit-Queue: Lei Zhang <thestig@chromium.org>