Skip to content

Update to scintilla 5.5.5 & Lexilla 5.4.3#16235

Closed
chcg wants to merge 1 commit intonotepad-plus-plus:masterfrom
chcg:scintilla_555
Closed

Update to scintilla 5.5.5 & Lexilla 5.4.3#16235
chcg wants to merge 1 commit intonotepad-plus-plus:masterfrom
chcg:scintilla_555

Conversation

@chcg
Copy link
Contributor

@chcg chcg commented Feb 27, 2025

Release 5.5.5 (https://www.scintilla.org/scintilla555.zip)

Released 25 February 2025.
Remember selection with undo and redo. Controlled with SCI_SETUNDOSELECTIONHISTORY. Feature #1273, Bug #1479, Bug #1224.
Serialize selection type and ranges with SCI_GETSELECTIONSERIALIZED and SCI_SETSELECTIONSERIALIZED.
For Win32, update Direct2D and DirectWrite interfaces used to 1.1 and add a lower-level approach to calling DirectWrite 1.1 by specifying SC_TECHNOLOGY_DIRECT_WRITE_1. Since Windows Vista does not support these API versions, Scintilla o longer supports DirectWrite on Windows Vista and will fall back to using GDI.
Fix segmentation of long lexemes to avoid breaking before modifiers like accents that must be drawn with their base letters. For wrapping, try to break lines without separating letters from modifiers.
For GTK on Windows, replace reverse arrow cursor with hand as reverse arrow was small in scaled modes. Bug #2460.
Fix bug on Qt where double-click stopped working when Scintilla instance had been running for weeks.

Release 5.4.3 (https://www.scintilla.org/lexilla543.zip)

Released 25 February 2025.
C++: Fix evaluation of != in preprocessor condition. Issue #299.
Modula-3: Allow digits in uppercase identifiers. Issue #297.
Pascal: Fix asm style extending past end. Issue #295.
Python: Fix detection of attributes and decorators. Issue #294, Pull request #302.
Ruby: Implement substyles for identifiers SCE_RB_IDENTIFIER.
Ruby: Recognize name as SCE_RB_DEFNAME in def when `::` used as well as `.`. Issue #300.

Release 5.5.5 (https://www.scintilla.org/scintilla555.zip)

    Released 25 February 2025.
    Remember selection with undo and redo. Controlled with SCI_SETUNDOSELECTIONHISTORY. Feature notepad-plus-plus#1273, Bug notepad-plus-plus#1479, Bug notepad-plus-plus#1224.
    Serialize selection type and ranges with SCI_GETSELECTIONSERIALIZED and SCI_SETSELECTIONSERIALIZED.
    For Win32, update Direct2D and DirectWrite interfaces used to 1.1 and add a lower-level approach to calling DirectWrite 1.1 by specifying SC_TECHNOLOGY_DIRECT_WRITE_1. Since Windows Vista does not support these API versions, Scintilla o longer supports DirectWrite on Windows Vista and will fall back to using GDI.
    Fix segmentation of long lexemes to avoid breaking before modifiers like accents that must be drawn with their base letters. For wrapping, try to break lines without separating letters from modifiers.
    For GTK on Windows, replace reverse arrow cursor with hand as reverse arrow was small in scaled modes. Bug notepad-plus-plus#2460.
    Fix bug on Qt where double-click stopped working when Scintilla instance had been running for weeks.

Release 5.4.3 (https://www.scintilla.org/lexilla543.zip)

    Released 25 February 2025.
    C++: Fix evaluation of != in preprocessor condition. Issue notepad-plus-plus#299.
    Modula-3: Allow digits in uppercase identifiers. Issue notepad-plus-plus#297.
    Pascal: Fix asm style extending past end. Issue notepad-plus-plus#295.
    Python: Fix detection of attributes and decorators. Issue notepad-plus-plus#294, Pull request notepad-plus-plus#302.
    Ruby: Implement substyles for identifiers SCE_RB_IDENTIFIER.
    Ruby: Recognize name as SCE_RB_DEFNAME in def when `::` used as well as `.`. Issue notepad-plus-plus#300.
@chcg chcg added scintilla dependent Can't be considered for N++ implementation unless/until Scintilla changes lexilla dependent Can't be considered for N++ implementation unless/until Lexiilla changes labels Feb 27, 2025
@chcg
Copy link
Contributor Author

chcg commented Feb 27, 2025

Unfortunately I have Platform asserts with the debug build from SurfaceD2D::MeasureWidths(), not sure if this is new to this PR. File is a bmp (https://github.com/chcg/NPP_HexEdit/blob/master/HexEditor/res/hex.bmp) With the release build there seems to be no problem arising from this.

 	KernelBase.dll!00007ff97dc81d72()	Unbekannt
 	notepad++.exe!Scintilla::Internal::Platform::Assert(const char * c, const char * file, int line) Zeile 4175	C++
>	notepad++.exe!Scintilla::Internal::SurfaceD2D::MeasureWidths(const Scintilla::Internal::Font * font_, std::basic_string_view<char,std::char_traits<char>> text, double * positions) Zeile 2571	C++
 	notepad++.exe!PositionCache::MeasureWidths(Scintilla::Internal::Surface * surface, const Scintilla::Internal::ViewStyle & vstyle, unsigned int styleNumber, bool unicode, std::basic_string_view<char,std::char_traits<char>> sv, double * positions, bool needsLocking) Zeile 1171	C++
 	notepad++.exe!`anonymous namespace'::LayoutSegments(Scintilla::Internal::IPositionCache * pCache, Scintilla::Internal::Surface * surface, const Scintilla::Internal::ViewStyle & vstyle, Scintilla::Internal::LineLayout * ll, const std::vector<Scintilla::Internal::TextSegment,std::allocator<Scintilla::Internal::TextSegment>> & segments, std::atomic<unsigned int> & nextIndex, const bool textUnicode, const bool multiThreaded) Zeile 376	C++
 	notepad++.exe!Scintilla::Internal::EditView::LayoutLine::__l51::<Lambdafunktion>() Zeile 513	C++
 	[Externer Code]	
 	notepad++.exe!Scintilla::Internal::EditView::LayoutLine(const Scintilla::Internal::EditModel & model, Scintilla::Internal::Surface * surface, const Scintilla::Internal::ViewStyle & vstyle, Scintilla::Internal::LineLayout * ll, int width, bool callerMultiThreaded) Zeile 518	C++
 	notepad++.exe!Scintilla::Internal::EditView::SPositionFromLocation(Scintilla::Internal::Surface * surface, const Scintilla::Internal::EditModel & model, Scintilla::Internal::PointDocument pt, bool canReturnInvalid, bool charPosition, bool virtualSpace, const Scintilla::Internal::ViewStyle & vs, const Scintilla::Internal::PRectangle rcClient) Zeile 720	C++
 	notepad++.exe!Scintilla::Internal::Editor::SPositionFromLocation(Scintilla::Internal::Point pt, bool canReturnInvalid, bool charPosition, bool virtualSpace) Zeile 380	C++
 	notepad++.exe!Scintilla::Internal::Editor::PositionFromLocation(Scintilla::Internal::Point pt, bool canReturnInvalid, bool charPosition) Zeile 385	C++
 	notepad++.exe!Scintilla::Internal::Editor::WndProc(Scintilla::Message iMessage, unsigned __int64 wParam, __int64 lParam) Zeile 6985	C++
 	notepad++.exe!Scintilla::Internal::ScintillaBase::WndProc(Scintilla::Message iMessage, unsigned __int64 wParam, __int64 lParam) Zeile 1142	C++
 	notepad++.exe!Scintilla::Internal::ScintillaWin::WndProc(Scintilla::Message iMessage, unsigned __int64 wParam, __int64 lParam) Zeile 2540	C++
 	notepad++.exe!Scintilla::Internal::ScintillaWin::DirectFunction(__int64 ptr, unsigned int iMessage, unsigned __int64 wParam, __int64 lParam) Zeile 4075	C++
 	notepad++.exe!ScintillaEditView::execute(unsigned int Msg, unsigned __int64 wParam, __int64 lParam) Zeile 449	C++
 	notepad++.exe!ScintillaEditView::getVisibleStartAndEndPosition(__int64 * startPos, __int64 * endPos) Zeile 2770	C++
 	notepad++.exe!Notepad_plus::addHotSpot(ScintillaEditView * view) Zeile 3476	C++
 	notepad++.exe!Notepad_plus::notify(SCNotification * notification) Zeile 511	C++
 	notepad++.exe!Notepad_plus::process(HWND__ * hwnd, unsigned int message, unsigned __int64 wParam, __int64 lParam) Zeile 2165	C++
 	notepad++.exe!Notepad_plus_Window::runProc(HWND__ * hwnd, unsigned int message, unsigned __int64 wParam, __int64 lParam) Zeile 244	C++
 	notepad++.exe!Notepad_plus_Window::Notepad_plus_Proc(HWND__ * hwnd, unsigned int message, unsigned __int64 wParam, __int64 lParam) Zeile 202	C++
 	[Externer Code]	
 	notepad++.exe!Scintilla::Internal::ScintillaWin::NotifyParent(Scintilla::NotificationData scn) Zeile 2839	C++
 	notepad++.exe!Scintilla::Internal::Editor::NotifyPainted() Zeile 2609	C++
 	notepad++.exe!Scintilla::Internal::Editor::Paint(Scintilla::Internal::Surface * surfaceWindow, Scintilla::Internal::PRectangle rcArea) Zeile 1956	C++
 	notepad++.exe!Scintilla::Internal::ScintillaWin::PaintDC(HDC__ * hdc) Zeile 1278	C++
 	notepad++.exe!Scintilla::Internal::ScintillaWin::WndPaint() Zeile 1317	C++
 	notepad++.exe!Scintilla::Internal::ScintillaWin::WndProc(Scintilla::Message iMessage, unsigned __int64 wParam, __int64 lParam) Zeile 2344	C++
 	notepad++.exe!Scintilla::Internal::ScintillaWin::SWndProc(HWND__ * hWnd, unsigned int iMessage, unsigned __int64 wParam, __int64 lParam) Zeile 4125	C++
 	[Externer Code]	
 	notepad++.exe!ScintillaEditView::scintillaNew_Proc(HWND__ * hwnd, unsigned int Message, unsigned __int64 wParam, __int64 lParam) Zeile 702	C++
 	notepad++.exe!ScintillaEditView::scintillaStatic_Proc(HWND__ * hwnd, unsigned int Message, unsigned __int64 wParam, __int64 lParam) Zeile 391	C++
 	[Externer Code]	
 	notepad++.exe!Scintilla::Internal::Platform::Assert(const char * c, const char * file, int line) Zeile 4163	C++
 	notepad++.exe!Scintilla::Internal::SurfaceD2D::MeasureWidths(const Scintilla::Internal::Font * font_, std::basic_string_view<char,std::char_traits<char>> text, double * positions) Zeile 2571	C++
 	notepad++.exe!PositionCache::MeasureWidths(Scintilla::Internal::Surface * surface, const Scintilla::Internal::ViewStyle & vstyle, unsigned int styleNumber, bool unicode, std::basic_string_view<char,std::char_traits<char>> sv, double * positions, bool needsLocking) Zeile 1171	C++
 	notepad++.exe!`anonymous namespace'::LayoutSegments(Scintilla::Internal::IPositionCache * pCache, Scintilla::Internal::Surface * surface, const Scintilla::Internal::ViewStyle & vstyle, Scintilla::Internal::LineLayout * ll, const std::vector<Scintilla::Internal::TextSegment,std::allocator<Scintilla::Internal::TextSegment>> & segments, std::atomic<unsigned int> & nextIndex, const bool textUnicode, const bool multiThreaded) Zeile 376	C++
 	notepad++.exe!Scintilla::Internal::EditView::LayoutLine::__l51::<Lambdafunktion>() Zeile 513	C++
 	[Externer Code]	
 	notepad++.exe!Scintilla::Internal::EditView::LayoutLine(const Scintilla::Internal::EditModel & model, Scintilla::Internal::Surface * surface, const Scintilla::Internal::ViewStyle & vstyle, Scintilla::Internal::LineLayout * ll, int width, bool callerMultiThreaded) Zeile 518	C++
 	notepad++.exe!Scintilla::Internal::EditView::LocationFromPosition(Scintilla::Internal::Surface * surface, const Scintilla::Internal::EditModel & model, Scintilla::Internal::SelectionPosition pos, __int64 topLine, const Scintilla::Internal::ViewStyle & vs, Scintilla::Internal::PointEnd pe, const Scintilla::Internal::PRectangle rcClient) Zeile 649	C++
 	notepad++.exe!Scintilla::Internal::Editor::LocationFromPosition(Scintilla::Internal::SelectionPosition pos, Scintilla::Internal::PointEnd pe) Zeile 350	C++
 	notepad++.exe!Scintilla::Internal::Editor::PointMainCaret() Zeile 926	C++
 	notepad++.exe!Scintilla::Internal::Editor::SetLastXChosen() Zeile 935	C++
 	notepad++.exe!Scintilla::Internal::Editor::WndProc(Scintilla::Message iMessage, unsigned __int64 wParam, __int64 lParam) Zeile 6581	C++
 	notepad++.exe!Scintilla::Internal::ScintillaBase::WndProc(Scintilla::Message iMessage, unsigned __int64 wParam, __int64 lParam) Zeile 1142	C++
 	notepad++.exe!Scintilla::Internal::ScintillaWin::WndProc(Scintilla::Message iMessage, unsigned __int64 wParam, __int64 lParam) Zeile 2540	C++
 	notepad++.exe!Scintilla::Internal::ScintillaWin::DirectFunction(__int64 ptr, unsigned int iMessage, unsigned __int64 wParam, __int64 lParam) Zeile 4075	C++
 	notepad++.exe!ScintillaEditView::execute(unsigned int Msg, unsigned __int64 wParam, __int64 lParam) Zeile 449	C++
 	notepad++.exe!ScintillaEditView::restoreCurrentPosPreStep() Zeile 2233	C++
 	notepad++.exe!ScintillaEditView::activateBuffer(Buffer * buffer, bool force) Zeile 2402	C++
 	notepad++.exe!Notepad_plus::activateBuffer(Buffer * id, int whichOne, bool forceApplyHilite) Zeile 5038	C++
 	notepad++.exe!Notepad_plus::loadBufferIntoView(Buffer * id, int whichOne, bool dontClose) Zeile 4691	C++
 	notepad++.exe!Notepad_plus::doOpen(const std::wstring & fileName, bool isRecursive, bool isReadOnly, int encoding, const wchar_t * backupFileName, _FILETIME fileNameTimestamp) Zeile 457	C++
 	notepad++.exe!Notepad_plus::loadSession(Session & session, bool isSnapshotMode, const wchar_t * userCreatedSessionName) Zeile 2402	C++
 	notepad++.exe!Notepad_plus::loadLastSession() Zeile 2352	C++
 	notepad++.exe!Notepad_plus_Window::init(HINSTANCE__ * hInst, HWND__ * parent, const wchar_t * cmdLine, CmdLineParams * cmdLineParams) Zeile 172	C++
 	notepad++.exe!wWinMain(HINSTANCE__ * hInstance, HINSTANCE__ * __formal, wchar_t * pCmdLine, int __formal) Zeile 712	C++
 	[Externer Code]	
Notepad++ v8.7.7   (64-bit)
Build time : Feb 27 2025 - 18:54:02
Path : ...\notepad++.exe
Command Line : 
Admin mode : OFF
Local Conf mode : ON
Cloud Config : OFF
Periodic Backup : ON
Placeholders : OFF
Scintilla Rendering Mode : SC_TECHNOLOGY_DIRECTWRITE (1)
Multi-instance Mode : monoInst
File Status Auto-Detection : cdEnabledNew (for current file/tab only) + cdAutoUpdate
Dark Mode : OFF
OS Name : Windows 10 Home (64-bit)
OS Version : 22H2
OS Build : 19045.5487
Current ANSI codepage : 65001
Plugins : none

@donho donho self-assigned this Mar 11, 2025
@donho
Copy link
Member

donho commented Mar 11, 2025

@chcg

Unfortunately I have Platform asserts with the debug build from SurfaceD2D::MeasureWidths(), not sure if this is new to this PR.

Could you provide me the instructions to reproduce the asserts please?

Copy link
Member

@donho donho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were 2 headers added by accident?

@donho
Copy link
Member

donho commented Mar 11, 2025

Unfortunately I have Platform asserts with the debug build from SurfaceD2D::MeasureWidths(), not sure if this is new to this PR.

Could you provide me the instructions to reproduce the asserts please?

@chcg
Any description is fine for me.

@chcg
Copy link
Contributor Author

chcg commented Mar 12, 2025

Happens to me also with x64 debug build of 8.7.8 from https://github.com/notepad-plus-plus/notepad-plus-plus/actions/runs/13733461955/artifacts/2714893369. It is just necessary to open the bmp file mentioned above.

@donho
Copy link
Member

donho commented Mar 12, 2025

@chcg

It is just necessary to open the bmp file mentioned above.

I cannot reproduce with the local build from my machine:

image

Edit: cannot reproduce also with the binary from the link you provided above

@nyamatongwe
Copy link

I can't reproduce the assertion failure either.

The top half of the stack trace, above the second Platform::Assert, isn't interesting as it is just re-failing when trying to draw while the assertion dialog is shown. For drawing problems, assertionPopUps in PlatWin.cxx is best set to false. Instead, place a breakpoint on the third line of Platform::Assert (or just look in debug output) to see what went wrong.

There are 3 assertions that may be triggered in SurfaceD2D::MeasureWidths.

  1. In the middle of SurfaceD2D::MeasureWidths: assert that, for an 8-bit encoding, there are the same number of UTF-16 code-points - failure fairly impossible.
  2. In MeasurePositions: that all of the text was measured.
  3. In FontDirectWrite::Cast: when casting (with dynamic_cast) the input Font argument into a FontDirectWrite which could fail if the font was a FontGDI which is fairly unlikely. Failures appear differently if RTTI (needed for dynamic_cast) is turned off or the input font is null. Maybe there is a stale font for some reason.

@donho
Copy link
Member

donho commented Mar 12, 2025

@nyamatongwe

Failures appear differently if RTTI (needed for dynamic_cast) is turned off or the input font is null. Maybe there is a stale font for some reason.

From the Debug Info, the OS of @chcg is Windows 10 Home, so what you have said makes sense.

@donho donho added the accepted label Mar 12, 2025
@donho donho closed this in e38a0f2 Mar 12, 2025
@chcg chcg deleted the scintilla_555 branch March 13, 2025 16:16
@nyamatongwe
Copy link

I've worked this out. The clue is from the debug info: Current ANSI codepage : 65001. That means that @chcg has set the ANSI code page to UTF-8 system-wide. Setting the 'Beta: Use Unicode UTF-8...' option in Region Settings causes the bug to occur for me in Notepad++ but not in SciTE.

With the ANSI UTF-8 setting, SurfaceD2D::MeasureWidths converts from UTF-8 to wchar_t resulting in a different number of wchar_t values than the input bytes. The actual input that fails is \xf0\xca\xa6 which is invalid UTF-8 (\xf0 is a lead byte that requires 3 trail bytes but there are only 2).

This doesn't occur in normal Scintilla since the bytes are treated as common European Latin code page 1252. However, Notepad++ has patched CodePageFromCharSet in a way that allows this.

	// Cyrillic / Turkish or other languages cannot be shown in ANSI mode.
	// This fixes such problem. For more information about this fix, check:
	// https://github.com/notepad-plus-plus/notepad-plus-plus/issues/5671
	//	case CharacterSet::Default: return documentCodePage ? documentCodePage : 1252;
	case CharacterSet::Default: return documentCodePage;

If the text is to be displayed as UTF-8, then the whole of Scintilla needs to know about that rather than just some platform layer functions. That way \xf0\xca\xa6 can be treated as invalid and displayed with byte blobs.

While it might be possible to have Scintilla detect this situation, Notepad++ should have a strategy for what it wants ANSI UTF-8 to mean. It could detect that GetACP() == CP_UTF8 and set the Scintilla code page to SC_CP_UTF8 but there may be further consequences from that.

The above-cited patch is troubling as it means that Scintilla may not correctly perform character set dependent operations like case-insensitive search when it is used to enable Cyrillic or Turkish. It would be better for Notepad++ to set an explicit character set like SC_CHARSET_TURKISH.

@chcg
Copy link
Contributor Author

chcg commented Mar 13, 2025

@nyamatongwe Thanks for digging into this. In the meantime I updated to Win11 and this issue still happens. As stated I changed the default code page, see e.g. : https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page

I think I had no specific reason for that. Maybe some testing for python stuff. The visual effect is that textboxes are stretched:

grafik

What I find still strange is that release builds seems to have no issue with that.

@chcg
Copy link
Contributor Author

chcg commented Mar 14, 2025

@donho At least some of the issues show Current ANSI codepage : 65001 , see https://github.com/search?q=repo%3Anotepad-plus-plus%2Fnotepad-plus-plus+Current+ANSI+codepage+%3A+65001&type=issues
So maybe this finding needs to be taken into account on ticket analysis.

@nyamatongwe
Copy link

The visual effect is that textboxes are stretched:

I haven't seen that and don't have a theory for it.

The length mismatch may lead to reading out-of-bounds uninitialized memory which can contain random values. However, the image appears systematic like its adding the width of the previous segment to the width of the current segment. Its unlikely to be worth chasing down as effort would be better applied to avoiding the situation.

@MarkusBodensee
Copy link
Contributor

@chcg
Since you pointed to the adressed issues which are fixed by this Scintilla/Lexilla update. Can you close those issues?

@chcg
Copy link
Contributor Author

chcg commented Mar 29, 2025

@MarkusBodensee There is no N++ release available which contains this update, so it can't be tested yet from normal users.

@MarkusBodensee
Copy link
Contributor

@chcg Thank you for your explanation :) I just asked because most of the issues are closed when the pull request is merged. But waiting for user feedback after release is valid and good point.

@Coises
Copy link
Contributor

Coises commented Oct 26, 2025

@nyamatongwe :

The above-cited patch is troubling as it means that Scintilla may not correctly perform character set dependent operations like case-insensitive search when it is used to enable Cyrillic or Turkish. It would be better for Notepad++ to set an explicit character set like SC_CHARSET_TURKISH.

I understand that the specific problem in this thread is due to the system default code page being CP_UTF8 (something which was impossible when most of the relevant Notepad++ code was written). Don is working on it.

As best I can tell, before the change in Scintilla which this patch reverts, a file encoded according to the system default code page, so long as that code page was a single byte character set, would display correctly in Scintilla by sending SCI_SETCODEPAGE(0).

It appears this change in Scintilla broke that behavior on systems where Windows-1252 is not the system default code page.

Scintilla documentation says single byte character sets should use SCI_SETCODEPAGE(0) and directs us to SCI_STYLESETCHARACTERSET. However, the suggested message sets individual styles, not the default for the whole document. Further, the relationship between character set values used in that message and the system default code page value from GetACP is unclear.

What is the correct code to achieve the desired behavior (that is, for users with system default code pages that represent a single byte character set, no matter what that code page is, editing files in their system’s native encoding “just works”) without patching Scintilla?

@nyamatongwe
Copy link

What is the correct code to achieve the desired behavior (that is, for users with system default code pages that represent a single byte character set, no matter what that code page is, editing files in their system’s native encoding “just works”)

A reasonable generic approach is to set Scintilla to UTF-8 and convert from/to the code page to load/save.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

accepted lexilla dependent Can't be considered for N++ implementation unless/until Lexiilla changes scintilla dependent Can't be considered for N++ implementation unless/until Scintilla changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants