XWin crashes with DirectX apps

Wed Sep 29 17:21:00 GMT 2010

On 29/09/2010 17:49, Kevin Goodsell wrote:
> I am seeing a 100% reproducible crash in XWin from xorg-server-1.8.2-1.
> I first saw it when I ran a full-screen DirectX game while the server
> was running, but I'm also able to reproduce it with the dxdiag tool.
> Here are the steps:
>
> 1) Start the X server. The exact method of starting it hasn't had any
> affect on my testing. Starting either with the installed Start menu
> shortcut or just running XWin.exe without arguments produces the same
> results.
>
> 2) In Start -> Run..., enter dxdiag to run the DirectX Diagnostic tool.
>
> 3) In dxdiag, select the Display tab.
>
> 4) Click the button labeled "Test Direct3D" (the button labeled "Test
> DirectDraw" also triggers the crash, but takes longer).
>
> XWin crashes when the first full-screen test begins.
>
> I rebuilt the X server with debugging enabled and got the following
> backtrace:

Thanks for the detailed problem report.

Since I've recently fixed a few crash bugs in this area, introduced with the 
new -resize functionality, you might like to test the latest snapshot [1] to 
see if you still have this problem, the source is available at [2].

[1] ftp://cygwin.com/pub/cygwinx/XWin.20100923-git-2172af4d1ea713f1.exe.bz2
[2] http://cgit.freedesktop.org/~jturney/xserver/log/?h=snapshot

> $ gdb hw/xwin/XWin.exe
> GNU gdb 6.8.0.20080328-cvs (cygwin-special)
> Copyright (C) 2008 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "i686-pc-cygwin"...
> (gdb) run
> Starting program: /usr/src/xorg-server-1.8.2-1/build/xwin-ddx/hw/xwin/XWin.exe
> [New thread 3084.0xc20]
> [New thread 3084.0xda8]
> warning: Lowest section in /cygdrive/c/WINDOWS/system32/wmi.dll is .text at
> 76d31000
> [New thread 3084.0xa68]
> [New thread 3084.0xd38]
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x00415677 in winFreeFBShadowDDNL (pScreen=0x1008c1d0)
> at
> /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/hw/xwin/winshadddnl.c:560
> 560 IDirectDrawSurface4_SetClipper (pScreenPriv->pddsPrimary4,
> (gdb) bt
> #0 0x00415677 in winFreeFBShadowDDNL (pScreen=0x1008c1d0)
> at
> /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/hw/xwin/winshadddnl.c:560
> #1 0x0042f917 in winDoRandRScreenSetSize (pScreen=0x1008c1d0, width=640,
> height=480,
> mmWidth=3435973836, mmHeight=1080233164)
> at /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/hw/xwin/winrandr.c:191
> #2 0x0041a369 in winWindowProc (hwnd=0x1c023c, message=126, wParam=16,
> lParam=31457920)
> at
> /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/hw/xwin/winwndproc.c:344
> #3 0x7e418734 in USER32!GetDC () from /cygdrive/c/WINDOWS/system32/user32.dll
> #4 0x001c023c in ?? ()
> #5 0x0000007e in ?? ()
> #6 0x00000010 in ?? ()
> #7 0x01e00280 in ?? ()
> #8 0x00419bcc in winReshapeRootless ()
> #9 0x7e418816 in USER32!GetDC () from /cygdrive/c/WINDOWS/system32/user32.dll
> #10 0x00419bcc in winReshapeRootless ()
> #11 0x7e428ea0 in USER32!DefWindowProcW () from
> /cygdrive/c/WINDOWS/system32/user32.dll
> #12 0x00000000 in ?? ()
> (gdb) p pScreenPriv->pddsPrimary4
> $1 = (LPDIRECTDRAWSURFACE4) 0x0
> (gdb)
>
> Using the option -engine 2 also crashes, but produces a slightly
> different backtrace:
>
> (gdb) run -engine 2
> The program being debugged has been started already.
> Start it from the beginning? (y or n) y
>
> Starting program: /usr/src/xorg-server-1.8.2-1/build/xwin-ddx/hw/xwin/XWin.exe
> -engine 2
> [New thread 2208.0x904]
> [New thread 2208.0x664]
> warning: Lowest section in /cygdrive/c/WINDOWS/system32/wmi.dll is .text at
> 76d31000
> [New thread 2208.0x350]
> [New thread 2208.0x39c]
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x00413586 in winFreeFBShadowDD (pScreen=0x1008c1d8)
> at
> /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/hw/xwin/winshaddd.c:528
> 528 IDirectDrawSurface2_SetClipper (pScreenPriv->pddsPrimary,
> (gdb) bt
> #0 0x00413586 in winFreeFBShadowDD (pScreen=0x1008c1d8)
> at
> /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/hw/xwin/winshaddd.c:528
> #1 0x0042f917 in winDoRandRScreenSetSize (pScreen=0x1008c1d8, width=640,
> height=480,
> mmWidth=3435973836, mmHeight=1080233164)
> at /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/hw/xwin/winrandr.c:191
> #2 0x0041a369 in winWindowProc (hwnd=0x6b00f2, message=126, wParam=16,
> lParam=31457920)
> at
> /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/hw/xwin/winwndproc.c:344
> #3 0x7e418734 in USER32!GetDC () from /cygdrive/c/WINDOWS/system32/user32.dll
> #4 0x006b00f2 in ?? ()
> #5 0x0000007e in ?? ()
> #6 0x00000010 in ?? ()
> #7 0x01e00280 in ?? ()
> #8 0x00419bcc in winReshapeRootless ()
> #9 0x7e418816 in USER32!GetDC () from /cygdrive/c/WINDOWS/system32/user32.dll
> #10 0x00419bcc in winReshapeRootless ()
> #11 0x7e428ea0 in USER32!DefWindowProcW () from
> /cygdrive/c/WINDOWS/system32/user32.dll
> #12 0x00000000 in ?? ()
> (gdb) p pScreenPriv->pddsPrimary
> $2 = (LPDIRECTDRAWSURFACE2) 0x0
> (gdb)
>
> It looks like these are both the result of some bad code copied and
> pasted in winshaddd.c and winshadddnl.c. From winshadddnl.c, in
> winFreeFBShadowDDNL:
>
> /* Detach the clipper from the primary surface and release the clipper. */
> if (pScreenPriv->pddcPrimary)
> {
> /* Detach the clipper */
> IDirectDrawSurface4_SetClipper (pScreenPriv->pddsPrimary4,
> NULL);
>
> /* Release the clipper object */
> IDirectDrawClipper_Release (pScreenPriv->pddcPrimary);
> pScreenPriv->pddcPrimary = NULL;
> }
>
> /* Release the primary surface, if there is one */
> if (pScreenPriv->pddsPrimary4)
> {
> IDirectDrawSurface4_Release (pScreenPriv->pddsPrimary4);
> ...
>
> The call to IDirectDrawSurface4_SetClipper appears to be passed a
> pointer that may be invalid (as suggested by the same variable being
> explicitly checked before being used a few lines later). Printing the
> value of the pointer confirms it is null at the time of the crash.
>
> Unfortunately, this is only the first problem. I patched the code to
> check the validity of the pointer before this call, and the crash simply
> moved to another section of the code. Here's the next backtrace, after
> patching (and adding a debug build of pixman):
>
> $ gdb ./hw/xwin/XWin.exe
> GNU gdb 6.8.0.20080328-cvs (cygwin-special)
> Copyright (C) 2008 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "i686-pc-cygwin"...
> (gdb) run
> Starting program: /usr/src/xorg-server-1.8.2-1/build/xwin-ddx/hw/xwin/XWin.exe
> [New thread 2200.0x430]
> [New thread 2200.0x758]
> warning: Lowest section in /cygdrive/c/WINDOWS/system32/wmi.dll is .text at
> 76d31000
> [New thread 2200.0xfb0]
> [New thread 2200.0xbbc]
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x6fc4d664 in pixman_fill_sse2 (bits=0x7f8c0008, stride=6376, bpp=32, x=0,
> y=0, width=640,
> height=479, data=0) at
> /usr/src/pixman-0.18.2-1/src/pixman-0.18.2/pixman/pixman-sse2.c:4025
> 4025 *(uint32_t *)d = data;
> (gdb) bt
> #0 0x6fc4d664 in pixman_fill_sse2 (bits=0x7f8c0008, stride=6376, bpp=32, x=0,
> y=0, width=640,
> height=479, data=0) at
> /usr/src/pixman-0.18.2-1/src/pixman-0.18.2/pixman/pixman-sse2.c:4025
> #1 0x6fc651c5 in sse2_fill (imp=0x10087730, bits=0x7f8c0008, stride=1594,
> bpp=32, x=0, y=0,
> width=640, height=480, xor=0)
> at /usr/src/pixman-0.18.2-1/src/pixman-0.18.2/pixman/pixman-sse2.c:5888
> #2 0x6fb57724 in _pixman_implementation_fill (imp=0x10087730, bits=0x7f8c0008,
> stride=1594,
> bpp=32, x=0, y=0, width=640, height=480, xor=0)
> at /usr/src/pixman-0.18.2-1/src/pixman-0.18.2/pixman/pixman-implementation.c:225
> #3 0x6fb7cab5 in pixman_fill (bits=0x7f8c0008, stride=1594, bpp=32, x=0, y=0,
> width=640,
> height=480, xor=0) at
> /usr/src/pixman-0.18.2-1/src/pixman-0.18.2/pixman/pixman.c:864
> #4 0x004492c7 in fbFill (pDrawable=0x10087b38, pGC=0x10086ac0, x=0, y=0,
> width=640, height=480)
> at /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/fb/fbfill.c:48
> #5 0x004474eb in fbPolyFillRect (pDrawable=0x10087b38, pGC=0x10086ac0,
> nrect=0, prect=0x10291860)
> at /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/fb/fbfillrect.c:77
> #6 0x0052730b in damagePolyFillRect (pDrawable=0x10087b38, pGC=0x10086ac0,
> nRects=1,
> pRects=0x10291858)
> at
> /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/miext/damage/damage.c:1404
>
> #7 0x005b1f44 in miPaintWindow (pWin=0x10087b38, prgn=0x10291838, what=0)
> at /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/mi/miexpose.c:673
> #8 0x005b1a5c in miWindowExposures (pWin=0x10087b38, prgn=0x10291838,
> other_exposed=0x0)
> at /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/mi/miexpose.c:504
> #9 0x005b76fd in miHandleValidateExposures (pWin=0x10087b38)
> at /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/mi/miwindow.c:246
> #10 0x0042f874 in xf86SetRootClip (pScreen=0x1008c1b8, enable=1)
> at /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/hw/xwin/winrandr.c:164
> #11 0x0042f9a9 in winDoRandRScreenSetSize (pScreen=0x1008c1b8, width=640,
> height=480,
> mmWidth=3435973836, mmHeight=1080233164)
> at /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/hw/xwin/winrandr.c:212
> #12 0x0041a375 in winWindowProc (hwnd=0x370184, message=126, wParam=16,
> lParam=31457920)
> at
> /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/hw/xwin/winwndproc.c:344
> #13 0x7e418734 in USER32!GetDC () from /cygdrive/c/WINDOWS/system32/user32.dll
> #14 0x00370184 in ?? ()
> #15 0x0000007e in ?? ()
> #16 0x00000010 in ?? ()
> #17 0x01e00280 in ?? ()
> #18 0x00419bd8 in winReshapeRootless ()
> #19 0x7e418816 in USER32!GetDC () from /cygdrive/c/WINDOWS/system32/user32.dll
> #20 0x00419bd8 in winReshapeRootless ()
> #21 0x7e428ea0 in USER32!DefWindowProcW () from
> /cygdrive/c/WINDOWS/system32/user32.dll
> #22 0x00000000 in ?? ()
> (gdb) p d
> $1 = (uint8_t *) 0x7f8c0008 <Address 0x7f8c0008 out of bounds>
> (gdb)
>
> The backtrace is similar for the patched version using -engine 2 (Shadow
> DirectDraw locking). At this point, I should mention yet another case:
> -engine 1 (Shadow GDI) has also been crashing all along with a backtrace
> that is somewhat similar to the ones seen with the patched DirectDraw
> engines. Here's that backtrace:
>
> (gdb) run -engine 1
> The program being debugged has been started already.
> Start it from the beginning? (y or n) y
>
> Starting program: /usr/src/xorg-server-1.8.2-1/build/xwin-ddx/hw/xwin/XWin.exe
> -engine 1
> [New thread 2920.0xb04]
> [New thread 2920.0xce4]
> warning: Lowest section in /cygdrive/c/WINDOWS/system32/wmi.dll is .text at
> 76d31000
> [New thread 2920.0xf40]
> [New thread 2920.0x654]
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x6fc4d707 in pixman_fill_sse2 (bits=0x2b90000, stride=1280, bpp=32, x=0, y=0,
> width=640,
> height=0, data=0) at /usr/lib/gcc/i686-pc-cygwin/4.3.4/include/emmintrin.h:699
> 699 *__P = __B;
> (gdb) bt
> #0 0x6fc4d707 in pixman_fill_sse2 (bits=0x2b90000, stride=1280, bpp=32, x=0,
> y=0, width=640,
> height=0, data=0) at /usr/lib/gcc/i686-pc-cygwin/4.3.4/include/emmintrin.h:699
> #1 0x6fc651c5 in sse2_fill (imp=0x10087b78, bits=0x2b90000, stride=320,
> bpp=32, x=0, y=0,
> width=640, height=480, xor=0)
> at /usr/src/pixman-0.18.2-1/src/pixman-0.18.2/pixman/pixman-sse2.c:5888
> #2 0x6fb57724 in _pixman_implementation_fill (imp=0x10087b78, bits=0x2b90000,
> stride=320, bpp=32,
> x=0, y=0, width=640, height=480, xor=0)
> at /usr/src/pixman-0.18.2-1/src/pixman-0.18.2/pixman/pixman-implementation.c:225
> #3 0x6fb7cab5 in pixman_fill (bits=0x2b90000, stride=320, bpp=32, x=0, y=0,
> width=640,
> height=480, xor=0) at
> /usr/src/pixman-0.18.2-1/src/pixman-0.18.2/pixman/pixman.c:864
> #4 0x004492c7 in fbFill (pDrawable=0x10087f80, pGC=0x10086f80, x=0, y=0,
> width=640, height=480)
> at /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/fb/fbfill.c:48
> #5 0x004474eb in fbPolyFillRect (pDrawable=0x10087f80, pGC=0x10086f80,
> nrect=0, prect=0x1011a8f0)
> at /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/fb/fbfillrect.c:77
> #6 0x0052730b in damagePolyFillRect (pDrawable=0x10087f80, pGC=0x10086f80,
> nRects=1,
> pRects=0x1011a8e8)
> at
> /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/miext/damage/damage.c:1404
>
> #7 0x005b1f44 in miPaintWindow (pWin=0x10087f80, prgn=0x10291c30, what=0)
> at /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/mi/miexpose.c:673
> #8 0x005b1a5c in miWindowExposures (pWin=0x10087f80, prgn=0x10291c30,
> other_exposed=0x0)
> at /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/mi/miexpose.c:504
> #9 0x005b76fd in miHandleValidateExposures (pWin=0x10087f80)
> at /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/mi/miwindow.c:246
> #10 0x0042f874 in xf86SetRootClip (pScreen=0x1008c1c0, enable=1)
> at /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/hw/xwin/winrandr.c:164
> #11 0x0042f9a9 in winDoRandRScreenSetSize (pScreen=0x1008c1c0, width=640,
> height=480,
> mmWidth=3435973836, mmHeight=1080233164)
> at /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/hw/xwin/winrandr.c:212
> #12 0x0041a375 in winWindowProc (hwnd=0x1b02a0, message=126, wParam=16,
> lParam=31457920)
> at
> /usr/src/xorg-server-1.8.2-1/src/xserver-cygwin-1.8.2-1/hw/xwin/winwndproc.c:344
> #13 0x7e418734 in USER32!GetDC () from /cygdrive/c/WINDOWS/system32/user32.dll
> #14 0x001b02a0 in ?? ()
> #15 0x0000007e in ?? ()
> #16 0x00000010 in ?? ()
> #17 0x01e00280 in ?? ()
> #18 0x00419bd8 in winReshapeRootless ()
> #19 0x7e418816 in USER32!GetDC () from /cygdrive/c/WINDOWS/system32/user32.dll
> #20 0x00419bd8 in winReshapeRootless ()
> #21 0x7e428ea0 in USER32!DefWindowProcW () from
> /cygdrive/c/WINDOWS/system32/user32.dll
> #22 0x00000000 in ?? ()
> (gdb)
>
> At this point I'm a bit stuck. It looks like fbFill might be passing an
> invalid pointer to pixman_fill, but the code is hard to follow due to
> macros and unfamiliar APIs so I don't know where the root problem is. I
> may continue to investigate.
>
> By the way, another backtrace similar to this one showed up in the
> mailing list archives from last month:
>
> http://cygwin.com/ml/cygwin-xfree/2010-08/msg00068.html

These pixman crashes are possibly the result of trying to draw outside the 
screen pixmap.

-- 
Jon TURNEY
Volunteer Cygwin/X X Server maintainer

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://x.cygwin.com/docs/
FAQ:                   http://x.cygwin.com/docs/faq/