crashing bug : found... destroyed

[Brian Michael Genisio]

I wanted to report a bug I found in the XWin server. ( It pops up in
 the VFB and probably Xnest as well.)  It has to do with the drawing of poly
lines. Unfortunately, I do not have a program I can give you to recreate the
bug, but take a look, and hopefully add the fix the the XFree source.

The problem is in the  mi/miwideline.c file.  It makes use of the math function
hypot ( hypotonuse ... sqrt(a*a + b*b) ) from math.h.  Because of the way the
defines are set up in the beginning, this file never gets an appropriate
prototype for `double hypot(double, double) `.

Since it had no proper prototype, I am assuming it was storing the value as a
float or int, instead of a double and the value in the line

L = hypot ((double) dx, (double) dy);

(inside the miWideSegment function) was being stored badly, causing unexpected
results... sometimes a crash due to data overwrites (which I was running into).

This has probably been run into before, since in the beginning of the file, we
see this code :

#ifdef _XOPEN_SOURCE
#include <math.h>
#else
#define _XOPEN_SOURCE   /* to get prototype for hypot on some systems */
#include <math.h>
#undef _XOPEN_SOURCE
#endif

I looked at the math.h file on my solaris box, and I see that hypot is
prototyped if _XOPEN_SOURCE is defined.  Soo.... it looks like it was an issue
for solaris.  Unfortunately, math.h in cygwin does not use the define
_XOPEN_SOURCE to determine when to use hypot.

Instead, it prototypes hypot ONLY when __STRICT_ANSI__ is NOT defined.  Well,
in the build of this file, STRICT_ANSI must be getting defined, since I have
determined that hypot is not being prototyped.  I determined this by compiling
miwideline.c with all of the XFree flags, and included the -E option, which
preprocesses the file, but does not compile it.  This output proved that hypot
was not being prototyped.





THE FIX :
I simply undefine __STRICT_ANSI__ before the math.h include, and redefine it
after :



#if defined(__STRICT_ANSI__) && defined(__CYGWIN__)
#undef __STRICT_ANSI__ /* undefine to insure hypot */

#ifdef _XOPEN_SOURCE
#include <math.h>
#else
#define _XOPEN_SOURCE   /* to get prototype for hypot on some systems */
#include <math.h>
#undef _XOPEN_SOURCE
#endif

#define __STRICT_ANSI__
#else /*#if defined(__STRICT_ANSI__) && defined(__CYGWIN__)*/

#ifdef _XOPEN_SOURCE
#include <math.h>
#else
#define _XOPEN_SOURCE   /* to get prototype for hypot on some systems */
#include <math.h>

#undef _XOPEN_SOURCE
#endif

#endif /*#if defined(__STRICT_ANSI__) && defined(__CYGWIN__)*/




By putting this at the beginning of the file, instead of the old version (noted
above), the bug is worked around... but we may run into it with other files.
Either way, I hope this fix can be incorporated into the XFree code.

Thanks,
Brian




--
---------------------------
Brian Genisio
genisiob@pilot.msu.edu