Determining if three points in a polygon are convex or not doesnt work

  angle, c++, geometry, graphics, math

I want to implement the ear-clipping algorithm, and I found a code which just does that.
The problem is, that it sometimes goes into infinite loop, because the function which is determining if three point in the polygon is convex or not, returns with an incorrect value.

Here’s the code:

bool isConvex(Point* active) {

    Point* a = active->prev;

    Point* b = active;

    Point* c = active->next;

    return ((a->x * (c->y - b->y)) + (b->x * (a->y - c->y)) + (c->x * (b->y - a->y))) < 0;

The points form a doubly linked list, the pointers points to the neighbors.

The problem always occurs when there’s only 3 points left.

An example of three points which produce infinite loop:


enter image description here

Source: Windows Questions C++