Software polygon rasterization with transparency using raw image data

Ok, here is the solution.

First, you need to ensure that your vertices are winded in a correct direction. For this explanation I'm going to use counter-clockwise order. Like this:

    1

2       3

And our coordinate system is:

Y
^
|
|
0------> X

Let's call our 3 points a, b and c.

Let's create some vectors: b - a = A, c - b = B, a - c = C.
Now we need to rotate them 90 degrees clockwise: [A.y,-A.x] = AA, [B.y,-B.x] = BB, [C.y,-C.x] = CC.

Now you need to understand what dot product is: dot(a, b) = a.x*b.x + a.y&b.y. It has useful properties. For example, dot product of a and b is equal to length of a * length of b * cos of angle between a and b.

And now we can test if some point is in this triangle: The point is inside if dot(AA, point - a) < 0 and dot(BB, point - b) < 0 and dot(CC, point - c) < 0.

And now we take a minimal square that contains all 3 points. Like this:

+--a----+
|       |
|       c
|       |
b-------+

And now you just test every point in this square.

You can use same method for quads, you just need to add 4th point.