First of all, use Bobos method, its slick.
But, I think you can use Bobos flip method for the final section of your initial idea, it just doesnt make sense to test the z component of the cross product at the end, since we have not aligned our coordinate system to the new Z Axis.
However, it does make sense to test the cross product vector against the (p4 – p2) vector. We can Compare our vector produced (using the cross product) to the normalized (p4 -p2) vector using a dot product! If they are parallel the dot product would be 1, since they are both normalized, but as the cross product flips, the result dot product should flip to -1, and you can test for this sign change
So your the Code may change to something like (using the intial method) …
--First Normalize all the vectors
VerticalVec = normalize(p4-p2) -- This is our reference vector
p5Vec = normalize(p5-p4)
p1Vec = normalize(p1 -p2)
--Now we can create our projected vectors onto the plane -- they should be normalised since both in the input vectors are normalized
p5ProjectedVec = cross VerticalVec p5Vec
p1ProjectedVec = cross VerticalVec p1Vec
--From there we have the two needed normalized vectors so we can calculate the angle using the function in help...
fn GetVectorsAngle v1 v2 =
(
theAngle = acos(dot (normalize v1) (normalize v2))
)
theAngle = GetVectorsAngle p5ProjectedVec p1ProjectedVec
--Now introduce the Check against VerticalVec, by producing a flip vector for the test
FlipTestVec = (cross p1ProjectedVec p5ProjectedVec) --this should be normalized
--Now we can compare this vector to our axis vector VerticalVec using the dot product and check for the sign change!
if dot (VerticalVec FlipTestVec) < 0 then theAngle = 360 - theAngle
--There may be a sign error here.... check it if you have time!
In principle I think this successfully builds in Bobos flip method to your initial way of solving the problem. I have not had time or max to test it, and I really should before pasting it around everywhere, but it really is here just for completeness, although again I would be interested to see if it actually works! If anyone else sees a flaw in the logic please point it out.
Proceed with Bobos method though, and in principle I would try and solve these types of problems with his matrixnormal approach…
Good luck
Rich