$\begingroup$

I've been trying to make an algorithm to find the number of all possible simple quadrilaterals in a N*M lattice. I already have a brute force solution but since this is a Project Euler problem I believe it should be possible to solve much faster than I'm doing and so I'm taking a math approach instead. I haven't figured out much unfortunately. I tried to go about this using binomial coefficients and failed. Now I'm trying to use Pick's theorem which pretty much ensures we are dealing with simple polygons but I am not sure how to handle overlapping quads. The reason I'm posting therefore is to see if there is any other math approach I can take. I don't want any solutions just clues on the math part since I am not that educated on these math subjects.

Edit: I took a completely different approach so this has no need for me anymore. I am leaving it open for others.