We start by putting this into matrix form:

Now, we know that this matrix equation cannot be solved if the inverse of the first matrix doesn't exist. For an inverse to exist, the matrix cannot be singular - this means that the determinant must not equal zero. So, if we find the determinant of the first matrix when it equals zero, this will give us
 - 4a = 0 \implies a=\frac{-5b}{4})
.
However, we're not done yet - graphically, we know that there are two situations when we cannot find a solution. When there are no solutions (parallel lines), or infinite solutions (they're actually the same line). The question wants us to find when there are no solutions, so we check which situation we have by picking values for a and b in the ratio given above. So, if we let b = 4, then a = -5, giving us:

So, we see there are no solutions as the lines are parallel. So, the given simultaneous equations have no solution when

, so the answer is C.
EDIT: upon re-reading the question, I realised that I actually went too far into it. Since the question only asks for when there are no unique solutions, you don't have to double check, essentially stopping once you found that

. However, if they ask for when there's no solutions or infinite solutions, you must confirm as above.