From the definition of linear dependence, the vectors a and b are linearly dependent if and only if xa + yb = 0, where x and y are scalars not all 0. This means that if there exists scalars x and y not all 0 such that the equation above is satisfied, then the set of vectors consisting of a and b is linearly dependent. Let's see whether such scalars do indeed exist...
First, assume that there does exist such scalars. Then:
xa + yb = 0
x(i - j + 3/2 k) + y(-2i + 2j + 3k) = 0
2xi - 2xj + 3/2x k - 2yi + 2yj + 3yk = 0
(2x - 2y) i + (2y - 2x) j + (3/2x + 3y)k = 0
As the textbook says, the next step is to equate coefficients. However, what many textbooks fail to explain is why it is legitimate in this case to simply equate coefficients to determine solutions for x and y. Normally, equating coefficients gives you only one of many solutions. However, in this case, equating coefficients gives you all the solutions. The reason is a little complicated, but worth getting your head around. Suppose you had a box containing three vectors that have an i, j and k component and there was no possible way in which you could form the zero vector by adding scalar multiples of the three vectors in the box. By definition, we say that the set of three vectors is linearly independent. Believe it or not, just by adding scalar multiples of the three vectors in the box, we can form every single vector in R^3. Another way of saying this is that every vector in R^3 can be represented as a linear combination of the three vectors in the box. But that's not all! Since the three vectors form a linearly independent set of vectors, we can go one step further and say that every vector in R^3 can be represented UNIQUELY as a linear combination of the three vectors in the box, a result which can be readily proven using proof by contradiction. This is significant. This means that if I wanted to write i + 2j - 3k as a linear combination of the three vectors in the box, there is only ONE way that I can do so. The implication of this is precisely that, for a linearly independent set of vectors, we CAN equate coefficients without losing any solutions (since there is always only ONE way of representing a particular vector in R^3 as a linear combination of the vectors in the set). Note in our case that the standard basis vectors i, j and k are linearly independent. Hence, we can legitimately equate coefficients...
2x - 2y = 0 and 2y - 2x = 0 and 3/2x + 2y = 0
Solving the three equations above simultaneously yields x = 0 and y = 0.
Recall the definition of linear dependence. The vectors a and b are linearly dependent if and only if xa + yb = 0, where x and y are scalars not all 0. From the working out above, it is clear that the only way in which we can satisfy the equation above is if we set x = 0 and y = 0. Hence, there does not exist scalars x and y not all 0 such that the equation above is satisfied. Hence, the vectors a and b are NOT linearly dependent, which means that they are linearly independent.
Hope this makes sense!