From memory (did my HSC last year), open market operations is how the RBA affects the cash supply in the overnight money market.
They can influence the cash rate by:
i) Selling government securities such as bonds to the major financial institutions (banks like CBA, ANZ, Westpac, NAB etc.). As such, the banks have to pay money for these bonds out of their exchange settlement accounts. Essentially, this removes money out of the overnight money market and causes an increase in the cost of credit or if you like, the interest rate.
ii) Conversely, the RBA can increase the interest rate by buying back government securities from the major banks. This will increase the money in the overnight money market (through the increased funds in the exchange settlement accounts) and therefore, reduce the cost of borrowing.
The best way of remembering or making sense of the open market operations is to use a supply and demand diagram.
Hope that helps!