After doing IPT hsc last year my brain went rusty with all these definitions, but i'll try. Also communications was my least favourite topic lol
Handshaking is done between different hardware devices, where they communicate with each other (using drivers) and decide the potocols used, the speed of communication etc. The devices would take into account all their hardware limitations and find a common ground where they can communicate. Handshaking is a verb, and is this process of 'negotiation'.
Protocols are set ways or instructions in which devices work together to function. There are many different ones for each layer in the communications model. These indicate which device will send what, when, to who, what to do when this happens, how etc. Like IP just keeps on sending data packets to the destination without much error checking, but TCP will make sure that they all get there, without errors in the right order.
I dont know if i explained that well, and i see why you are confused. I think that maybe you can think of it as handshaking is like establishing protocols/limitations between devices and then protocols are used within various layers between nodes on a network??
handshaking = establishing communication between IPT/OSI layers (e.g establishing equal grounds between a NIC and an DAC)
protocols = ways in which nodes can communicate (e.g TCP/IP is used to send data between 2 nodes on a network (computer and server))
hope that helps....