You're right, I can't deny that Christ authorised His disciples to establish a Church in His name. Yes they did receive the Holy Spirit and did perform miracles, however I don't see how this is relevant to priests forgiving sin?
Funnily enough, "the Church", also known as the body of Christ...