I guess we never got an answer. I believe pajarito7 was very likely correct and was probably referring to a Hill cipher. If that's true, some linear algebra techniques might help - let me see.... um... yeah last took that in the old abacus days of the 80's. All forgotten.
If this were an extra credit assignment, that might explain the professor professed behavior.