I don't see why you need to implement it in the core files. You would need to:
1) Store a public and private key for each user, you can either augment the ossnusers table or make your own additional table.
2) make a component that does the encryption/decryption routines on messages right before they are sent and right after being received. you _could alternatively do this by editing the OssnMessages component, but it's not suggested to do that.