FIX: Correctly save a ArgillaTrainer + TRL model with peft_config

[tomaarsen]

Description

If you pass peft_config to trainer.update_config, then it will currently still save the full (but broken) model with some LoRA (or other PEFT) parts interspersed that can't be loaded again. This is because the self._transformers_model was saved, but that model was the original model (e.g. Llama 2 7b) and not the PEFT model, which is created internally in e.g. the SFTTrainer.

In short, this now saves the actual model and tokenizer used by the underlying trainer, so that if that trainer make any other changes to the model, those changes will be saved too.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

How Has This Been Tested

Created a new test and verified that there's now adapter_model.bin and adapter_config.json files after training.

Checklist

• I followed the style guidelines of this project
• I did a self-review of my code
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I filled out the contributor form (see text above)
• I have added relevant notes to the CHANGELOG.md file (See https://keepachangelog.com/)

---

• Tom Aarsen

[tomaarsen]

This also works in my RLHF tutorial.

[codecov]

Codecov Report

Patch has no changes to coverable lines.

📢 Thoughts on this report? Let us know!.

[github-actions]

The URL of the deployed environment for this PR is https://argilla-quickstart-pr-3795-ki24f765kq-no.a.run.app