Fine-tune my Coding-LLM w/ PEFT LoRA Quantization - PART 2
Coding-LLM are trained on old data. Even the latest GPT-4 Turbo Code Interpreter (CI) has a knowledge cut-off at April 2023. All AI research from the last 7 moths are not in the training data of commercial coding LLMs. And RAG lines of code do not help at all, given the complex interdependencies of code libs.
Therefore an elegant solution for AI researcher is to fine-tune your own Coding-LLM on the latest GitHub repos and coding data. Which is exactly the content of this video: How to fine-tune your personal coding-LLM (or a Co-pilot like Microsoft's GitHub co-pilot or any CODE-LLM like StarCoder).
We code a classical fine-tuning of a Code LLM (StarCoder), then code a PEFT-LoRA tuning for a personal Code LLM and also code a QLoRA tuning based on a special code dataset and a pre-trained Code LLM.
We compare compute infrastructure requirements for distributed computing (8x A100 80GB GPU) and apply DeepSpeed and fully shared Data Parallel (FSDP) for memory optimization.
Python files available for download at (all rights with authors of GitHub repos /HuggingFace wizards)):
-----------------------------------------------------------------------------------------------------------------------------------------
TRAIN.PY
https://github.com/pacman100/DHS-LLM-...
DEEPSPEED
https://github.com/pacman100/DHS-LLM-...
FSDP
https://github.com/pacman100/DHS-LLM-...
PEFT LoRA
https://github.com/pacman100/DHS-LLM-...
Complete Colab NB for PEFT LoRA Quantization Fine-tuning my personal CODE LLM:
https://colab.research.google.com/dri...
#ai
#coding
#pythonprogramming
Watch video Fine-tune my Coding-LLM w/ PEFT LoRA Quantization - PART 2 online without registration, duration hours minute second in high quality. This video was added by user Discover AI 12 November 2023, don't forget to share it with your friends and acquaintances, it has been viewed on our site 2,273 once and liked it 94 people.