| | --- |
| | license: other |
| | base_model: deepseek-ai/deepseek-coder-1.3b-base |
| | tags: |
| | - axolotl |
| | - generated_from_trainer |
| | model-index: |
| | - name: deepseek-coder-1.3b-typescript |
| | results: [] |
| | datasets: |
| | - bigcode/the-stack-dedup |
| | widget: |
| | - text: "class Person {\n constructor(public name:" |
| | example_title: "class" |
| | - text: "function quickSort" |
| | example_title: "function" |
| | --- |
| | |
| | <p align="center"> |
| | <img width="1000px" alt="CodeGPT: DeepSeek Coder - Typescript" src="codegpt-deepseek-typescript.png?raw=true"> |
| | </p> |
| | <p align="center"><a href="https://codegpt.co/">[CodeGPT.co]</a> | <a href="https://ollama.ai/codegpt/deepseek-coder-1.3b-typescript">[🦙 Ollama]</a> | <a href="https://discord.gg/fKyyJX5pne">[Discord]</a> | <a href="https://marketplace.visualstudio.com/items?itemName=DanielSanMedium.dscodegpt">[VSCode Extension]</a> </p> |
| | <hr> |
| |
|
| | [<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl) |
| | <details><summary>See axolotl config</summary> |
| |
|
| | axolotl version: `0.3.0` |
| | ```yaml |
| | base_model: deepseek-ai/deepseek-coder-1.3b-base |
| | model_type: AutoModelForCausalLM |
| | trust_remote_code: true |
| | load_in_8bit: false |
| | load_in_4bit: false |
| | strict: false |
| | |
| | |
| | datasets: |
| | - path: CodeGPTPlus/typescript-0-500000-seq1024 |
| | type: completion |
| | field: text |
| | |
| | |
| | val_set_size: 0.001 |
| | output_dir: ./fft-out |
| | |
| | sequence_len: 1024 |
| | |
| | adapter: |
| | lora_model_dir: |
| | lora_r: |
| | lora_alpha: |
| | lora_dropout: |
| | lora_target_linear: |
| | lora_fan_in_fan_out: |
| | lora_modules_to_save: |
| | |
| | wandb_project: deepseek_1.3_fft |
| | wandb_entity: |
| | wandb_watch: |
| | wandb_name: aws_a10g |
| | wandb_log_model: end |
| | |
| | |
| | gradient_accumulation_steps: 2 |
| | micro_batch_size: 20 |
| | num_epochs: 1 |
| | optimizer: adamw_bnb_8bit |
| | adam_beta1: 0.9 |
| | adam_beta2: 0.999 |
| | adam_epsilon: 0.000001 |
| | max_grad_norm: 1.0 |
| | weight_decay: 0.1 |
| | lr_scheduler: cosine |
| | learning_rate: 0.00002 |
| | train_on_inputs: false |
| | group_by_length: false |
| | bf16: true |
| | fp16: false |
| | tf32: false |
| | gradient_checkpointing: true |
| | early_stopping_patience: |
| | resume_from_checkpoint: |
| | local_rank: |
| | logging_steps: 1 |
| | xformers_attention: |
| | flash_attention: true |
| | |
| | loss_watchdog_threshold: 5.0 |
| | loss_watchdog_patience: 3 |
| | |
| | hub_model_id: CodeGPTPlus/deepseek_coder_1.3b_typescript |
| | hub_strategy: every_save |
| | warmup_ratio: 0.01 |
| | evals_per_epoch: 20 |
| | saves_per_epoch: 3 |
| | debug: |
| | deepspeed: |
| | |
| | fsdp: |
| | fsdp_config: |
| | special_tokens: |
| | bos_token: "<｜begin▁of▁sentence｜>" |
| | eos_token: "<｜end▁of▁sentence｜>" |
| | pad_token: "<｜end▁of▁sentence｜>" |
| | ``` |
| |
|
| | </details><br> |
| |
|
| | # deepseek-coder-1.3b-typescript |
| |
|
| | CodeGPTPlus/deepseek-coder-1.3b-typescript, emerges as a fine-tuned iteration of [deepseek-ai/deepseek-coder-1.3b-base](https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-base), meticulously crafted by the CodeGPT team to excel in generating expert code in TypeScript. With specific fine-tuning for TypeScript and a dataset of 0.5B tokens, this model excels in producing precise and efficient solutions in this programming language. |
| |
|
| | The 16K window size and an additional fill-in-the-middle task are employed to deliver project-level code completion. |
| |
|
| | This new model stands as the ideal choice for those seeking a specialized code generator for TypeScript, backed by the expertise of the CodeGPT team. |
| |
|
| | It achieves the following results on the evaluation set: |
| | - Loss: 0.7681 |
| |
|
| | **Model Developers** CodeGPT Team |
| |
|
| | **Variations** 1.3B |
| |
|
| | **Input** Models input text only. |
| |
|
| | **Output** Models generate text only. |
| |
|
| | ## How to Use |
| | This model is for completion purposes only. Here give some examples of how to use the model. |
| |
|
| | #### Running the model on a GPU |
| | ```python |
| | from transformers import AutoTokenizer, AutoModelForCausalLM |
| | tokenizer = AutoTokenizer.from_pretrained("CodeGPTPlus/deepseek-coder-1.3b-typescript", |
| | trust_remote_code=True) |
| | model = AutoModelForCausalLM.from_pretrained("CodeGPTPlus/deepseek-coder-1.3b-typescript", |
| | trust_remote_code=True).cuda() |
| | |
| | input_text = """<｜fim▁begin｜>function quickSort(arr: number[]): number[] { |
| | if (arr.length <= 1) { |
| | return arr; |
| | } |
| | const pivot = arr[0]; |
| | const left = []; |
| | const right = []; |
| | <｜fim▁hole｜> |
| | return [...quickSort(left), pivot, ...quickSort(right)]; |
| | }<｜fim▁end｜>""" |
| | |
| | inputs = tokenizer(input_text, return_tensors="pt").to(model.device) |
| | outputs = model.generate(**inputs, max_length=256) |
| | print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| | ``` |
| |
|
| | ### Running with Ollama |
| | **Model:** https://ollama.ai/codegpt/deepseek-coder-1.3b-typescript |
| |
|
| | ```ollama run codegpt/deepseek-coder-1.3b-typescript``` |
| |
|
| | ### Running with Ollama and CodeGPT Autocomplete in VSCode |
| |
|
| | **Documentation:** https://docs.codegpt.co/docs/tutorial-features/code_autocompletion |
| | |
| | Select "Ollama - codegpt/deepseek-coder-1.3b-typescript" in the autocomplete model selector. |
| | |
| | Then, write any code or comment in the vscode text editor, and the model will provide you with code suggestions through the CodeGPT code autocomplete. |
| | |
| | <img width="1000px" alt="CodeGPT: DeepSeek Coder - Typescript" src="ollama_autocomplete_codegpt.gif"> |
| | |
| | ### Fill In the Middle (FIM) |
| | ```python |
| | <｜fim▁begin｜>function quickSort(arr: number[]): number[] { |
| | if (arr.length <= 1) { |
| | return arr; |
| | } |
| | const pivot = arr[0]; |
| | const left = []; |
| | const right = []; |
| | <｜fim▁hole｜> |
| | return [...quickSort(left), pivot, ...quickSort(right)]; |
| | }<｜fim▁end｜> |
| | ``` |
| | |
| | ## Training procedure |
| | |
| | ### Training hyperparameters |
| | |
| | The following hyperparameters were used during training: |
| | - learning_rate: 2e-05 |
| | - train_batch_size: 20 |
| | - eval_batch_size: 20 |
| | - seed: 42 |
| | - gradient_accumulation_steps: 2 |
| | - total_train_batch_size: 40 |
| | - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-06 |
| | - lr_scheduler_type: cosine |
| | - lr_scheduler_warmup_steps: 261 |
| | - num_epochs: 1 |
| | |
| | ### Training results |
| | |
| | | Training Loss | Epoch | Step | Validation Loss | |
| | |:-------------:|:-----:|:-----:|:---------------:| |
| | | 1.0745 | 0.0 | 1 | 0.8681 | |
| | | 1.2267 | 0.05 | 1308 | 0.8130 | |
| | | 1.1594 | 0.1 | 2616 | 0.8018 | |
| | | 0.7674 | 0.15 | 3924 | 0.7942 | |
| | | 0.6443 | 0.2 | 5232 | 0.7889 | |
| | | 0.9155 | 0.25 | 6540 | 0.7847 | |
| | | 0.7501 | 0.3 | 7848 | 0.7819 | |
| | | 0.8835 | 0.35 | 9156 | 0.7792 | |
| | | 0.7261 | 0.4 | 10464 | 0.7769 | |
| | | 0.9746 | 0.45 | 11772 | 0.7748 | |
| | | 0.6884 | 0.5 | 13080 | 0.7734 | |
| | | 0.6104 | 0.55 | 14388 | 0.7722 | |
| | | 0.8876 | 0.6 | 15696 | 0.7710 | |
| | | 0.9567 | 0.65 | 17004 | 0.7703 | |
| | | 0.6915 | 0.7 | 18312 | 0.7696 | |
| | | 0.8874 | 0.75 | 19620 | 0.7691 | |
| | | 0.6124 | 0.8 | 20928 | 0.7686 | |
| | | 0.8147 | 0.85 | 22236 | 0.7684 | |
| | | 0.8021 | 0.9 | 23544 | 0.7683 | |
| | | 0.8665 | 0.95 | 24852 | 0.7681 | |
| | |
| | |
| | ### Framework versions |
| | |
| | - Transformers 4.37.0.dev0 |
| | - Pytorch 2.0.1+cu118 |
| | - Datasets 2.16.1 |
| | - Tokenizers 0.15.0 |
