running_log.txt

[INFO|2025-05-29 22:52:47] configuration_utils.py:696 >> loading configuration file config.json from cache at /home/kiho/.cache/huggingface/hub/models--deepseek-ai--deepseek-coder-7b-instruct-v1.5/snapshots/2a050a4c59d687a85324d32e147517992117ed30/config.json

[INFO|2025-05-29 22:52:47] configuration_utils.py:768 >> Model config LlamaConfig {
  "_name_or_path": "deepseek-ai/deepseek-coder-7b-instruct-v1.5",
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 100000,
  "eos_token_id": 100015,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 11008,
  "max_position_embeddings": 4096,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 30,
  "num_key_value_heads": 32,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 10000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.48.2",
  "use_cache": true,
  "vocab_size": 102400
}


[INFO|2025-05-29 22:52:47] tokenization_utils_base.py:2034 >> loading file tokenizer.model from cache at None

[INFO|2025-05-29 22:52:47] tokenization_utils_base.py:2034 >> loading file tokenizer.json from cache at /home/kiho/.cache/huggingface/hub/models--deepseek-ai--deepseek-coder-7b-instruct-v1.5/snapshots/2a050a4c59d687a85324d32e147517992117ed30/tokenizer.json

[INFO|2025-05-29 22:52:47] tokenization_utils_base.py:2034 >> loading file added_tokens.json from cache at None

[INFO|2025-05-29 22:52:47] tokenization_utils_base.py:2034 >> loading file special_tokens_map.json from cache at None

[INFO|2025-05-29 22:52:47] tokenization_utils_base.py:2034 >> loading file tokenizer_config.json from cache at /home/kiho/.cache/huggingface/hub/models--deepseek-ai--deepseek-coder-7b-instruct-v1.5/snapshots/2a050a4c59d687a85324d32e147517992117ed30/tokenizer_config.json

[INFO|2025-05-29 22:52:47] tokenization_utils_base.py:2034 >> loading file chat_template.jinja from cache at None

[INFO|2025-05-29 22:52:47] tokenization_utils_base.py:2304 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.

[INFO|2025-05-29 22:52:48] configuration_utils.py:696 >> loading configuration file config.json from cache at /home/kiho/.cache/huggingface/hub/models--deepseek-ai--deepseek-coder-7b-instruct-v1.5/snapshots/2a050a4c59d687a85324d32e147517992117ed30/config.json

[INFO|2025-05-29 22:52:48] configuration_utils.py:768 >> Model config LlamaConfig {
  "_name_or_path": "deepseek-ai/deepseek-coder-7b-instruct-v1.5",
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 100000,
  "eos_token_id": 100015,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 11008,
  "max_position_embeddings": 4096,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 30,
  "num_key_value_heads": 32,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 10000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.48.2",
  "use_cache": true,
  "vocab_size": 102400
}


[INFO|2025-05-29 22:52:49] tokenization_utils_base.py:2034 >> loading file tokenizer.model from cache at None

[INFO|2025-05-29 22:52:49] tokenization_utils_base.py:2034 >> loading file tokenizer.json from cache at /home/kiho/.cache/huggingface/hub/models--deepseek-ai--deepseek-coder-7b-instruct-v1.5/snapshots/2a050a4c59d687a85324d32e147517992117ed30/tokenizer.json

[INFO|2025-05-29 22:52:49] tokenization_utils_base.py:2034 >> loading file added_tokens.json from cache at None

[INFO|2025-05-29 22:52:49] tokenization_utils_base.py:2034 >> loading file special_tokens_map.json from cache at None

[INFO|2025-05-29 22:52:49] tokenization_utils_base.py:2034 >> loading file tokenizer_config.json from cache at /home/kiho/.cache/huggingface/hub/models--deepseek-ai--deepseek-coder-7b-instruct-v1.5/snapshots/2a050a4c59d687a85324d32e147517992117ed30/tokenizer_config.json

[INFO|2025-05-29 22:52:49] tokenization_utils_base.py:2034 >> loading file chat_template.jinja from cache at None

[INFO|2025-05-29 22:52:49] tokenization_utils_base.py:2304 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.

[INFO|2025-05-29 22:52:49] logging.py:157 >> Loading dataset Codes_query_filtered_330k_ns_over8_1.json...

[INFO|2025-05-29 22:53:02] configuration_utils.py:696 >> loading configuration file config.json from cache at /home/kiho/.cache/huggingface/hub/models--deepseek-ai--deepseek-coder-7b-instruct-v1.5/snapshots/2a050a4c59d687a85324d32e147517992117ed30/config.json

[INFO|2025-05-29 22:53:02] configuration_utils.py:768 >> Model config LlamaConfig {
  "_name_or_path": "deepseek-ai/deepseek-coder-7b-instruct-v1.5",
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 100000,
  "eos_token_id": 100015,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 11008,
  "max_position_embeddings": 4096,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 30,
  "num_key_value_heads": 32,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 10000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.48.2",
  "use_cache": true,
  "vocab_size": 102400
}


[WARNING|2025-05-29 22:53:02] logging.py:162 >> Input length is smaller than max length. Consider increase input length.

[INFO|2025-05-29 22:53:02] logging.py:157 >> Using llama3 scaling strategy and setting scaling factor to 1.0.

[INFO|2025-05-29 22:53:02] logging.py:157 >> Using block diagonal attention for sequence packing without cross-attention.

[INFO|2025-05-29 22:53:02] logging.py:157 >> Liger kernel has been applied to the model.

[INFO|2025-05-29 22:53:02] modeling_utils.py:3904 >> loading weights file model.safetensors from cache at /home/kiho/.cache/huggingface/hub/models--deepseek-ai--deepseek-coder-7b-instruct-v1.5/snapshots/2a050a4c59d687a85324d32e147517992117ed30/model.safetensors.index.json

[INFO|2025-05-29 22:53:02] modeling_utils.py:1582 >> Instantiating LlamaForCausalLM model under default dtype torch.bfloat16.

[INFO|2025-05-29 22:53:02] configuration_utils.py:1140 >> Generate config GenerationConfig {
  "bos_token_id": 100000,
  "eos_token_id": 100015
}


[INFO|2025-05-29 22:53:11] modeling_utils.py:4888 >> All model checkpoint weights were used when initializing LlamaForCausalLM.


[INFO|2025-05-29 22:53:11] modeling_utils.py:4896 >> All the weights of LlamaForCausalLM were initialized from the model checkpoint at deepseek-ai/deepseek-coder-7b-instruct-v1.5.
If your task is similar to the task the model of the checkpoint was trained on, you can already use LlamaForCausalLM for predictions without further training.

[INFO|2025-05-29 22:53:11] configuration_utils.py:1095 >> loading configuration file generation_config.json from cache at /home/kiho/.cache/huggingface/hub/models--deepseek-ai--deepseek-coder-7b-instruct-v1.5/snapshots/2a050a4c59d687a85324d32e147517992117ed30/generation_config.json

[INFO|2025-05-29 22:53:11] configuration_utils.py:1140 >> Generate config GenerationConfig {
  "bos_token_id": 100000,
  "eos_token_id": 100015
}


[INFO|2025-05-29 22:53:11] logging.py:157 >> Gradient checkpointing enabled.

[INFO|2025-05-29 22:53:11] logging.py:157 >> Using torch SDPA for faster training and inference.

[INFO|2025-05-29 22:53:11] logging.py:157 >> Upcasting trainable params to float32.

[INFO|2025-05-29 22:53:11] logging.py:157 >> Fine-tuning method: Freeze

[INFO|2025-05-29 22:53:11] logging.py:157 >> Set trainable layers: .28.,.29.

[INFO|2025-05-29 22:53:11] logging.py:157 >> trainable params: 404,766,720 || all params: 6,910,365,696 || trainable%: 5.8574

[INFO|2025-05-29 22:53:11] trainer.py:741 >> Using auto half precision backend

[INFO|2025-05-29 22:53:11] logging.py:157 >> Found linear modules: down_proj,gate_proj,o_proj,k_proj,up_proj,q_proj,v_proj

[INFO|2025-05-29 22:53:11] logging.py:157 >> Using APOLLO optimizer with args: {'rank': 256, 'proj': 'random', 'proj_type': 'std', 'update_proj_gap': 200, 'scale': 1, 'scale_type': 'channel', 'scale_front': False}.

[INFO|2025-05-29 22:53:11] trainer.py:2369 >> ***** Running training *****

[INFO|2025-05-29 22:53:11] trainer.py:2370 >>   Num examples = 6,272

[INFO|2025-05-29 22:53:11] trainer.py:2371 >>   Num Epochs = 1

[INFO|2025-05-29 22:53:11] trainer.py:2372 >>   Instantaneous batch size per device = 16

[INFO|2025-05-29 22:53:11] trainer.py:2375 >>   Total train batch size (w. parallel, distributed & accumulation) = 512

[INFO|2025-05-29 22:53:11] trainer.py:2376 >>   Gradient Accumulation steps = 8

[INFO|2025-05-29 22:53:11] trainer.py:2377 >>   Total optimization steps = 12

[INFO|2025-05-29 22:53:11] trainer.py:2378 >>   Number of trainable parameters = 404,766,720

[INFO|2025-05-29 22:54:58] logging.py:157 >> {'loss': 0.8480, 'learning_rate': 4.9148e-05, 'epoch': 0.08, 'throughput': 19880.44}

[INFO|2025-05-29 22:56:38] logging.py:157 >> {'loss': 0.8111, 'learning_rate': 4.6651e-05, 'epoch': 0.16, 'throughput': 20427.48}

[INFO|2025-05-29 22:58:17] logging.py:157 >> {'loss': 0.7984, 'learning_rate': 4.2678e-05, 'epoch': 0.24, 'throughput': 20649.67}

[INFO|2025-05-29 22:59:56] logging.py:157 >> {'loss': 0.7649, 'learning_rate': 3.7500e-05, 'epoch': 0.33, 'throughput': 20755.74}

[INFO|2025-05-29 23:01:36] logging.py:157 >> {'loss': 0.7687, 'learning_rate': 3.1470e-05, 'epoch': 0.41, 'throughput': 20825.02}

[INFO|2025-05-29 23:03:15] logging.py:157 >> {'loss': 0.7696, 'learning_rate': 2.5000e-05, 'epoch': 0.49, 'throughput': 20874.16}

[INFO|2025-05-29 23:04:54] logging.py:157 >> {'loss': 0.7488, 'learning_rate': 1.8530e-05, 'epoch': 0.57, 'throughput': 20909.51}

[INFO|2025-05-29 23:06:34] logging.py:157 >> {'loss': 0.7368, 'learning_rate': 1.2500e-05, 'epoch': 0.65, 'throughput': 20931.46}

[INFO|2025-05-29 23:08:13] logging.py:157 >> {'loss': 0.7088, 'learning_rate': 7.3223e-06, 'epoch': 0.73, 'throughput': 20947.01}

[INFO|2025-05-29 23:09:53] logging.py:157 >> {'loss': 0.7054, 'learning_rate': 3.3494e-06, 'epoch': 0.82, 'throughput': 20961.84}

[INFO|2025-05-29 23:11:32] logging.py:157 >> {'loss': 0.7032, 'learning_rate': 8.5185e-07, 'epoch': 0.90, 'throughput': 20974.14}

[INFO|2025-05-29 23:13:11] logging.py:157 >> {'loss': 0.7144, 'learning_rate': 0.0000e+00, 'epoch': 0.98, 'throughput': 20986.53}

[INFO|2025-05-29 23:13:11] trainer.py:3910 >> Saving model checkpoint to saves/DeepSeek-Coder-7B-Instruct/freeze/deepseek_nsx_8_1/checkpoint-12

[INFO|2025-05-29 23:13:11] configuration_utils.py:420 >> Configuration saved in saves/DeepSeek-Coder-7B-Instruct/freeze/deepseek_nsx_8_1/checkpoint-12/config.json

[INFO|2025-05-29 23:13:11] configuration_utils.py:909 >> Configuration saved in saves/DeepSeek-Coder-7B-Instruct/freeze/deepseek_nsx_8_1/checkpoint-12/generation_config.json

[INFO|2025-05-29 23:13:32] modeling_utils.py:2996 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 3 checkpoint shards. You can find where each parameters has been saved in the index located at saves/DeepSeek-Coder-7B-Instruct/freeze/deepseek_nsx_8_1/checkpoint-12/model.safetensors.index.json.

[INFO|2025-05-29 23:13:32] tokenization_utils_base.py:2491 >> tokenizer config file saved in saves/DeepSeek-Coder-7B-Instruct/freeze/deepseek_nsx_8_1/checkpoint-12/tokenizer_config.json

[INFO|2025-05-29 23:13:32] tokenization_utils_base.py:2500 >> Special tokens file saved in saves/DeepSeek-Coder-7B-Instruct/freeze/deepseek_nsx_8_1/checkpoint-12/special_tokens_map.json

[INFO|2025-05-29 23:13:32] trainer.py:2643 >> 

Training completed. Do not forget to share your model on huggingface.co/models =)



[INFO|2025-05-29 23:13:32] trainer.py:3910 >> Saving model checkpoint to saves/DeepSeek-Coder-7B-Instruct/freeze/deepseek_nsx_8_1

[INFO|2025-05-29 23:13:32] configuration_utils.py:420 >> Configuration saved in saves/DeepSeek-Coder-7B-Instruct/freeze/deepseek_nsx_8_1/config.json

[INFO|2025-05-29 23:13:32] configuration_utils.py:909 >> Configuration saved in saves/DeepSeek-Coder-7B-Instruct/freeze/deepseek_nsx_8_1/generation_config.json

[INFO|2025-05-29 23:13:53] modeling_utils.py:2996 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 3 checkpoint shards. You can find where each parameters has been saved in the index located at saves/DeepSeek-Coder-7B-Instruct/freeze/deepseek_nsx_8_1/model.safetensors.index.json.

[INFO|2025-05-29 23:13:53] tokenization_utils_base.py:2491 >> tokenizer config file saved in saves/DeepSeek-Coder-7B-Instruct/freeze/deepseek_nsx_8_1/tokenizer_config.json

[INFO|2025-05-29 23:13:53] tokenization_utils_base.py:2500 >> Special tokens file saved in saves/DeepSeek-Coder-7B-Instruct/freeze/deepseek_nsx_8_1/special_tokens_map.json

[WARNING|2025-05-29 23:13:54] logging.py:162 >> No metric eval_loss to plot.

[WARNING|2025-05-29 23:13:54] logging.py:162 >> No metric eval_accuracy to plot.

[INFO|2025-05-29 23:13:54] modelcard.py:449 >> Dropping the following result as it does not have all the necessary fields:
{'task': {'name': 'Causal Language Modeling', 'type': 'text-generation'}}