Eval bug: example/finetune.cpp crashing

[killjaqular]

Name and Version

./build/bin/llama-finetune --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA H100 NVL, compute capability 9.0, VMM: yes
  Device 1: NVIDIA H100 NVL, compute capability 9.0, VMM: yes
version: 5754 (2bf9d539)
built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu

Operating systems

Linux

GGML backends

CPU

Hardware

some_user@some_hardware:~$ lscpu

Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          52 bits physical, 57 bits virtual
  Byte Order:             Little Endian
CPU(s):                   256
  On-line CPU(s) list:    0-255
Vendor ID:                AuthenticAMD
  Model name:             AMD EPYC 9554 64-Core Processor
    CPU family:           25
    Model:                17
    Thread(s) per core:   2
    Core(s) per socket:   64
    Socket(s):            2
    Stepping:             1
    Frequency boost:      enabled
    CPU max MHz:          3100.0000
    CPU min MHz:          1500.0000
    BogoMIPS:             6199.89
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_
                          opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 f
                          ma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a
                           misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l
                          3 invpcid_single hw_pstate ssbd mba ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_
                          a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsav
                          es cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat
                          npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif
                           v_spec_ctrl avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdp
                          id overflow_recov succor smca fsrm flush_l1d
Virtualization features:
  Virtualization:         AMD-V
Caches (sum of all):
  L1d:                    4 MiB (128 instances)
  L1i:                    4 MiB (128 instances)
  L2:                     128 MiB (128 instances)
  L3:                     512 MiB (16 instances)
NUMA:
  NUMA node(s):           2
  NUMA node0 CPU(s):      0-63,128-191
  NUMA node1 CPU(s):      64-127,192-255
Vulnerabilities:
  Gather data sampling:   Not affected
  Itlb multihit:          Not affected
  L1tf:                   Not affected
  Mds:                    Not affected
  Meltdown:               Not affected
  Mmio stale data:        Not affected
  Reg file data sampling: Not affected
  Retbleed:               Not affected
  Spec rstack overflow:   Mitigation; safe RET
  Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl and seccomp
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not aff
                          ected
  Srbds:                  Not affected
  Tsx async abort:        Not affected

Models

Dolphin3.0-Llama3.2-1B-f32.gguf
smollm2-135M-base-f32.gguf

I was really hoping this model would work, seeing how it's the one being used in examples/training/README.md
llama3.2-1b-f32.gguf

Problem description & steps to reproduce

[BUILD]:
From the file: docs/build.md34-35

cmake -B build -DCMAKE_BUILD_TYPE=Debug
cmake --build build

[RUN]:
From the file: examples/training/README.md:12-14

./build/bin/llama-finetune --file datasets/wikitext-2-raw/wiki.test.raw -ngl 999 --model models/${model_name}-${quantization}.gguf -c 512 -b 512 -ub 512

I have tried using these models:
Dolphin3.0-Llama3.2-1B-f32.gguf
llama3.2-1b-f32.gguf
smollm2-135M-base-f32.gguf

Has anyone else encountered this issue when trying to run the finetune example as written?

I am trying to run the finetune example as-is following the information given in the docs/build.md and README.md and examples/training/README.md

First Bad Commit

some_user@some_hardware:~/dev/some_project/llama.cpp$: git log
commit 2bf9d539dd158345e3a3b096e16474af535265b4 (HEAD -> master, tag: b5754, origin/master, origin/HEAD)
Author: Anton Mitkov <anton_b_mitkov@abv.bg>
Date:   Wed Jun 25 17:09:55 2025 +0100

    sycl: GGML_SYCL_DISABLE_OPT on by default for all Intel Devices (#13973)

Relevant log output

[RESULT]:
some_user@some_hardware:~/dev/some_project/llama.cpp$ ./build/bin/llama-finetune --file datasets/wikitext-2-raw/wiki.test.raw -ngl 999 --model models/${model_name}-${quantization}.gguf -c 512 -b 512 -ub 512

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA H100 NVL, compute capability 9.0, VMM: yes
  Device 1: NVIDIA H100 NVL, compute capability 9.0, VMM: yes
register_backend: registered backend CUDA (2 devices)
register_device: registered device CUDA0 (NVIDIA H100 NVL)
register_device: registered device CUDA1 (NVIDIA H100 NVL)
register_backend: registered backend CPU (1 devices)
register_device: registered device CPU (AMD EPYC 9554 64-Core Processor)
load_backend: failed to find ggml_backend_init in /home/adonay/dev/sentinel/llama.cpp/build/bin/libggml-cuda.so
load_backend: failed to find ggml_backend_init in /home/adonay/dev/sentinel/llama.cpp/build/bin/libggml-cpu.so
main: force disabling memory mapping because it would result in-read-only pointers to the weights
main: force changing k cache type to f32 due to a lack of f16 support for OUT_PROD
main: force changing v cache type to f32 due to a lack of f16 support for OUT_PROD
build: 5754 (2bf9d539) with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu (debug)
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA H100 NVL) - 94035 MiB free
llama_model_load_from_file_impl: using device CUDA1 (NVIDIA H100 NVL) - 94805 MiB free
llama_model_loader: loaded meta data with 33 key-value pairs and 147 tensors from models/llama3.2-1b-f32.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Llama 3.2 1b
llama_model_loader: - kv   3:                       general.organization str              = Meta Llama
llama_model_loader: - kv   4:                           general.basename str              = llama-3.2
llama_model_loader: - kv   5:                         general.size_label str              = 1B
llama_model_loader: - kv   6:                            general.license str              = llama3.2
llama_model_loader: - kv   7:                   general.base_model.count u32              = 1
llama_model_loader: - kv   8:                  general.base_model.0.name str              = Llama 3.2 1B
llama_model_loader: - kv   9:          general.base_model.0.organization str              = Meta Llama
llama_model_loader: - kv  10:              general.base_model.0.repo_url str              = https://huggingface.co/meta-llama/Lla...
llama_model_loader: - kv  11:                               general.tags arr[str,2]       = ["gguf-connector", "text-generation"]
llama_model_loader: - kv  12:                          llama.block_count u32              = 16
llama_model_loader: - kv  13:                       llama.context_length u32              = 131072
llama_model_loader: - kv  14:                     llama.embedding_length u32              = 2048
llama_model_loader: - kv  15:                  llama.feed_forward_length u32              = 8192
llama_model_loader: - kv  16:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv  17:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  18:                       llama.rope.freq_base f32              = 500000.000000
llama_model_loader: - kv  19:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  20:                 llama.attention.key_length u32              = 64
llama_model_loader: - kv  21:               llama.attention.value_length u32              = 64
llama_model_loader: - kv  22:                          general.file_type u32              = 0
llama_model_loader: - kv  23:                           llama.vocab_size u32              = 128256
llama_model_loader: - kv  24:                 llama.rope.dimension_count u32              = 64
llama_model_loader: - kv  25:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  26:                         tokenizer.ggml.pre str              = llama-bpe
llama_model_loader: - kv  27:                      tokenizer.ggml.tokens arr[str,128256]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  28:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  29:                      tokenizer.ggml.merges arr[str,280147]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv  30:                tokenizer.ggml.bos_token_id u32              = 128000
llama_model_loader: - kv  31:                tokenizer.ggml.eos_token_id u32              = 128001
llama_model_loader: - kv  32:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  147 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = all F32
print_info: file size   = 4.60 GiB (32.00 BPW)
load: special tokens cache size = 256
load: token to piece cache size = 0.7999 MB
print_info: arch             = llama
print_info: vocab_only       = 0
print_info: n_ctx_train      = 131072
print_info: n_embd           = 2048
print_info: n_layer          = 16
print_info: n_head           = 32
print_info: n_head_kv        = 8
print_info: n_rot            = 64
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 64
print_info: n_embd_head_v    = 64
print_info: n_gqa            = 4
print_info: n_embd_k_gqa     = 512
print_info: n_embd_v_gqa     = 512
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 8192
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 0
print_info: rope scaling     = linear
print_info: freq_base_train  = 500000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 131072
print_info: rope_finetuned   = unknown
print_info: ssm_d_conv       = 0
print_info: ssm_d_inner      = 0
print_info: ssm_d_state      = 0
print_info: ssm_dt_rank      = 0
print_info: ssm_dt_b_c_rms   = 0
print_info: model type       = 1B
print_info: model params     = 1.24 B
print_info: general.name     = Llama 3.2 1b
print_info: vocab type       = BPE
print_info: n_vocab          = 128256
print_info: n_merges         = 280147
print_info: BOS token        = 128000 '<|begin_of_text|>'
print_info: EOS token        = 128001 '<|end_of_text|>'
print_info: EOT token        = 128009 '<|eot_id|>'
print_info: EOM token        = 128008 '<|eom_id|>'
print_info: LF token         = 198 'Ċ'
print_info: EOG token        = 128001 '<|end_of_text|>'
print_info: EOG token        = 128008 '<|eom_id|>'
print_info: EOG token        = 128009 '<|eot_id|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: offloading 16 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 17/17 layers to GPU
load_tensors:    CUDA_Host model buffer size =  1002.00 MiB
load_tensors:        CUDA0 model buffer size =  2088.14 MiB
load_tensors:        CUDA1 model buffer size =  2626.12 MiB
..................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 512
llama_context: n_ctx_per_seq = 512
llama_context: n_batch       = 512
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 0
llama_context: freq_base     = 500000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (512) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
llama_context:  CUDA_Host  output buffer size =     0.49 MiB
llama_kv_cache_unified:      CUDA0 KV buffer size =    18.00 MiB
llama_kv_cache_unified:      CUDA1 KV buffer size =    14.00 MiB
llama_kv_cache_unified: size =   32.00 MiB (   512 cells,  16 layers,  1 seqs), K (f32):   16.00 MiB, V (f32):   16.00 MiB
llama_context: pipeline parallelism enabled (n_copies=4)
llama_context:      CUDA0 compute buffer size =    68.01 MiB
llama_context:      CUDA1 compute buffer size =   290.52 MiB
llama_context:  CUDA_Host compute buffer size =     8.02 MiB
llama_context: graph nodes  = 582
llama_context: graph splits = 3
common_init_from_params: setting dry_penalty_last_n to ctx_size = 512
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)

system_info: n_threads = 128 (n_threads_batch = 128) / 256 | CUDA : ARCHS = 500,610,700,750,800,860,890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
output_reserve: reallocating output buffer from size 0.49 MiB to 250.50 MiB
/home/adonay/dev/sentinel/llama.cpp/ggml/src/ggml-backend.cpp:750: pre-allocated tensor (AdamW step for output_norm.weight) in a buffer (CUDA1) that cannot run the operation (OPT_STEP_ADAMW)
[New LWP 607561]
[New LWP 607562]
[New LWP 607563]
[New LWP 607564]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f54391d242f in __GI___wait4 (pid=607565, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30      ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory.
#0  0x00007f54391d242f in __GI___wait4 (pid=607565, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30      in ../sysdeps/unix/sysv/linux/wait4.c
#1  0x00007f543975b221 in ggml_print_backtrace () at /home/adonay/dev/sentinel/llama.cpp/ggml/src/ggml.c:199
199             waitpid(child_pid, NULL, 0);
#2  0x00007f543975b351 in ggml_abort (file=0x7f54397d7450 "/home/adonay/dev/sentinel/llama.cpp/ggml/src/ggml-backend.cpp", line=750, fmt=0x7f54397d78a8 "pre-allocated tensor (%s) in a buffer (%s) that cannot run the operation (%s)") at /home/adonay/dev/sentinel/llama.cpp/ggml/src/ggml.c:220
220         ggml_print_backtrace();
#3  0x00007f543977460d in ggml_backend_sched_backend_id_from_cur (sched=0x55f1d1857d60, tensor=0x7f4fe2442a00) at /home/adonay/dev/sentinel/llama.cpp/ggml/src/ggml-backend.cpp:750
750             GGML_ABORT("pre-allocated tensor (%s) in a buffer (%s) that cannot run the operation (%s)", tensor->name, ggml_backend_buffer_name(buffer), ggml_op_name(tensor->op));
#4  0x00007f5439774f81 in ggml_backend_sched_split_graph (sched=0x55f1d1857d60, graph=0x7f4fe203e6d0) at /home/adonay/dev/sentinel/llama.cpp/ggml/src/ggml-backend.cpp:899
899                 *node_backend_id = ggml_backend_sched_backend_id_from_cur(sched, node);
#5  0x00007f5439778316 in ggml_backend_sched_alloc_graph (sched=0x55f1d1857d60, graph=0x7f4fe203e6d0) at /home/adonay/dev/sentinel/llama.cpp/ggml/src/ggml-backend.cpp:1568
1568        ggml_backend_sched_split_graph(sched, graph);
#6  0x00007f543977d35c in ggml_opt_alloc (opt_ctx=0x55f1da695620, backward=true) at /home/adonay/dev/sentinel/llama.cpp/ggml/src/ggml-opt.cpp:750
750         ggml_backend_sched_alloc_graph(opt_ctx->backend_sched, opt_ctx->allocated_graph_copy);
#7  0x00007f544b15656a in llama_context::opt_epoch_iter (this=0x55f1cfee2fe0, dataset=0x55f1d4cbba50, result=0x55f1da696f40, tokens=std::vector of length 512, capacity 512 = {...}, labels_sparse=std::vector of length 512, capacity 512 = {...}, batch=..., callback=0x7f543977dff9 <ggml_opt_epoch_callback_progress_bar(bool, ggml_opt_context_t, ggml_opt_dataset_t, ggml_opt_result_t, int64_t, int64_t, int64_t)>, train=true, idata_in_loop=0, ndata_in_loop=1070, t_loop_start=161701219169) at /home/adonay/dev/sentinel/llama.cpp/src/llama-context.cpp:2082
2082                ggml_opt_alloc(opt_ctx, train);
#8  0x00007f544b156b2b in llama_context::opt_epoch (this=0x55f1cfee2fe0, dataset=0x55f1d4cbba50, result_train=0x55f1da696f40, result_eval=0x55f1da699bc0, idata_split=1070, callback_train=0x7f543977dff9 <ggml_opt_epoch_callback_progress_bar(bool, ggml_opt_context_t, ggml_opt_dataset_t, ggml_opt_result_t, int64_t, int64_t, int64_t)>, callback_eval=0x7f543977dff9 <ggml_opt_epoch_callback_progress_bar(bool, ggml_opt_context_t, ggml_opt_dataset_t, ggml_opt_result_t, int64_t, int64_t, int64_t)>) at /home/adonay/dev/sentinel/llama.cpp/src/llama-context.cpp:2137
2137            opt_epoch_iter(dataset, result_train, tokens, labels_sparse, batch,
#9  0x00007f544b1586f4 in llama_opt_epoch (ctx=0x55f1cfee2fe0, dataset=0x55f1d4cbba50, result_train=0x55f1da696f40, result_eval=0x55f1da699bc0, idata_split=1070, callback_train=0x7f543977dff9 <ggml_opt_epoch_callback_progress_bar(bool, ggml_opt_context_t, ggml_opt_dataset_t, ggml_opt_result_t, int64_t, int64_t, int64_t)>, callback_eval=0x7f543977dff9 <ggml_opt_epoch_callback_progress_bar(bool, ggml_opt_context_t, ggml_opt_dataset_t, ggml_opt_result_t, int64_t, int64_t, int64_t)>) at /home/adonay/dev/sentinel/llama.cpp/src/llama-context.cpp:2838
2838        ctx->opt_epoch(
#10 0x000055f1aea6abd1 in main (argc=13, argv=0x7ffd172199b8) at /home/adonay/dev/sentinel/llama.cpp/examples/training/finetune.cpp:81
81              llama_opt_epoch(ctx.get(), dataset, result_train, result_eval, idata_split,
[Inferior 1 (process 607560) detached]
Aborted (core dumped)

[killjaqular]

Eval bug seemed the most appropriate type for this problem.
If I am mistaken, please refer me to the correct form.

[killjaqular]

finetune works with:

./build/bin/llama-finetune --file datasets/wikitext-2-raw/wiki.test.raw -ngl 999 --model models/${model_name}-${quantization}.gguf -c 512 -b 512 -ub 512 --device none

However, llama.cpp/examples/training:7:**For CUDA training, use the maximum number of GPU layers.** implies ability to train on GPU hardware.

[lexasub]

May be this

[lexasub]

we need some initialisation of memory (may be)

[lexasub]

in this case models is have no cache

    case LLM_ARCH_BERT:
    case LLM_ARCH_JINA_BERT_V2:
    case LLM_ARCH_NOMIC_BERT:
    case LLM_ARCH_NOMIC_BERT_MOE:
    case LLM_ARCH_NEO_BERT:
    case LLM_ARCH_WAVTOKENIZER_DEC:

[lexasub]

[lexasub]

Everything I said above is my opinion. I am not an expert on this issue, I encountered this problem myself when I tried to implement parquet (I am still trying locally, but my strength is running out (I get a similar error as you, yes, I went further and get an error in another place, but this looks more like hacks and I edit my version, not the original finetuning.cpp)). If anyone wants - tag me - I will send my attempts to screw parquet into finetuning (my issue is #14442 )) )

[lexasub]

@JohannesGaessler May be you may solve bug of @killjaqular

[JohannesGaessler]

Maybe but right now I don't have enough time to debug and fix this. I think that as long as the training code is in a state where it's not really usable anyways it's not a priority to have high availability.

[killjaqular]

Maybe but right now I don't have enough time to debug and fix this. I think that as long as the training code is in a state where it's not really usable anyways it's not a priority to have high availability.

That is fair!

However, please note that following all READMEs as written does not yield a favorable result.

In the end, I was able to run finetune.cpp on the CPU with --device none, which is painfully slow (~4.5hrs per epoch for 2 epochs) even though the CPU I have on my server is a beast. So this is functional, though not favorable. Reading examples/training/README.md it seems like training on the GPU should have worked. Specifically when using the same wiki.test.raw file and llama_3.2-1b model at f32 quantization that is in said README.md file.

For now, training on CPU is fine. In the mid-to-far future, I want to investigate this deeper on my home setup and see if I can begin to contribute to LlamaCPP as much as I complain 😆

[killjaqular]

Everything I said above is my opinion. I am not an expert on this issue, I encountered this problem myself when I tried to implement parquet (I am still trying locally, but my strength is running out (I get a similar error as you, yes, I went further and get an error in another place, but this looks more like hacks and I edit my version, not the original finetuning.cpp)). If anyone wants - tag me - I will send my attempts to screw parquet into finetuning (my issue is #14442 )) )

Thank you for your effort! I won't dive into this just yet, as training on CPU will allow me to continue with my work. But I will revisit this later in a few months. Thank you for your selfless help!

[killjaqular]

@JohannesGaessler If available, would you mind sharing your results of following the documentation at examples/training/README.md?

I followed the README.md exactly, and found my model to 1. Have its loss increase over training, and 2. Have the model's perplexity increase before and after training.

During training:

some_user@some_hardware:~/dev/some_project/llama.cpp$ ./build/bin/llama-finetune --file datasets/wikitext-2-raw/wiki.test.raw -ngl 999 --model models/${model_name}-${quantization}.gguf -c 512 -b 512 -ub 512 --device none
<...>
system_info: n_threads = 128 (n_threads_batch = 128) / 256 | CUDA : ARCHS = 500,610,700,750,800,860,890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
output_reserve: reallocating output buffer from size 0.49 MiB to 250.50 MiB
train: [███████▉] data=0001070/0001070 loss=2.58603±0.01048 acc=48.06±0.07% t=04:32:23 ETA=00:00:00
val:   [███████▉] data=0000057/0000057 loss=2.84143±0.03781 acc=44.53±0.29% t=00:03:35 ETA=00:00:00

train: [███████▉] data=0001070/0001070 loss=2.96127±0.01210 acc=42.47±0.07% t=04:28:27 ETA=00:00:00
val:   [███████▉] data=0000057/0000057 loss=3.70364±0.04238 acc=33.56±0.28% t=00:03:36 ETA=00:00:00

Before training:

some_user@some_hardware:~/dev/sentinel/llama.cpp$ ./build/bin/llama-perplexity --file datasets/wikitext-2-raw/wiki.test.raw -ngl 999 --model models/llama3.2-1b-f32.gguf
<...>
system_info: n_threads = 128 (n_threads_batch = 128) / 256 | CUDA : ARCHS = 500,610,700,750,800,860,890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
perplexity: tokenizing the input ..
perplexity: tokenization took 1740.28 ms
perplexity: calculating perplexity over 564 chunks, n_ctx=512, batch_size=2048, n_seq=4
output_reserve: reallocating output buffer from size 1.96 MiB to 501.00 MiB
perplexity: 0.53 seconds per pass - ETA 1.25 minutes
<...>
Final estimate: PPL = 10.1894 +/- 0.07044

llama_perf_context_print:        load time =    3256.46 ms
llama_perf_context_print: prompt eval time =   19340.27 ms / 288768 tokens (    0.07 ms per token, 14930.92 tokens per second)
llama_perf_context_print:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_perf_context_print:       total time =   35383.02 ms / 288769 tokens

After training:

some_user@some_hardware:~/dev/sentinel/llama.cpp$ ./build/bin/llama-perplexity --file datasets/wikitext-2-raw/wiki.test.raw -ngl 999 --model finetuned-model.gguf
<...>
system_info: n_threads = 128 (n_threads_batch = 128) / 256 | CUDA : ARCHS = 500,610,700,750,800,860,890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
perplexity: tokenizing the input ..
perplexity: tokenization took 1718.49 ms
perplexity: calculating perplexity over 564 chunks, n_ctx=512, batch_size=2048, n_seq=4
output_reserve: reallocating output buffer from size 1.96 MiB to 501.00 MiB
perplexity: 0.80 seconds per pass - ETA 1.87 minutes
<...>
Final estimate: PPL = 27.7374 +/- 0.21937

llama_perf_context_print:        load time =   10512.62 ms
llama_perf_context_print: prompt eval time =   20023.66 ms / 288768 tokens (    0.07 ms per token, 14421.34 tokens per second)
llama_perf_context_print:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_perf_context_print:       total time =   35995.21 ms / 288769 tokens

Has anyone else been able to follow the steps described in examples/training/README.md and found their Perplexity value to decrease, as mentioned in the same README.md?

[killjaqular]

@lexasub Would you mind trying to reproduce examples/training/README.md and posting your results please? I would appreciate it tremendously! 🙏

[lexasub]

@lexasub Would you mind trying to reproduce examples/training/README.md and posting your results please? I would appreciate it tremendously! 🙏

I think I'll be able to test the example from the Readme this week.