How to train a layer before the first layer of existing models (e.g. gemma) using axolotl?

[GeorgeCarpenter]

In our cases, we wanna train a layer before the first layer of existing LLM models (e.g. gemma) using axolotl. The new added layer is used to process the image. What can we do to implement this?

[NanoCode012]

Hey, do you know the layer path?

We support freezing the model for FFT and unfreezing certain params via regex:

axolotl/examples/mistral/bigstral-ds-zero3.yaml

Lines 10 to 16 in 4a8af60

	unfrozen_parameters:
	- ^lm_head.weight$
	- ^model.embed_tokens.weight$
	- model.layers.4[4-9]+.block_sparse_moe.gate
	- model.layers.4[4-9]+.block_sparse_moe.experts
	- model.layers.5[0-5]+.block_sparse_moe.gate
	- model.layers.5[0-5]+.block_sparse_moe.experts

[GeorgeCarpenter]

What is layer path？ Should we write a new model that contains the added layer and the original model using torch.nn.Module?

[GeorgeCarpenter]

Thanks. Our model is based on Qwen-VL2. The only difference between our model and Qwen-VL2 is that we add a conv layer before the original Qwen-VL2 in our model.The following is the layer path:

model
model.visual
model.visual.patch_embed
model.visual.patch_embed.proj
model.visual.rotary_pos_emb
model.visual.blocks
model.visual.blocks.0
model.visual.blocks.0.norm1
model.visual.blocks.0.norm2
model.visual.blocks.0.attn
model.visual.blocks.0.attn.qkv
model.visual.blocks.0.attn.proj
model.visual.blocks.0.mlp
model.visual.blocks.0.mlp.fc1
model.visual.blocks.0.mlp.act
model.visual.blocks.0.mlp.fc2
model.visual.blocks.1
model.visual.blocks.1.norm1
model.visual.blocks.1.norm2
model.visual.blocks.1.attn
model.visual.blocks.1.attn.qkv
model.visual.blocks.1.attn.proj
model.visual.blocks.1.mlp
model.visual.blocks.1.mlp.fc1
model.visual.blocks.1.mlp.act
model.visual.blocks.1.mlp.fc2
model.visual.blocks.2
model.visual.blocks.2.norm1
model.visual.blocks.2.norm2
model.visual.blocks.2.attn
model.visual.blocks.2.attn.qkv
model.visual.blocks.2.attn.proj
model.visual.blocks.2.mlp
model.visual.blocks.2.mlp.fc1
model.visual.blocks.2.mlp.act
model.visual.blocks.2.mlp.fc2
model.visual.blocks.3
model.visual.blocks.3.norm1
model.visual.blocks.3.norm2
model.visual.blocks.3.attn
model.visual.blocks.3.attn.qkv
model.visual.blocks.3.attn.proj
model.visual.blocks.3.mlp
model.visual.blocks.3.mlp.fc1
model.visual.blocks.3.mlp.act
model.visual.blocks.3.mlp.fc2
model.visual.blocks.4
model.visual.blocks.4.norm1
model.visual.blocks.4.norm2
model.visual.blocks.4.attn
model.visual.blocks.4.attn.qkv
model.visual.blocks.4.attn.proj
model.visual.blocks.4.mlp
model.visual.blocks.4.mlp.fc1
model.visual.blocks.4.mlp.act
model.visual.blocks.4.mlp.fc2
model.visual.blocks.5
model.visual.blocks.5.norm1
model.visual.blocks.5.norm2
model.visual.blocks.5.attn
model.visual.blocks.5.attn.qkv
model.visual.blocks.5.attn.proj
model.visual.blocks.5.mlp
model.visual.blocks.5.mlp.fc1
model.visual.blocks.5.mlp.act
model.visual.blocks.5.mlp.fc2
model.visual.blocks.6
model.visual.blocks.6.norm1
model.visual.blocks.6.norm2
model.visual.blocks.6.attn
model.visual.blocks.6.attn.qkv
model.visual.blocks.6.attn.proj
model.visual.blocks.6.mlp
model.visual.blocks.6.mlp.fc1
model.visual.blocks.6.mlp.act
model.visual.blocks.6.mlp.fc2
model.visual.blocks.7
model.visual.blocks.7.norm1
model.visual.blocks.7.norm2
model.visual.blocks.7.attn
model.visual.blocks.7.attn.qkv
model.visual.blocks.7.attn.proj
model.visual.blocks.7.mlp
model.visual.blocks.7.mlp.fc1
model.visual.blocks.7.mlp.act
model.visual.blocks.7.mlp.fc2
model.visual.blocks.8
model.visual.blocks.8.norm1
model.visual.blocks.8.norm2
model.visual.blocks.8.attn
model.visual.blocks.8.attn.qkv
model.visual.blocks.8.attn.proj
model.visual.blocks.8.mlp
model.visual.blocks.8.mlp.fc1
model.visual.blocks.8.mlp.act
model.visual.blocks.8.mlp.fc2
model.visual.blocks.9
model.visual.blocks.9.norm1
model.visual.blocks.9.norm2
model.visual.blocks.9.attn
model.visual.blocks.9.attn.qkv
model.visual.blocks.9.attn.proj
model.visual.blocks.9.mlp
model.visual.blocks.9.mlp.fc1
model.visual.blocks.9.mlp.act
model.visual.blocks.9.mlp.fc2
model.visual.blocks.10
model.visual.blocks.10.norm1
model.visual.blocks.10.norm2
model.visual.blocks.10.attn
model.visual.blocks.10.attn.qkv
model.visual.blocks.10.attn.proj
model.visual.blocks.10.mlp
model.visual.blocks.10.mlp.fc1
model.visual.blocks.10.mlp.act
model.visual.blocks.10.mlp.fc2
model.visual.blocks.11
model.visual.blocks.11.norm1
model.visual.blocks.11.norm2
model.visual.blocks.11.attn
model.visual.blocks.11.attn.qkv
model.visual.blocks.11.attn.proj
model.visual.blocks.11.mlp
model.visual.blocks.11.mlp.fc1
model.visual.blocks.11.mlp.act
model.visual.blocks.11.mlp.fc2
model.visual.blocks.12
model.visual.blocks.12.norm1
model.visual.blocks.12.norm2
model.visual.blocks.12.attn
model.visual.blocks.12.attn.qkv
model.visual.blocks.12.attn.proj
model.visual.blocks.12.mlp
model.visual.blocks.12.mlp.fc1
model.visual.blocks.12.mlp.act
model.visual.blocks.12.mlp.fc2
model.visual.blocks.13
model.visual.blocks.13.norm1
model.visual.blocks.13.norm2
model.visual.blocks.13.attn
model.visual.blocks.13.attn.qkv
model.visual.blocks.13.attn.proj
model.visual.blocks.13.mlp
model.visual.blocks.13.mlp.fc1
model.visual.blocks.13.mlp.act
model.visual.blocks.13.mlp.fc2
model.visual.blocks.14
model.visual.blocks.14.norm1
model.visual.blocks.14.norm2
model.visual.blocks.14.attn
model.visual.blocks.14.attn.qkv
model.visual.blocks.14.attn.proj
model.visual.blocks.14.mlp
model.visual.blocks.14.mlp.fc1
model.visual.blocks.14.mlp.act
model.visual.blocks.14.mlp.fc2
model.visual.blocks.15
model.visual.blocks.15.norm1
model.visual.blocks.15.norm2
model.visual.blocks.15.attn
model.visual.blocks.15.attn.qkv
model.visual.blocks.15.attn.proj
model.visual.blocks.15.mlp
model.visual.blocks.15.mlp.fc1
model.visual.blocks.15.mlp.act
model.visual.blocks.15.mlp.fc2
model.visual.blocks.16
model.visual.blocks.16.norm1
model.visual.blocks.16.norm2
model.visual.blocks.16.attn
model.visual.blocks.16.attn.qkv
model.visual.blocks.16.attn.proj
model.visual.blocks.16.mlp
model.visual.blocks.16.mlp.fc1
model.visual.blocks.16.mlp.act
model.visual.blocks.16.mlp.fc2
model.visual.blocks.17
model.visual.blocks.17.norm1
model.visual.blocks.17.norm2
model.visual.blocks.17.attn
model.visual.blocks.17.attn.qkv
model.visual.blocks.17.attn.proj
model.visual.blocks.17.mlp
model.visual.blocks.17.mlp.fc1
model.visual.blocks.17.mlp.act
model.visual.blocks.17.mlp.fc2
model.visual.blocks.18
model.visual.blocks.18.norm1
model.visual.blocks.18.norm2
model.visual.blocks.18.attn
model.visual.blocks.18.attn.qkv
model.visual.blocks.18.attn.proj
model.visual.blocks.18.mlp
model.visual.blocks.18.mlp.fc1
model.visual.blocks.18.mlp.act
model.visual.blocks.18.mlp.fc2
model.visual.blocks.19
model.visual.blocks.19.norm1
model.visual.blocks.19.norm2
model.visual.blocks.19.attn
model.visual.blocks.19.attn.qkv
model.visual.blocks.19.attn.proj
model.visual.blocks.19.mlp
model.visual.blocks.19.mlp.fc1
model.visual.blocks.19.mlp.act
model.visual.blocks.19.mlp.fc2
model.visual.blocks.20
model.visual.blocks.20.norm1
model.visual.blocks.20.norm2
model.visual.blocks.20.attn
model.visual.blocks.20.attn.qkv
model.visual.blocks.20.attn.proj
model.visual.blocks.20.mlp
model.visual.blocks.20.mlp.fc1
model.visual.blocks.20.mlp.act
model.visual.blocks.20.mlp.fc2
model.visual.blocks.21
model.visual.blocks.21.norm1
model.visual.blocks.21.norm2
model.visual.blocks.21.attn
model.visual.blocks.21.attn.qkv
model.visual.blocks.21.attn.proj
model.visual.blocks.21.mlp
model.visual.blocks.21.mlp.fc1
model.visual.blocks.21.mlp.act
model.visual.blocks.21.mlp.fc2
model.visual.blocks.22
model.visual.blocks.22.norm1
model.visual.blocks.22.norm2
model.visual.blocks.22.attn
model.visual.blocks.22.attn.qkv
model.visual.blocks.22.attn.proj
model.visual.blocks.22.mlp
model.visual.blocks.22.mlp.fc1
model.visual.blocks.22.mlp.act
model.visual.blocks.22.mlp.fc2
model.visual.blocks.23
model.visual.blocks.23.norm1
model.visual.blocks.23.norm2
model.visual.blocks.23.attn
model.visual.blocks.23.attn.qkv
model.visual.blocks.23.attn.proj
model.visual.blocks.23.mlp
model.visual.blocks.23.mlp.fc1
model.visual.blocks.23.mlp.act
model.visual.blocks.23.mlp.fc2
model.visual.blocks.24
model.visual.blocks.24.norm1
model.visual.blocks.24.norm2
model.visual.blocks.24.attn
model.visual.blocks.24.attn.qkv
model.visual.blocks.24.attn.proj
model.visual.blocks.24.mlp
model.visual.blocks.24.mlp.fc1
model.visual.blocks.24.mlp.act
model.visual.blocks.24.mlp.fc2
model.visual.blocks.25
model.visual.blocks.25.norm1
model.visual.blocks.25.norm2
model.visual.blocks.25.attn
model.visual.blocks.25.attn.qkv
model.visual.blocks.25.attn.proj
model.visual.blocks.25.mlp
model.visual.blocks.25.mlp.fc1
model.visual.blocks.25.mlp.act
model.visual.blocks.25.mlp.fc2
model.visual.blocks.26
model.visual.blocks.26.norm1
model.visual.blocks.26.norm2
model.visual.blocks.26.attn
model.visual.blocks.26.attn.qkv
model.visual.blocks.26.attn.proj
model.visual.blocks.26.mlp
model.visual.blocks.26.mlp.fc1
model.visual.blocks.26.mlp.act
model.visual.blocks.26.mlp.fc2
model.visual.blocks.27
model.visual.blocks.27.norm1
model.visual.blocks.27.norm2
model.visual.blocks.27.attn
model.visual.blocks.27.attn.qkv
model.visual.blocks.27.attn.proj
model.visual.blocks.27.mlp
model.visual.blocks.27.mlp.fc1
model.visual.blocks.27.mlp.act
model.visual.blocks.27.mlp.fc2
model.visual.blocks.28
model.visual.blocks.28.norm1
model.visual.blocks.28.norm2
model.visual.blocks.28.attn
model.visual.blocks.28.attn.qkv
model.visual.blocks.28.attn.proj
model.visual.blocks.28.mlp
model.visual.blocks.28.mlp.fc1
model.visual.blocks.28.mlp.act
model.visual.blocks.28.mlp.fc2
model.visual.blocks.29
model.visual.blocks.29.norm1
model.visual.blocks.29.norm2
model.visual.blocks.29.attn
model.visual.blocks.29.attn.qkv
model.visual.blocks.29.attn.proj
model.visual.blocks.29.mlp
model.visual.blocks.29.mlp.fc1
model.visual.blocks.29.mlp.act
model.visual.blocks.29.mlp.fc2
model.visual.blocks.30
model.visual.blocks.30.norm1
model.visual.blocks.30.norm2
model.visual.blocks.30.attn
model.visual.blocks.30.attn.qkv
model.visual.blocks.30.attn.proj
model.visual.blocks.30.mlp
model.visual.blocks.30.mlp.fc1
model.visual.blocks.30.mlp.act
model.visual.blocks.30.mlp.fc2
model.visual.blocks.31
model.visual.blocks.31.norm1
model.visual.blocks.31.norm2
model.visual.blocks.31.attn
model.visual.blocks.31.attn.qkv
model.visual.blocks.31.attn.proj
model.visual.blocks.31.mlp
model.visual.blocks.31.mlp.fc1
model.visual.blocks.31.mlp.act
model.visual.blocks.31.mlp.fc2
model.visual.merger
model.visual.merger.ln_q
model.visual.merger.mlp
model.visual.merger.mlp.0
model.visual.merger.mlp.1
model.visual.merger.mlp.2
model.model
model.model.embed_tokens
model.model.layers
model.model.layers.0
model.model.layers.0.self_attn
model.model.layers.0.self_attn.q_proj
model.model.layers.0.self_attn.k_proj
model.model.layers.0.self_attn.v_proj
model.model.layers.0.self_attn.o_proj
model.model.layers.0.self_attn.rotary_emb
model.model.layers.0.mlp
model.model.layers.0.mlp.gate_proj
model.model.layers.0.mlp.up_proj
model.model.layers.0.mlp.down_proj
model.model.layers.0.mlp.act_fn
model.model.layers.0.input_layernorm
model.model.layers.0.post_attention_layernorm
model.model.layers.1
model.model.layers.1.self_attn
model.model.layers.1.self_attn.q_proj
model.model.layers.1.self_attn.k_proj
model.model.layers.1.self_attn.v_proj
model.model.layers.1.self_attn.o_proj
model.model.layers.1.self_attn.rotary_emb
model.model.layers.1.mlp
model.model.layers.1.mlp.gate_proj
model.model.layers.1.mlp.up_proj
model.model.layers.1.mlp.down_proj
model.model.layers.1.mlp.act_fn
model.model.layers.1.input_layernorm
model.model.layers.1.post_attention_layernorm
model.model.layers.2
model.model.layers.2.self_attn
model.model.layers.2.self_attn.q_proj
model.model.layers.2.self_attn.k_proj
model.model.layers.2.self_attn.v_proj
model.model.layers.2.self_attn.o_proj
model.model.layers.2.self_attn.rotary_emb
model.model.layers.2.mlp
model.model.layers.2.mlp.gate_proj
model.model.layers.2.mlp.up_proj
model.model.layers.2.mlp.down_proj
model.model.layers.2.mlp.act_fn
model.model.layers.2.input_layernorm
model.model.layers.2.post_attention_layernorm
model.model.layers.3
model.model.layers.3.self_attn
model.model.layers.3.self_attn.q_proj
model.model.layers.3.self_attn.k_proj
model.model.layers.3.self_attn.v_proj
model.model.layers.3.self_attn.o_proj
model.model.layers.3.self_attn.rotary_emb
model.model.layers.3.mlp
model.model.layers.3.mlp.gate_proj
model.model.layers.3.mlp.up_proj
model.model.layers.3.mlp.down_proj
model.model.layers.3.mlp.act_fn
model.model.layers.3.input_layernorm
model.model.layers.3.post_attention_layernorm
model.model.layers.4
model.model.layers.4.self_attn
model.model.layers.4.self_attn.q_proj
model.model.layers.4.self_attn.k_proj
model.model.layers.4.self_attn.v_proj
model.model.layers.4.self_attn.o_proj
model.model.layers.4.self_attn.rotary_emb
model.model.layers.4.mlp
model.model.layers.4.mlp.gate_proj
model.model.layers.4.mlp.up_proj
model.model.layers.4.mlp.down_proj
model.model.layers.4.mlp.act_fn
model.model.layers.4.input_layernorm
model.model.layers.4.post_attention_layernorm
model.model.layers.5
model.model.layers.5.self_attn
model.model.layers.5.self_attn.q_proj
model.model.layers.5.self_attn.k_proj
model.model.layers.5.self_attn.v_proj
model.model.layers.5.self_attn.o_proj
model.model.layers.5.self_attn.rotary_emb
model.model.layers.5.mlp
model.model.layers.5.mlp.gate_proj
model.model.layers.5.mlp.up_proj
model.model.layers.5.mlp.down_proj
model.model.layers.5.mlp.act_fn
model.model.layers.5.input_layernorm
model.model.layers.5.post_attention_layernorm
model.model.layers.6
model.model.layers.6.self_attn
model.model.layers.6.self_attn.q_proj
model.model.layers.6.self_attn.k_proj
model.model.layers.6.self_attn.v_proj
model.model.layers.6.self_attn.o_proj
model.model.layers.6.self_attn.rotary_emb
model.model.layers.6.mlp
model.model.layers.6.mlp.gate_proj
model.model.layers.6.mlp.up_proj
model.model.layers.6.mlp.down_proj
model.model.layers.6.mlp.act_fn
model.model.layers.6.input_layernorm
model.model.layers.6.post_attention_layernorm
model.model.layers.7
model.model.layers.7.self_attn
model.model.layers.7.self_attn.q_proj
model.model.layers.7.self_attn.k_proj
model.model.layers.7.self_attn.v_proj
model.model.layers.7.self_attn.o_proj
model.model.layers.7.self_attn.rotary_emb
model.model.layers.7.mlp
model.model.layers.7.mlp.gate_proj
model.model.layers.7.mlp.up_proj
model.model.layers.7.mlp.down_proj
model.model.layers.7.mlp.act_fn
model.model.layers.7.input_layernorm
model.model.layers.7.post_attention_layernorm
model.model.layers.8
model.model.layers.8.self_attn
model.model.layers.8.self_attn.q_proj
model.model.layers.8.self_attn.k_proj
model.model.layers.8.self_attn.v_proj
model.model.layers.8.self_attn.o_proj
model.model.layers.8.self_attn.rotary_emb
model.model.layers.8.mlp
model.model.layers.8.mlp.gate_proj
model.model.layers.8.mlp.up_proj
model.model.layers.8.mlp.down_proj
model.model.layers.8.mlp.act_fn
model.model.layers.8.input_layernorm
model.model.layers.8.post_attention_layernorm
model.model.layers.9
model.model.layers.9.self_attn
model.model.layers.9.self_attn.q_proj
model.model.layers.9.self_attn.k_proj
model.model.layers.9.self_attn.v_proj
model.model.layers.9.self_attn.o_proj
model.model.layers.9.self_attn.rotary_emb
model.model.layers.9.mlp
model.model.layers.9.mlp.gate_proj
model.model.layers.9.mlp.up_proj
model.model.layers.9.mlp.down_proj
model.model.layers.9.mlp.act_fn
model.model.layers.9.input_layernorm
model.model.layers.9.post_attention_layernorm
model.model.layers.10
model.model.layers.10.self_attn
model.model.layers.10.self_attn.q_proj
model.model.layers.10.self_attn.k_proj
model.model.layers.10.self_attn.v_proj
model.model.layers.10.self_attn.o_proj
model.model.layers.10.self_attn.rotary_emb
model.model.layers.10.mlp
model.model.layers.10.mlp.gate_proj
model.model.layers.10.mlp.up_proj
model.model.layers.10.mlp.down_proj
model.model.layers.10.mlp.act_fn
model.model.layers.10.input_layernorm
model.model.layers.10.post_attention_layernorm
model.model.layers.11
model.model.layers.11.self_attn
model.model.layers.11.self_attn.q_proj
model.model.layers.11.self_attn.k_proj
model.model.layers.11.self_attn.v_proj
model.model.layers.11.self_attn.o_proj
model.model.layers.11.self_attn.rotary_emb
model.model.layers.11.mlp
model.model.layers.11.mlp.gate_proj
model.model.layers.11.mlp.up_proj
model.model.layers.11.mlp.down_proj
model.model.layers.11.mlp.act_fn
model.model.layers.11.input_layernorm
model.model.layers.11.post_attention_layernorm
model.model.layers.12
model.model.layers.12.self_attn
model.model.layers.12.self_attn.q_proj
model.model.layers.12.self_attn.k_proj
model.model.layers.12.self_attn.v_proj
model.model.layers.12.self_attn.o_proj
model.model.layers.12.self_attn.rotary_emb
model.model.layers.12.mlp
model.model.layers.12.mlp.gate_proj
model.model.layers.12.mlp.up_proj
model.model.layers.12.mlp.down_proj
model.model.layers.12.mlp.act_fn
model.model.layers.12.input_layernorm
model.model.layers.12.post_attention_layernorm
model.model.layers.13
model.model.layers.13.self_attn
model.model.layers.13.self_attn.q_proj
model.model.layers.13.self_attn.k_proj
model.model.layers.13.self_attn.v_proj
model.model.layers.13.self_attn.o_proj
model.model.layers.13.self_attn.rotary_emb
model.model.layers.13.mlp
model.model.layers.13.mlp.gate_proj
model.model.layers.13.mlp.up_proj
model.model.layers.13.mlp.down_proj
model.model.layers.13.mlp.act_fn
model.model.layers.13.input_layernorm
model.model.layers.13.post_attention_layernorm
model.model.layers.14
model.model.layers.14.self_attn
model.model.layers.14.self_attn.q_proj
model.model.layers.14.self_attn.k_proj
model.model.layers.14.self_attn.v_proj
model.model.layers.14.self_attn.o_proj
model.model.layers.14.self_attn.rotary_emb
model.model.layers.14.mlp
model.model.layers.14.mlp.gate_proj
model.model.layers.14.mlp.up_proj
model.model.layers.14.mlp.down_proj
model.model.layers.14.mlp.act_fn
model.model.layers.14.input_layernorm
model.model.layers.14.post_attention_layernorm
model.model.layers.15
model.model.layers.15.self_attn
model.model.layers.15.self_attn.q_proj
model.model.layers.15.self_attn.k_proj
model.model.layers.15.self_attn.v_proj
model.model.layers.15.self_attn.o_proj
model.model.layers.15.self_attn.rotary_emb
model.model.layers.15.mlp
model.model.layers.15.mlp.gate_proj
model.model.layers.15.mlp.up_proj
model.model.layers.15.mlp.down_proj
model.model.layers.15.mlp.act_fn
model.model.layers.15.input_layernorm
model.model.layers.15.post_attention_layernorm
model.model.layers.16
model.model.layers.16.self_attn
model.model.layers.16.self_attn.q_proj
model.model.layers.16.self_attn.k_proj
model.model.layers.16.self_attn.v_proj
model.model.layers.16.self_attn.o_proj
model.model.layers.16.self_attn.rotary_emb
model.model.layers.16.mlp
model.model.layers.16.mlp.gate_proj
model.model.layers.16.mlp.up_proj
model.model.layers.16.mlp.down_proj
model.model.layers.16.mlp.act_fn
model.model.layers.16.input_layernorm
model.model.layers.16.post_attention_layernorm
model.model.layers.17
model.model.layers.17.self_attn
model.model.layers.17.self_attn.q_proj
model.model.layers.17.self_attn.k_proj
model.model.layers.17.self_attn.v_proj
model.model.layers.17.self_attn.o_proj
model.model.layers.17.self_attn.rotary_emb
model.model.layers.17.mlp
model.model.layers.17.mlp.gate_proj
model.model.layers.17.mlp.up_proj
model.model.layers.17.mlp.down_proj
model.model.layers.17.mlp.act_fn
model.model.layers.17.input_layernorm
model.model.layers.17.post_attention_layernorm
model.model.layers.18
model.model.layers.18.self_attn
model.model.layers.18.self_attn.q_proj
model.model.layers.18.self_attn.k_proj
model.model.layers.18.self_attn.v_proj
model.model.layers.18.self_attn.o_proj
model.model.layers.18.self_attn.rotary_emb
model.model.layers.18.mlp
model.model.layers.18.mlp.gate_proj
model.model.layers.18.mlp.up_proj
model.model.layers.18.mlp.down_proj
model.model.layers.18.mlp.act_fn
model.model.layers.18.input_layernorm
model.model.layers.18.post_attention_layernorm
model.model.layers.19
model.model.layers.19.self_attn
model.model.layers.19.self_attn.q_proj
model.model.layers.19.self_attn.k_proj
model.model.layers.19.self_attn.v_proj
model.model.layers.19.self_attn.o_proj
model.model.layers.19.self_attn.rotary_emb
model.model.layers.19.mlp
model.model.layers.19.mlp.gate_proj
model.model.layers.19.mlp.up_proj
model.model.layers.19.mlp.down_proj
model.model.layers.19.mlp.act_fn
model.model.layers.19.input_layernorm
model.model.layers.19.post_attention_layernorm
model.model.layers.20
model.model.layers.20.self_attn
model.model.layers.20.self_attn.q_proj
model.model.layers.20.self_attn.k_proj
model.model.layers.20.self_attn.v_proj
model.model.layers.20.self_attn.o_proj
model.model.layers.20.self_attn.rotary_emb
model.model.layers.20.mlp
model.model.layers.20.mlp.gate_proj
model.model.layers.20.mlp.up_proj
model.model.layers.20.mlp.down_proj
model.model.layers.20.mlp.act_fn
model.model.layers.20.input_layernorm
model.model.layers.20.post_attention_layernorm
model.model.layers.21
model.model.layers.21.self_attn
model.model.layers.21.self_attn.q_proj
model.model.layers.21.self_attn.k_proj
model.model.layers.21.self_attn.v_proj
model.model.layers.21.self_attn.o_proj
model.model.layers.21.self_attn.rotary_emb
model.model.layers.21.mlp
model.model.layers.21.mlp.gate_proj
model.model.layers.21.mlp.up_proj
model.model.layers.21.mlp.down_proj
model.model.layers.21.mlp.act_fn
model.model.layers.21.input_layernorm
model.model.layers.21.post_attention_layernorm
model.model.layers.22
model.model.layers.22.self_attn
model.model.layers.22.self_attn.q_proj
model.model.layers.22.self_attn.k_proj
model.model.layers.22.self_attn.v_proj
model.model.layers.22.self_attn.o_proj
model.model.layers.22.self_attn.rotary_emb
model.model.layers.22.mlp
model.model.layers.22.mlp.gate_proj
model.model.layers.22.mlp.up_proj
model.model.layers.22.mlp.down_proj
model.model.layers.22.mlp.act_fn
model.model.layers.22.input_layernorm
model.model.layers.22.post_attention_layernorm
model.model.layers.23
model.model.layers.23.self_attn
model.model.layers.23.self_attn.q_proj
model.model.layers.23.self_attn.k_proj
model.model.layers.23.self_attn.v_proj
model.model.layers.23.self_attn.o_proj
model.model.layers.23.self_attn.rotary_emb
model.model.layers.23.mlp
model.model.layers.23.mlp.gate_proj
model.model.layers.23.mlp.up_proj
model.model.layers.23.mlp.down_proj
model.model.layers.23.mlp.act_fn
model.model.layers.23.input_layernorm
model.model.layers.23.post_attention_layernorm
model.model.layers.24
model.model.layers.24.self_attn
model.model.layers.24.self_attn.q_proj
model.model.layers.24.self_attn.k_proj
model.model.layers.24.self_attn.v_proj
model.model.layers.24.self_attn.o_proj
model.model.layers.24.self_attn.rotary_emb
model.model.layers.24.mlp
model.model.layers.24.mlp.gate_proj
model.model.layers.24.mlp.up_proj
model.model.layers.24.mlp.down_proj
model.model.layers.24.mlp.act_fn
model.model.layers.24.input_layernorm
model.model.layers.24.post_attention_layernorm
model.model.layers.25
model.model.layers.25.self_attn
model.model.layers.25.self_attn.q_proj
model.model.layers.25.self_attn.k_proj
model.model.layers.25.self_attn.v_proj
model.model.layers.25.self_attn.o_proj
model.model.layers.25.self_attn.rotary_emb
model.model.layers.25.mlp
model.model.layers.25.mlp.gate_proj
model.model.layers.25.mlp.up_proj
model.model.layers.25.mlp.down_proj
model.model.layers.25.mlp.act_fn
model.model.layers.25.input_layernorm
model.model.layers.25.post_attention_layernorm
model.model.layers.26
model.model.layers.26.self_attn
model.model.layers.26.self_attn.q_proj
model.model.layers.26.self_attn.k_proj
model.model.layers.26.self_attn.v_proj
model.model.layers.26.self_attn.o_proj
model.model.layers.26.self_attn.rotary_emb
model.model.layers.26.mlp
model.model.layers.26.mlp.gate_proj
model.model.layers.26.mlp.up_proj
model.model.layers.26.mlp.down_proj
model.model.layers.26.mlp.act_fn
model.model.layers.26.input_layernorm
model.model.layers.26.post_attention_layernorm
model.model.layers.27
model.model.layers.27.self_attn
model.model.layers.27.self_attn.q_proj
model.model.layers.27.self_attn.k_proj
model.model.layers.27.self_attn.v_proj
model.model.layers.27.self_attn.o_proj
model.model.layers.27.self_attn.rotary_emb
model.model.layers.27.mlp
model.model.layers.27.mlp.gate_proj
model.model.layers.27.mlp.up_proj
model.model.layers.27.mlp.down_proj
model.model.layers.27.mlp.act_fn
model.model.layers.27.input_layernorm
model.model.layers.27.post_attention_layernorm
model.model.norm
model.model.rotary_emb
model.lm_head
conv

[NanoCode012]

In this case, maybe your regex should only be set to filter for conv?

[GeorgeCarpenter]

If our regex is set to conv, will axolotl only train the specified conv layer?

Can we use axolotl to specify any layers in any models using regex to finetune the specified layers?

[NanoCode012]

1. Yes
2. Any :)