[1/N] Android JNI llama cache temperature in class

[kirklandsign]

So far, for LLM, we can use the new config, with cached temperature from ctor. For llava, we use the old workflow.

Test: instrumentation test

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10287

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• [Infra] Jobs got intermittently cancelled/fail midway checkout

❌ 1 New Failure, 2 Cancelled Jobs, 28 Unrelated Failures

As of commit c11f8dc with merge base d82e852 ():

NEW FAILURE - The following job has failed:

• Check Labels / Check labels (gh)

CANCELLED JOBS - The following jobs were cancelled. Please retry:

• Lint / android-java-format / linux-job (gh)
• pull / android / build-llm-demo / linux-job (gh)

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• pull / test-llama-runner-linux (bf16, custom, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh) (similar failure)
• pull / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh) (similar failure)
• pull / test-models-linux (add_mul, portable, linux.2xlarge) / linux-job (gh) (matched linux rule in flaky-rules.json)
  The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
• pull / test-models-linux (ic3, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh) (similar failure)
• pull / test-models-linux (linear, portable, linux.2xlarge) / linux-job (gh) (similar failure)
• pull / test-models-linux (mobilebert, portable, linux.2xlarge) / linux-job (gh) (similar failure)
• pull / test-models-linux (mobilebert, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh) (similar failure)
• pull / test-models-linux (resnet50, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh) (similar failure)
• pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh) (similar failure)
• pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh) (similar failure)
• pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, linux.2xlarge, executorch-u... / linux-job (gh) (similar failure)
• pull / test-openvino-linux / linux-job (gh) (similar failure)
• pull / test-pybind-build-linux / linux-job (gh) (similar failure)
• pull / test-setup-linux-gcc / linux-job (gh) (similar failure)
• pull / unittest-arm / linux-job (gh) (similar failure)
• pull / unittest-editable / linux / linux-job (gh) (similar failure)

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Build documentation / check-links (gh) (trunk failure)
  Process completed with exit code 1.
• Build documentation / check-urls (gh) (trunk failure)
  Process completed with exit code 1.
• pull / test-binary-size-linux / linux-job (gh) (trunk failure)
• pull / test-binary-size-linux-gcc / linux-job (gh) (trunk failure)
• pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh) (trunk failure)
• pull / test-models-linux (add, portable, linux.2xlarge) / linux-job (gh) (trunk failure)
• pull / test-models-linux (emformer_transcribe, portable, linux.2xlarge) / linux-job (gh) (trunk failure)
• pull / test-models-linux (emformer_transcribe, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh) (trunk failure)
• pull / test-models-linux (mv2, portable, linux.2xlarge) / linux-job (gh) (trunk failure)
• pull / test-models-linux-basic (vit, portable, cmake, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh) (trunk failure)
• pull / test-static-llama-qnn-linux / linux-job (gh) (trunk failure)
• pull / unittest-buck / linux / linux-job (gh) (trunk failure)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Copilot]

Pull Request Overview

This PR introduces caching for the temperature parameter in the Android JNI layer for llama models. The changes include adding a new member variable to store temperature, initializing that variable in the constructor, and updating the generation configuration to use the cached value.

• Added a new member variable (temperature_) to cache the temperature.
• Assigned the temperature parameter to temperature_ in the constructor.
• Updated the generation configuration to use temperature_.

Comments suppressed due to low confidence (1)

extension/android/jni/jni_layer_llama.cpp:186

• Please verify that removing the temperature parameter from the MTKLlamaRunner constructor is intentional. If the temperature was previously required by the runner, additional changes in MTKLlamaRunner or its usage may be needed.

tokenizer_path->toStdString().c_str());

[github-actions]

This PR needs a release notes: label

If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.