Unexpected Runtime Selection in GAE Standard – Instances Randomly Using EE8 (Jetty 12 + Java 21) Instead of Java 17 (Jetty 9)

[vivekbhatnagar]

One of our application, deployed on Google App Engine (GAE) Standard Environment, is explicitly designed to run on Java 17 with Jetty 9. However, we have observed inconsistent behavior where some instances are randomly being assigned to an EE8 runtime profile (Jetty 12 + Java 21) instead of Java 17 with Jetty 9.
This unexpected transition to Jetty 12 causes multipart request handling failures, leading to the following error:

java.lang.IllegalStateException: No multipart config for servlet
    at org.eclipse.jetty.ee8.nested.Request.getParts(Request.java:1714)
    at org.eclipse.jetty.ee8.nested.Request.getParts(Request.java:1698)
    at org.eclipse.jetty.ee8.nested.Request.getPart(Request.java:1689)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:517)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:584)
    at org.eclipse.jetty.ee8.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1151)
    at org.eclipse.jetty.ee8.servlet.ServletHolder.handle(ServletHolder.java:640)

Observed Behavior in Logs:
When instances use the EE8 profile (Jetty 12 + Java 21), multipart requests fail. The log messages for such instances include:

INFO: Using runtime classpath: /base/java_runtime/runtime-impl-jetty12.jar
Feb 25, 2025 3:08:02 PM com.google.apphosting.runtime.ClassPathUtils initForJava11OrAbove
INFO: AppEngine is using EE8 profile.

On the other hand, when instances run on Java 17 (Jetty 9), multipart handling works correctly. The working instances log the following:

INFO: Using runtime classpath: /base/java_runtime/runtime-impl-jetty9.jar

Key Issues:

• Random Runtime Selection: Despite our app being configured for Java 17, some instances randomly switch to EE8 (Java 21 + Jetty 12) without any changes in our configuration.
• Multipart Handling Breaks in Jetty 12: Since our app was developed for Jetty 9, the multipart servlet configuration is not working correctly under Jetty 12.

Questions

• Why is GAE Standard randomly choosing the EE8 profile (Jetty 12 + Java 21) for some instances instead of Java 17 (Jetty 9)?
• Is there an explicit way to enforce Java 17 (Jetty 9) and prevent GAE from switching to Jetty 12?

(appengine-web.xml)

<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
    <sessions-enabled>true</sessions-enabled>
    <runtime>java17</runtime>
    <app-engine-apis>true</app-engine-apis>
    <instance-class>F2</instance-class>
    <automatic-scaling>
        <target-cpu-utilization>0.95</target-cpu-utilization>
        <max-pending-latency>8s</max-pending-latency>
    </automatic-scaling>
    <staging>
        <enable-jar-classes>true</enable-jar-classes>
    </staging>
</appengine-web-app>

[vivekbhatnagar]

https://cloud.google.com/appengine/docs/standard/java-gen2/upgrade-java-runtime?utm_source=chatgpt.com#java_runtime_compatibility

[lachlan-roberts]

This is due to an experiment on rolling out the Java21 runtime with Jetty EE8 which will eventually replace the runtimes using Jetty 9.4. @maigovannon will have more info on this, and how to get excluded from this experiment.

But there should be no changes in behavior while migrating to the Jetty 12 with EE8, so this is likely a bug.

Can you please detail how you are configuring multipart for your Servlet, I cannot seem to reproduce this change in behavior.

[maigovannon]

Hi thanks for reaching out to us. We are indeed conducting an internal migration from Jetty 9 (EE7) to Jetty 12 (EE8). However, I would like to point out that this experiment is only scoped to Java17 apps and not for Java21 or Java 11, etc. We are not upgrading the Java versions so I would be surprised to see such behaviour. @vivekbhatnagar could you please clarify where exactly you are seeing Java21 in your logs? The lgos show only Jetty12 being used.

Having said that the above is a bug and we will look into it. In order to mitigate it, please revert to using the following flag (ref: https://cloud.google.com/appengine/docs/standard/java-gen2/upgrade-java-runtime#ee8):

<system-properties>
    <property name="appengine.use.EE8" value="false"/>
</system-properties>

This would explicitly force the runtime to use EE7.

[vivekbhatnagar]

Hi @maigovannon ,

Thank you for looking into this issue. I appreciate the clarification regarding the migration from Jetty 9 (EE7) to Jetty 12 (EE8) and that this experiment is scoped only to Java 17 applications.

To clarify, I do not explicitly see "Java 21" in the startup logs, but I do observe that when an instance starts with the EE8 profile, it exhibits different logging behavior. Specifically:

Instances running with EE8 (Jetty 12) show:

INFO: Using runtime classpath: /base/java_runtime/runtime-impl-jetty12.jar  
Feb 25, 2025 3:08:02 PM com.google.apphosting.runtime.ClassPathUtils initForJava11OrAbove  
INFO: AppEngine is using EE8 profile.  

Whereas instances running with Jetty 9 (EE7), where everything works as expected, show:

INFO: Using runtime classpath: /base/java_runtime/runtime-impl-jetty9.jar

When an instance is assigned the EE8 profile, our servlets using @MultipartConfig fail to initialize properly (this is confirmed, we ran multiple scenarios and everytime instance with EE8 profile failed as), leading to the following error:

java.lang.IllegalStateException: No multipart config for servlet
    at org.eclipse.jetty.ee8.nested.Request.getParts(Request.java:1714)
    at org.eclipse.jetty.ee8.nested.Request.getPart(Request.java:1689)
    at com.test.blah.blah.ABCServlet.doPost(ABCServlet.java:74)

This does not happen when running on Jetty 9. Mind you, our application is on Java 17.

Observations & Speculations:

• My initial assumption was that EE8 was tied to Java 21 based on the documentation here:
  Java Runtime Compatibility - Cloud App Engine.
• This led to some confusion, but I now understand that Java 21 is not being used in this experiment.
• My gut feeling is that @MultipartConfig requires a different initialization approach in Jetty 12, causing the issue.
• The provided workaround (appengine.use.EE8=false) has resolved the issue for us, and we no longer see instances using EE8.

Servlet Example:

For reference, our servlet is structured as follows:

import javax.servlet.RequestDispatcher;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletResponse;
import javax.servlet.annotation.MultipartConfig;
import javax.servlet.annotation.WebServlet;

@WebServlet("/abc")
@MultipartConfig
public class ABCServlet extends HttpServlet {
    private static final Logger log = Logger.getLogger(ABCServlet.class.getName());

    @Override
    protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
        // Handle file uploads
    }
}

We are aware that a migration to Jakarta EE (from javax.servlet.* to jakarta.servlet.*) is required in the future.

If you need additional details to reproduce the issue, please let me know.
Also, please don’t judge us for still using "old-school" servlets 😊, they remain robust and serve our needs well!

Looking forward to your insights. Thanks again for your help!

[maigovannon]

Also, please don’t judge us for still using "old-school" servlets 😊, they remain robust and serve our needs well!

Not at all. We use servlets too 😊. I would request you to file a Google support bug on this, share the bugId link and we will fast-track the mitigation. I request this process since you will ahve to share your customer project details and we don't do that in Github.

We will root-cause and fix it but we will mitigate it in our backend so that your apps are pinned to Jetty9. Though by doing the same from #358 (comment) will mitigate it fully directly through your end.

[lachlan-roberts]

This happens because of an issue in the quickstart-web.xml generated by Jetty 9.4, which causes issues when deployed in Jetty 12.

See jetty/jetty.project#12981 and jetty/jetty.project#12984 for more details.

The fix will be included in Jetty 12.0.20 which will be released at the end of this month.

[ludoch]

#368 is the fix