Skip to content

Commit 830e635

Browse files
xCASxMarcelo Vanzin
authored andcommitted
[SPARK-27868][CORE][FOLLOWUP] Recover the default value to -1 again
The default value for backLog set back to -1, as any other value may break existing configuration by overriding Netty's default io.netty.util.NetUtil#SOMAXCONN. The documentation accordingly adjusted. See discussion thread: #24732 ### What changes were proposed in this pull request? Partial rollback of #24732 (default for backLog set back to -1). ### Why are the changes needed? Previous change introduces backward incompatibility by overriding default of Netty's `io.netty.util.NetUtil#SOMAXCONN` Closes #27230 from xCASx/master. Authored-by: Maxim Kolesnikov <[email protected]> Signed-off-by: Marcelo Vanzin <[email protected]>
1 parent fd308ad commit 830e635

File tree

2 files changed

+9
-4
lines changed

2 files changed

+9
-4
lines changed

common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -108,8 +108,12 @@ public int numConnectionsPerPeer() {
108108
return conf.getInt(SPARK_NETWORK_IO_NUMCONNECTIONSPERPEER_KEY, 1);
109109
}
110110

111-
/** Requested maximum length of the queue of incoming connections. Default is 64. */
112-
public int backLog() { return conf.getInt(SPARK_NETWORK_IO_BACKLOG_KEY, 64); }
111+
/**
112+
* Requested maximum length of the queue of incoming connections. If &lt; 1,
113+
* the default Netty value of {@link io.netty.util.NetUtil#SOMAXCONN} will be used.
114+
* Default to -1.
115+
*/
116+
public int backLog() { return conf.getInt(SPARK_NETWORK_IO_BACKLOG_KEY, -1); }
113117

114118
/** Number of threads used in the server thread pool. Default to 0, which is 2x#cores. */
115119
public int serverThreads() { return conf.getInt(SPARK_NETWORK_IO_SERVERTHREADS_KEY, 0); }

docs/configuration.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -844,13 +844,14 @@ Apart from these, the following properties are also available, and may be useful
844844
</tr>
845845
<tr>
846846
<td><code>spark.shuffle.io.backLog</code></td>
847-
<td>64</td>
847+
<td>-1</td>
848848
<td>
849849
Length of the accept queue for the shuffle service. For large applications, this value may
850850
need to be increased, so that incoming connections are not dropped if the service cannot keep
851851
up with a large number of connections arriving in a short period of time. This needs to
852852
be configured wherever the shuffle service itself is running, which may be outside of the
853-
application (see <code>spark.shuffle.service.enabled</code> option below).
853+
application (see <code>spark.shuffle.service.enabled</code> option below). If set below 1,
854+
will fallback to OS default defined by Netty's <code>io.netty.util.NetUtil#SOMAXCONN</code>.
854855
</td>
855856
</tr>
856857
<tr>

0 commit comments

Comments
 (0)