Does the API support multi-thread/multi-processing?

[cofiiwu]

What happened (please include outputs or screenshots):
I implement a scheduler with multi-thread/multi-process to create/delete/binding PODs and monitor them. I often get the following error information, so I want to make sure if the API can be used in this way.

2019-11-12 00:11:57,724 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: DECRYPTION_FAILED_OR_BAD_REC[384/727345]
ryption failed or bad record mac (_ssl.c:2508)'))': /api/v1/namespaces/default/pods/py-mnist-pod-14505?gracePeriodSeconds=20
Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac
(_ssl.c:2508)'))': /api/v1/namespaces/default/pods/py-mnist-pod-14505?gracePeriodSeconds=20
Process Executor_k8s-5:
Traceback (most recent call last):
File "/home/xxx/storage/miniconda3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/mnt/client/xxx/zzz/yyy/src/system/enforcement/executor.py", line 288, in run
workload = self.execute(self.job)
File "/mnt/client/xxx/zzz/yyy/src/system/enforcement/executor.py", line 269, in execute
self.k8s_wait_finish(job)
File "/mnt/client/xxx/zzz/yyy/src/system/enforcement/executor.py", line 235, in k8s_wait_finish
for event in w.stream(v1.list_namespaced_pod,"default"):
File "/home/xxx/storage/miniconda3/lib/python3.7/site-packages/kubernetes/watch/watch.py", line 117, in stream
for line in iter_resp_lines(resp):
File "/home/xxx/storage/miniconda3/lib/python3.7/site-packages/kubernetes/watch/watch.py", line 45, in iter_resp_lines
for seg in resp.read_chunked(decode_content=False):
File "/home/xxx/storage/miniconda3/lib/python3.7/site-packages/urllib3/response.py", line 752, in read_chunked
self._update_chunk_length()
File "/home/xxx/storage/miniconda3/lib/python3.7/site-packages/urllib3/response.py", line 682, in _update_chunk_length
line = self._fp.fp.readline()
File "/home/xxx/storage/miniconda3/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
File "/home/xxx/storage/miniconda3/lib/python3.7/ssl.py", line 1071, in recv_into
return self.read(nbytes, buffer)
File "/home/xxx/storage/miniconda3/lib/python3.7/ssl.py", line 929, in read
return self._sslobj.read(len, buffer)

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):
I just call config.load_kube_config() and client.Configuration() once in the main function, and then
use the API to create/delete/monitor POD in multiple threads.

Anything else we need to know?:

• I have searched the error message, some results hint the error message is about multi-process/multi-thread. https://stackoverflow.com/questions/3724900/python-ssl-problem-with-multiprocessing
• I develop it on single node.

Environment:

• Kubernetes version (kubectl version): 1.15
• OS: CentOS 7
• Python version :3.7.4
• Python client version :2.0.0 , I must use the version because of the issue 547 (create_namespaced_binding() method shows target.name: Required value #547)

UPDATE
If I implement my design with multi-thread(threading module), the error never show again.
The error seems to appear in multi-processing(multiprocessing module) only.
For performance, I hope I can use the API in multi-process.

[roycaihw]

/assign @jiahuif

[k8s-ci-robot]

@roycaihw: GitHub didn't allow me to assign the following users: jiahuif.

Note that only kubernetes-client members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @jiahuif

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[cofiiwu]

Hi @roycaihw ,
do you have any information?

[jiahuif]

This issue does not seem like threading problem to me. According to the exception, it seems more like a protocol/networking issue. Plus, it is very strange to see SSLV3 mentioned.

Could you please try to create a minimal script for reproduction?

[cofiiwu]

@jiahuif
I can try it.
I refer to multi-thread because I read this https://stackoverflow.com/questions/3724900/python-ssl-problem-with-multiprocessing
However the related operations to SSL are implemented in the client API, so maybe I use the client API in a wrong way? If I should share some objects related to SSL among processes?

Here is the concept of the current implementation. If it is a wrong usage, let me know please.

• Thread/Process 1
  config.load_kube_config()
  client.Configuration()
  create Thread 2 and Thread 3
  call client API to delete POD

The functions are called

v1=client.CoreV1Api()
v1.delete_namespaced_pod(....)

• Thread/process 2 : watching events to pick PODs I want to schedule

The functions are called

 v1=client.CoreV1Api()
 w = watch.Watch()
 for event in w.stream(v1.list_namespaced_pod,"default"):

• Thread/process 3: binding POD and monitor PODs

The functions are called

v1=client.CoreV1Api()
v1.create_namespaced_pod
v1.create_namespaced_binding_binding
v1.delete_namespaced_pod
w = watch.Watch()
for event in w.stream(v1.list_namespaced_pod,"default"):

[cofiiwu]

Update: The error seems to appear in multi-process only(multiprocessing module).

[cofiiwu]

update
The issue is similar to the current issue
psf/requests#1906
Is there any way to disable SSL? I don't care security.

[ceroloy]

Hello @cofiiwu
Yes you can disable SSL via

from kubernetes import config, client

config = client.Configuration()
config.verify_ssl = False

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.