Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BDX transfer support for Python tests #34821

Open
wants to merge 55 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 48 commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
281d8e2
Add the python-C++ translation.
harimau-qirex Jun 3, 2024
1d17e6c
Add a BDX transfer server to handle unsolicited BDX init messages.
harimau-qirex Jun 3, 2024
8832ed9
Add the manager to implement the transfer pool.
harimau-qirex Jun 3, 2024
d8cce79
Add the initial implementation of a BDX transfer.
harimau-qirex Jun 4, 2024
0f92e4d
Use BdxTransfer in the other classes.
harimau-qirex Jun 4, 2024
1b9b9d4
Update constructors to set the delegates etc. correctly.
harimau-qirex Jun 4, 2024
39ca533
Implement the C++ side of the barrier. Move the data callback into th…
harimau-qirex Jun 4, 2024
d330e72
Add a way to map the transfer to the python contexts.
harimau-qirex Jun 4, 2024
837d30a
Fix some of the minor TODOs.
harimau-qirex Jun 6, 2024
c50590f
Add init/shutdown to the transfer server.
harimau-qirex Jun 6, 2024
e4ea82a
Start on the implementation of the Python side.
harimau-qirex Jun 6, 2024
6eaa9ac
Listen for all BDX protocol messages rather than just the init messages.
harimau-qirex Jun 7, 2024
c15f43d
Fix minor issues in the transfer server.
harimau-qirex Jun 7, 2024
d2fe4d0
Implement a good chunk of the python side.
harimau-qirex Jun 11, 2024
4e5bb67
Fix compile errors.
harimau-qirex Jul 23, 2024
7c458b4
Fix a number of issues preventing the BDX python code from running at…
harimau-qirex Jul 25, 2024
614bafd
Return the results of the python-C methods.
harimau-qirex Jul 30, 2024
2a62e8a
Fix the async-ness of the methods that prepare the system to receive …
harimau-qirex Jul 30, 2024
fee179b
Initialise the BDX transfer server.
harimau-qirex Aug 1, 2024
bac4a10
Fixes necessary to await on the future from PrepareToReceive/SendBdxD…
harimau-qirex Aug 2, 2024
0f16585
Fix sending the accept message.
harimau-qirex Aug 2, 2024
175c347
Acknowledge received blocks so the BDX transfer continues.
harimau-qirex Aug 2, 2024
a67f783
Fix the parameters of the python callback methods.
harimau-qirex Aug 2, 2024
b114126
Add another async transaction class to handle the transfer completed …
harimau-qirex Aug 2, 2024
a5b115d
Add comments to the C++ code.
harimau-qirex Aug 6, 2024
a345c55
Add a test for the BDX transfer that uses the diagnostic logs cluster.
harimau-qirex Aug 6, 2024
4e0fb49
Move the calls to release a transfer out of the manager so it works t…
harimau-qirex Aug 6, 2024
dec62e9
Delay releasing the C++ BDX transfer object until after it's no longe…
harimau-qirex Aug 6, 2024
2670d2d
Verify the diagnostic logs response is a success.
harimau-qirex Aug 6, 2024
4049521
Restyled by whitespace
restyled-commits Aug 6, 2024
e64362d
Restyled by clang-format
restyled-commits Aug 6, 2024
4129083
Restyled by gn
restyled-commits Aug 6, 2024
8bfde70
Restyled by autopep8
restyled-commits Aug 6, 2024
e10ed98
Restyled by isort
restyled-commits Aug 6, 2024
de8c8a5
Improve BdxTransferManager's comments.
harimau-qirex Aug 13, 2024
453afaa
Use a vector for the data to send over a BDX transfer rather than a r…
harimau-qirex Aug 13, 2024
d618ec3
Minor renames.
harimau-qirex Aug 13, 2024
2148873
Improve the error message when the BDX transfer pool is exhausted.
harimau-qirex Aug 20, 2024
36ee48f
Minor fixes.
harimau-qirex Aug 21, 2024
4423da6
Pass the status report's status code up the stack.
harimau-qirex Aug 21, 2024
37a9721
Merge the BDX transfer server into the manager.
harimau-qirex Aug 22, 2024
fdb9bcc
Rename BdxTransferManager to TestBdxTransferServer.
harimau-qirex Aug 22, 2024
7d62ba8
Minor cleanup.
harimau-qirex Sep 16, 2024
3b4d586
Improve the documentation of the ownership in the C++ side.
harimau-qirex Nov 1, 2024
1ab7896
Restyled by clang-format
restyled-commits Nov 7, 2024
0d90892
Restyled by autopep8
restyled-commits Nov 7, 2024
2c4704a
Update the new test to work with the new formatting.
harimau-qirex Nov 7, 2024
69d4db7
Lint fixes.
harimau-qirex Nov 8, 2024
0b52760
Fix clang-tidy errors.
harimau-qirex Nov 8, 2024
8c23dc2
Several fixes suggested by Andrei.
harimau-qirex Nov 11, 2024
e94fd49
Fix a name in a comment.
harimau-qirex Nov 14, 2024
fb1a231
Fix issues preventing test from working.
harimau-qirex Nov 14, 2024
c092c64
Rename the methods that accept transfers so it's clear which way the …
harimau-qirex Nov 14, 2024
ea73b94
Add doc comments to the Python classes and methods.
harimau-qirex Nov 14, 2024
3f9372a
Fix issues found by mypy.
harimau-qirex Nov 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions src/controller/python/BUILD.gn
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,11 @@ shared_library("ChipDeviceCtrl") {
"ChipDeviceController-StorageDelegate.cpp",
"ChipDeviceController-StorageDelegate.h",
"OpCredsBinding.cpp",
"chip/bdx/bdx-transfer.cpp",
"chip/bdx/bdx-transfer.h",
"chip/bdx/bdx.cpp",
"chip/bdx/test-bdx-transfer-server.cpp",
"chip/bdx/test-bdx-transfer-server.h",
"chip/clusters/attribute.cpp",
"chip/clusters/command.cpp",
"chip/commissioning/PlaceholderOperationalCredentialsIssuer.h",
Expand Down Expand Up @@ -166,6 +171,9 @@ chip_python_wheel_action("chip-core") {
"chip/ChipStack.py",
"chip/FabricAdmin.py",
"chip/__init__.py",
"chip/bdx/Bdx.py",
"chip/bdx/BdxTransfer.py",
"chip/bdx/__init__.py",
"chip/ble/__init__.py",
"chip/ble/commissioning/__init__.py",
"chip/ble/get_adapters.py",
Expand Down Expand Up @@ -235,6 +243,7 @@ chip_python_wheel_action("chip-core") {

py_packages = [
"chip",
"chip.bdx",
"chip.ble",
"chip.ble.commissioning",
"chip.configuration",
Expand Down
31 changes: 31 additions & 0 deletions src/controller/python/chip/ChipDeviceCtrl.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@
from . import FabricAdmin
from . import clusters as Clusters
from . import discovery
from .bdx import Bdx
from .clusters import Attribute as ClusterAttribute
from .clusters import ClusterObjects as ClusterObjects
from .clusters import Command as ClusterCommand
Expand Down Expand Up @@ -1343,6 +1344,36 @@ def WriteGroupAttribute(
# An empty list is the expected return for sending group write attribute.
return []

def TestOnlyPrepareToReceiveBdxData(self) -> asyncio.Future:
'''
Sets up the system to expect a node to initiate a BDX transfer. The transfer will send data here.

Returns:
- a future that will yield a BdxTransfer with the init message from the transfer.
'''
self.CheckIsActive()

eventLoop = asyncio.get_running_loop()
future = eventLoop.create_future()

Bdx.PrepareToReceiveBdxData(future).raise_on_error()
return future

def TestOnlyPrepareToSendBdxData(self, data: bytes) -> asyncio.Future:
'''
Sets up the system to expect a node to initiate a BDX transfer. The transfer will send data to the node.

Returns:
- a future that will yield a BdxTransfer with the init message from the transfer.
'''
self.CheckIsActive()

eventLoop = asyncio.get_running_loop()
future = eventLoop.create_future()

Bdx.PrepareToSendBdxData(future, data).raise_on_error()
return future

def _parseAttributePathTuple(self, pathTuple: typing.Union[
None, # Empty tuple, all wildcard
typing.Tuple[int], # Endpoint
Expand Down
2 changes: 2 additions & 0 deletions src/controller/python/chip/ChipStack.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
import chip.native
from chip.native import PyChipError

from .bdx import Bdx
from .clusters import Attribute as ClusterAttribute
from .clusters import Command as ClusterCommand
from .exceptions import ChipStackError, ChipStackException, DeviceError
Expand Down Expand Up @@ -175,6 +176,7 @@ def HandleChipThreadRun(callback):
im.InitIMDelegate()
ClusterAttribute.Init()
ClusterCommand.Init()
Bdx.Init()
cecille marked this conversation as resolved.
Show resolved Hide resolved

builtins.chipStack = self

Expand Down
184 changes: 184 additions & 0 deletions src/controller/python/chip/bdx/Bdx.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,184 @@
#
# Copyright (c) 2024 Project CHIP Authors
# All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

import asyncio
import builtins
import ctypes
from asyncio.futures import Future
from ctypes import CFUNCTYPE, POINTER, c_char_p, c_size_t, c_uint8, c_uint16, c_uint64, c_void_p, py_object

import chip
from chip.native import PyChipError

from . import BdxTransfer

c_uint8_p = POINTER(c_uint8)


_OnTransferObtainedCallbackFunct = CFUNCTYPE(
None, py_object, c_void_p, c_uint8, c_uint16, c_uint64, c_uint64, c_uint8_p, c_uint16, c_uint8_p, c_size_t)
_OnFailedToObtainTransferCallbackFunct = CFUNCTYPE(None, py_object, PyChipError)
_OnDataReceivedCallbackFunct = CFUNCTYPE(None, py_object, c_uint8_p, c_size_t)
_OnTransferCompletedCallbackFunct = CFUNCTYPE(None, py_object, PyChipError)


class AsyncTransferObtainedTransaction:
harimau-qirex marked this conversation as resolved.
Show resolved Hide resolved
def __init__(self, future, event_loop, data=None):
self._future = future
self._data = data
self._event_loop = event_loop

def _handleTransfer(self, bdxTransfer, initMessage: BdxTransfer.InitMessage):
transfer = BdxTransfer.BdxTransfer(bdx_transfer=bdxTransfer, init_message=initMessage, data=self._data)
self._future.set_result(transfer)

def handleTransfer(self, bdxTransfer, initMessage: BdxTransfer.InitMessage):
self._event_loop.call_soon_threadsafe(self._handleTransfer, bdxTransfer, initMessage)

def _handleError(self, result: PyChipError):
self._future.set_exception(result.to_exception())

def handleError(self, result: PyChipError):
self._event_loop.call_soon_threadsafe(self._handleError, result)


class AsyncTransferCompletedTransaction:
def __init__(self, future, event_loop):
self._future = future
self._event_loop = event_loop

def _handleResult(self, result: PyChipError):
if result.is_success:
self._future.set_result(result)
else:
self._future.set_exception(result.to_exception())

def handleResult(self, result: PyChipError):
self._event_loop.call_soon_threadsafe(self._handleResult, result)


@_OnTransferObtainedCallbackFunct
def _OnTransferObtainedCallback(transaction: AsyncTransferObtainedTransaction, bdxTransfer, transferControlFlags: int,
maxBlockSize: int, startOffset: int, length: int, fileDesignator, fileDesignatorLength: int,
metadata, metadataLength: int):
fileDesignatorData = ctypes.string_at(fileDesignator, fileDesignatorLength)
metadataData = ctypes.string_at(metadata, metadataLength)

initMessage = BdxTransfer.InitMessage(
transferControlFlags,
maxBlockSize,
startOffset,
length,
fileDesignatorData[:],
metadataData[:],
)

transaction.handleTransfer(bdxTransfer, initMessage)


@_OnFailedToObtainTransferCallbackFunct
def _OnFailedToObtainTransferCallback(transaction: AsyncTransferObtainedTransaction, result: PyChipError):
transaction.handleError(result)


@_OnDataReceivedCallbackFunct
def _OnDataReceivedCallback(context, dataBuffer: c_uint8_p, bufferLength: int):
data = ctypes.string_at(dataBuffer, bufferLength)
context(data)


@_OnTransferCompletedCallbackFunct
def _OnTransferCompletedCallback(transaction: AsyncTransferCompletedTransaction, result: PyChipError):
transaction.handleResult(result)


def PrepareToReceiveBdxData(future: Future) -> PyChipError:
harimau-qirex marked this conversation as resolved.
Show resolved Hide resolved
handle = chip.native.GetLibraryHandle()
transaction = AsyncTransferObtainedTransaction(future=future, event_loop=asyncio.get_running_loop())

ctypes.pythonapi.Py_IncRef(ctypes.py_object(transaction))
res = builtins.chipStack.Call(
lambda: handle.pychip_Bdx_ExpectBdxTransfer(ctypes.py_object(transaction))
)
if not res.is_success:
ctypes.pythonapi.Py_DecRef(ctypes.py_object(transaction))
return res


def PrepareToSendBdxData(future: Future, data: bytes) -> PyChipError:
handle = chip.native.GetLibraryHandle()
transaction = AsyncTransferObtainedTransaction(future=future, event_loop=asyncio.get_running_loop(), data=data)
harimau-qirex marked this conversation as resolved.
Show resolved Hide resolved

ctypes.pythonapi.Py_IncRef(ctypes.py_object(transaction))
res = builtins.chipStack.Call(
lambda: handle.pychip_Bdx_ExpectBdxTransfer(ctypes.py_object(transaction))
)
if not res.is_success:
ctypes.pythonapi.Py_DecRef(ctypes.py_object(transaction))
return res


def AcceptSendTransfer(transfer: c_void_p, dataReceivedClosure, transferComplete: Future):
harimau-qirex marked this conversation as resolved.
Show resolved Hide resolved
handle = chip.native.GetLibraryHandle()
complete_transaction = AsyncTransferCompletedTransaction(future=transferComplete, event_loop=asyncio.get_running_loop())
ctypes.pythonapi.Py_IncRef(ctypes.py_object(dataReceivedClosure))
ctypes.pythonapi.Py_IncRef(ctypes.py_object(complete_transaction))
res = builtins.chipStack.Call(
lambda: handle.pychip_Bdx_AcceptSendTransfer(transfer, dataReceivedClosure, complete_transaction)
)
if not res.is_success:
ctypes.pythonapi.Py_DecRef(ctypes.py_object(dataReceivedClosure))
ctypes.pythonapi.Py_DecRef(ctypes.py_object(complete_transaction))
return res


def AcceptReceiveTransfer(transfer: c_void_p, data: bytes, transferComplete: Future):
harimau-qirex marked this conversation as resolved.
Show resolved Hide resolved
handle = chip.native.GetLibraryHandle()
return builtins.chipStack.Call(
lambda: handle.pychip_Bdx_AcceptReceiveTransfer(transfer, c_char_p(data), len(data), transferComplete)
)


async def RejectTransfer(transfer: c_void_p):
handle = chip.native.GetLibraryHandle()
return await builtins.chipStack.CallAsyncWithResult(
lambda: handle.pychip_Bdx_RejectTransfer(transfer)
)


def Init():
handle = chip.native.GetLibraryHandle()
# Uses one of the type decorators as an indicator for everything being initialized.
if not handle.pychip_Bdx_ExpectBdxTransfer.argtypes:
setter = chip.native.NativeLibraryHandleMethodArguments(handle)

setter.Set('pychip_Bdx_ExpectBdxTransfer',
PyChipError, [py_object])
setter.Set('pychip_Bdx_StopExpectingBdxTransfer',
PyChipError, [py_object])
setter.Set('pychip_Bdx_AcceptSendTransfer',
PyChipError, [c_void_p, py_object, py_object])
setter.Set('pychip_Bdx_AcceptReceiveTransfer',
PyChipError, [c_void_p, c_uint8_p, c_size_t])
setter.Set('pychip_Bdx_RejectTransfer',
PyChipError, [c_void_p])
setter.Set('pychip_Bdx_InitCallbacks', None, [
_OnTransferObtainedCallbackFunct, _OnFailedToObtainTransferCallbackFunct, _OnDataReceivedCallbackFunct,
_OnTransferCompletedCallbackFunct])

handle.pychip_Bdx_InitCallbacks(
_OnTransferObtainedCallback, _OnFailedToObtainTransferCallback, _OnDataReceivedCallback, _OnTransferCompletedCallback)
59 changes: 59 additions & 0 deletions src/controller/python/chip/bdx/BdxTransfer.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
#
# Copyright (c) 2024 Project CHIP Authors
# All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

import asyncio
from ctypes import c_void_p
from dataclasses import dataclass

from . import Bdx


@dataclass
class InitMessage:
TransferControlFlags: int
harimau-qirex marked this conversation as resolved.
Show resolved Hide resolved
MaxBlockSize: int
StartOffset: int
Length: int
FileDesignator: bytes
Metadata: bytes


class BdxTransfer:
def __init__(self, bdx_transfer: c_void_p, init_message: InitMessage, data: bytes = None):
self.init_message = init_message
self._bdx_transfer = bdx_transfer
self._data = bytearray(data) if data else None
harimau-qirex marked this conversation as resolved.
Show resolved Hide resolved

async def accept(self):
harimau-qirex marked this conversation as resolved.
Show resolved Hide resolved
eventLoop = asyncio.get_running_loop()
future = eventLoop.create_future()

if self._data is not None:
res = Bdx.AcceptReceiveTransfer(self._bdx_transfer, self._data, future)
res.raise_on_error()
return await future
else:
self._data = bytearray()
res = Bdx.AcceptSendTransfer(self._bdx_transfer, lambda data: self._data.extend(data), future)
res.raise_on_error()
await future
return self._data[:]

async def reject(self):
res = await Bdx.RejectTransfer(self._bdx_transfer)
res.raise_on_error()
return res
27 changes: 27 additions & 0 deletions src/controller/python/chip/bdx/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#
# Copyright (c) 2024 Project CHIP Authors
# All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

#
# @file
# Provides Python APIs for CHIP.
#

"""Provides Python APIs for CHIP."""

from . import Bdx, BdxTransfer

__all__ = [Bdx, BdxTransfer]
Loading
Loading