diff --git a/docs/architecture.rst b/docs/architecture.rst index b25e848..92835b4 100644 --- a/docs/architecture.rst +++ b/docs/architecture.rst @@ -1,353 +1,8 @@ ๐Ÿ  Architecture ================ - The content below assumes you have fairly good knowledge of the following: +.. toctree:: - - OOP and descriptors, especially - - Type annotations - - Binary data types and streams - -.. svgbob:: - :align: center - - โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” - โ”‚ Events - binary representation - low level API - Stage 1 parser โ”‚ - โ”‚ โ”‚ - โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ - โ”‚ โ”‚ Project-wide / 1-time โ”‚ โ”‚ Per-instance โ”‚ โ”‚ Shared โ”‚ โ”‚ - โ”‚ โ”‚โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”โ”‚ โ”‚โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚ - โ”‚ โ”‚โ”‚ Event 1 โ”‚ โ”‚ Event 2 โ”‚โ”‚ โ”‚โ”‚ Event 3 โ”‚ โ”‚ Event 4 โ”‚โ”‚ โ”‚ โ”‚ Event 5 โ”‚ โ”‚ โ”‚ - โ”‚ โ”‚โ”‚ id: 199 โ”‚ โ†’ โ”‚ id: 159 โ”‚โ”‚โ†’โ”‚โ”‚ id: 64 โ”‚ โ†’ โ”‚ id: 215 โ”‚โ”‚โ†’โ”‚ โ”‚ id: 225 โ”‚ โ”‚ โ”‚ - โ”‚ โ”‚โ”‚ string โ”‚ โ”‚ integer โ”‚โ”‚ โ”‚โ”‚ integer โ”‚ โ”‚ struct โ”‚โ”‚ โ”‚ โ”‚ AoS โ”‚ โ”‚ โ”‚ - โ”‚ โ”‚โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜โ”‚ โ”‚โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ - โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”‚โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”‚โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ - โ”‚ โ”‚ โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ โ”‚ โ”‚ - โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ - โ”‚ โ”‚ Model A โ”‚ โ”‚ Model โ”‚ Model B1 โ”‚ attr_64: int โ”‚ โ”‚ Model C1: e[0] โ”‚ โ”‚ - โ”‚ โ”‚ attr_199: str โ”‚ โ”‚ list โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ - โ”‚ โ”‚ attr_159: int โ”‚ โ”‚ of B โ”‚ Model B2 โ”‚ attr_215: X โ”‚ โ”‚ Model C2: e[1] โ”‚ โ”‚ - โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ - โ”‚ โ”‚ - โ”‚ Models - PyFLP's representation - high level API - Stage 2 parser โ”‚ - โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - -PyFLP provides a high-level and a low-level API. Normally the high-level API -should get your work done. However, it might be possible that due to a bug or -super old versions of FLPs the high level API fails to parse. In that case, -one can use the low-level API. Using it requires a deeper understanding of -the FLP format and how the GUI hierarchies relate to their underlying events. - -.. caution:: - - Using the high-level and low-level API simultaneously can cause a loss of - synchronisation across the state, although it normally shouldn't this - use-case should be considered untested. - -โฌ‡ The sections below are ordered from low-level to high-level concepts. - -.. _architecture-event: - -Understanding events --------------------- - -.. automodule:: pyflp._events - :show-inheritance: - -The FLP format uses a :wikipedia:`Type-length-value` encoding to store almost -all of it's data. *It's an incredibly bad format, full of bad design decisions -AFAIK.* - -That being said, all the data except: - -- PPQ - :attr:`pyflp.project.Project.ppq` -- Number of channels - :attr:`pyflp.project.Project.channel_count` -- Internal file format - :attr:`pyflp.project.Project.format` - -is stored in a structure called an **Event**. - -โ” What is an **Event**? -^^^^^^^^^^^^^^^^^^^^^^^^ - -.. note:: C terminology - - I use some C terminology below like ``struct`` and its data types. - I recommend you to get acquainted with these topics, however as a - contributor, I am sure you have an equivalent programming background. - -Following can be considered as a pseudo C-style structure of an event: - -.. code-block:: c - - typedef struct { - uint8_t id; - void* data; - } event; - -It means that every event begins with an ID (known as the event ID) followed by -its data. The size of this ``data`` is fixed or variable sized depending on -``id``. - -This table shows how the size of ``data`` is decided: - -+----------+------------------------------+-------------------------------+ -| Event ID | Size of ``data`` (in bytes) | Total event size (in bytes) | -+==========+==============================+===============================+ -| 0-63 | 1 | 1 + 1 = **2** | -+----------+------------------------------+-------------------------------+ -| 64-127 | 2 | 1 + 2 = **3** | -+----------+------------------------------+-------------------------------+ -| 128-191 | 4 | 1 + 4 = **5** | -+----------+------------------------------+-------------------------------+ -| 192-255 | ``varint`` | 1 + ``encoded`` + ``decoded`` | -+----------+------------------------------+-------------------------------+ - -Events are the first stage of parsing in PyFLP. The :meth:`pyflp.parse` method -gathers all events by reading an FLP file as a binary stream. - -Representation -^^^^^^^^^^^^^^ - -An event ID is represented in an ``EventEnum`` subclass. - -.. autoclass:: _EventEnumMeta -.. autoclass:: EventEnum - -These enums are documented throughout the :doc:`reference`. - -For each of the range above, I have created a number of classes to match the -exact type of ``data`` indicated by its usage. What I mean by this statement -is, multiple types with different value ranges exist for a single ID range. - - For example, - - - 4 bytes can represent :wikipedia:`Single-precision_floating-point_format` - or an :wikipedia:`Integer_(computer_science)` or even a tuple of two - 2-byte integers. - - 1 byte can represent a number from -128 to 127 or a number from 0 to 255, - a boolean or event an :wikipedia:`ASCII` character. - - *.. and so on* - -.. autoclass:: EventBase - :private-members: - :special-members: - -Below are the list of classes PyFLP has, grouped according the ID range. - -.. dropdown:: 0-63 - - .. autoclass:: ByteEventBase - .. autoclass:: U8Event - .. autoclass:: BoolEvent - .. autoclass:: I8Event - -.. dropdown:: 64-127 - - .. autoclass:: WordEventBase - .. autoclass:: U16Event - .. autoclass:: I16Event - -.. dropdown:: 128-191 - - .. autoclass:: DWordEventBase - .. autoclass:: U32Event - .. autoclass:: I32Event - .. autoclass:: ColorEvent - .. autoclass:: U16TupleEvent - -.. dropdown:: 192-255 - - .. autoclass:: VarintEventBase - .. autoclass:: StrEventBase - .. autoclass:: AsciiEvent - .. autoclass:: UnicodeEvent - .. autoclass:: DataEventBase - .. autoclass:: UnknownDataEvent - .. autoclass:: StructEventBase - .. autoclass:: ListEventBase - -Parsing -^^^^^^^ - -Let's understand two terms first: - -- Fixed size events: Events with an ``id`` between 0 to 191, basically those - whose size is only decided by the ``id``. -- Variable size events: Events with an ``id`` between 192 to 255. The ``data``, - its size and the existance of these events itself is decided by a number of - factors, including but not limited to the FL Studio version used to save the - project file in which these events are saved - -Fixed size events are pretty much easy to understand, just by looking at the -code, so they won't be covered in much depth. They exist for simple things. - -Variable size events store their size encoded in a "varint", followed by the -actual data whose size is equal to the contents of the decoded "varint". This -is used for strings and custom structures. - -Custom structures are very similar to a collection of events collected in a -single C-style ``struct``. Why so? Event IDs are stored in a single byte, -which means a maximum of 256 IDs can be used in addition to the constraints -applied by the ID range itself. - - Image-Line, as shortsighted ๐Ÿ”ญ it was initially, didn't probably realise - that they will run out of the available space of 255 events pretty soon. - There however has an alternative ๐Ÿ’ก, which wouldn't cause a major breaking - change to the format itself. - - Now I don't work for Image-Line, but they probably thought ๐Ÿค”: - - We already use variable size events for strings. We can use them for - saving this valuable event ID space as well โ• - -Fast forward many versions later, FL still uses this weird mixture of fixed and -variable size events to represent what I call a :ref:`model `. - -.. todo:: - - Explain different types of "custom structures" (:class:`DataEventBase` - subclasses). - -.. todo:: - - Explain :class:`EventTree` - -.. _architecture-model: - -๐Ÿ“ฆ Understanding models ------------------------- - -A **model** is an entity, or an object, programmatically speaking. - - Models are **my** estimations of object hierarchies which mainly mimic FL - Studio's GUI hierarchy. I figured out that this is the easiest way to - expose an API programmatically. - - The FLP format has no such notion of "models" as it is entirely based on - the sequence of :doc:`events <./architecture>`. - - PyFLP's modules are categorized to follow FL Studio' GUI hierarchy as well. - Every module *generally* represents a **separate window** in the GUI. - -In PyFLP, a model is **composed** of several :ref:`descriptors `, -properties and some additional helper methods, optionally. It *might* contain -additional parsing logic for nested models and collections of models. - -A model's internal state is stored in :ref:`events ` and its -shared state is passed to it via keyword arguments. *For example*, many models -depend on :attr:`pyflp.project.Project.version` to decide the parsing logic for -certain properties. This creates a "dependancy" of the model to a "shared" -property. Such "dependencies" are passed to the model in the form of keyword -arguments and consumed by the :ref:`descriptors `. - -A model **does NOT cache** its state in any way. This is done, mainly to: - -1. Implement lazily evaluated properties and avoid use of private variables. -2. Keep the property values in sync with the event data. - -Implementing a model -^^^^^^^^^^^^^^^^^^^^ - -A look at the **source code** will definitely help, although these are a few -points that must be kept in mind when Implementing a model: - -1. Does the model mimic the hierarchy exposed by FL Studio's GUI? - - .. tip:: - - Browse through the hierarchies of :class:`pyflp.channel.Channel` - subclasses to get a very good idea of this. - -2. Are ``__dunder__`` methods provided by Python used whenever possible? -3. Is either :class:`ModelReprMixin` subclassed or ``__repr__`` implemented? - -Reference -^^^^^^^^^ - -.. automodule:: pyflp._models - :show-inheritance: - :members: - -.. _architecture-descriptor: - -\ :fas:`bars-staggered` Understanding descriptors -------------------------------------------------- - -.. automodule:: pyflp._descriptors - :show-inheritance: - -A "descriptor" provides low-level managed attribute access, according to -Python docs. *(slightly rephrased for my convenience)*. - - IMO, it allows separation of attribute logic from the class implementation - itself and this saves a huge amount of repretitive error-prone code. - - .. note:: More about descriptors in Python - - - ``_ - - ``_, **especially** the - `Why use Python descriptors? - `_ - section. - -In PyFLP, descriptors are used for attributes of a :ref:`model `. -Internally, they access the value of an :ref:`event ` or -one if its keys for :class:`StructEventBase`. They can be called *stateless* -because they never cache the value which they fetch and directly dump back into -the event when their setter is invoked. - -Some common descriptors like ``name`` ๐Ÿ”ค, ``color`` ๐ŸŽจ or ``icon`` ๐Ÿ–ผ are used by -multiple different types of models. The descriptors used for these can be -different depending upon the internal representation inside :ref:`events `. - -Despite all this, they are normal attributes from a type-checker's POV ๐Ÿ‘ when -accessed from an instance. - -.. note:: - - Throughout the documentation, I have used the term **descriptors** and - **properties** interchangeably. - -Protocols -^^^^^^^^^ - -๐Ÿ™„ Since the ``typing`` module doesn't provide any type for descriptors, I -needed to create my own: - -.. autoprotocol:: ROProperty -.. autoprotocol:: RWProperty - -Descriptors -^^^^^^^^^^^ - -.. autoclass:: PropBase -.. autoclass:: StructProp -.. autoclass:: EventProp -.. autoclass:: FlagProp -.. autoclass:: NestedProp -.. autoclass:: KWProp - -Helpers -^^^^^^^ - -.. autoclass:: NamedPropMixin - -Adapters -^^^^^^^^ - -Adapters used by :class:`construct.Struct` objects. - -.. autoclass:: LinearMusical - :members: -.. autoclass:: Log2 - :members: -.. autoclass:: LogNormal - :members: -.. autoclass:: StdEnum - :members: - -Shared models -^^^^^^^^^^^^^ - -.. autoclass:: MusicalTime - :members: + 1๏ธโƒฃ FLP Format & Events + 2๏ธโƒฃ How it works? + 3๏ธโƒฃ Developer Reference diff --git a/docs/architecture/flp-format.rst b/docs/architecture/flp-format.rst new file mode 100644 index 0000000..2a1f345 --- /dev/null +++ b/docs/architecture/flp-format.rst @@ -0,0 +1,124 @@ +Part I: FLP Format & Events +=========================== + +FLP is a binary format used by Image-Line FL Studio, a music production +software, to store project files. Instead of using C-style structs entirely, +the FLP format has evolved from what once was a MIDI-like format to a really +bad and messy combination of :wikipedia:`type-length-value` encoded "events" +and structs. + +Specification +------------- + +An FLP file contains of basically 2 sections or "chunks", one is the header +and other is the "data" section, which contains all the "events". + +Header chunk +^^^^^^^^^^^^ + +.. tab-set:: + + .. tab-item:: C / C++ + + .. code-block:: c + + struct { + char magic[4]; // 'FLhd' + uint32_t size; // always been 6 + int16_t format; // Internal file format + uint16_t num_channels; // Number of channels in channel rack + uint16_t ppq; // Pulses per quarter + } + + .. tab-item:: Python + + .. code-block:: python + + class Header: + magic: str + size: int + format: int + num_channels: int + ppq: int + +.. currentmodule:: pyflp.project +.. seealso:: + + :attr:`Project.format`, :attr:`Project.channel_count`, :attr:`Project.ppq` + +Data chunk +^^^^^^^^^^ + +.. code-block:: c + + struct { + char magic[4]; // 'FLdt' + uint32_t size; // Total combined size of events + void* events; // Event data + } + +.. _architecture-event: + +Event +----- + +An event can be thought of as a "flattened" :class:`dict` of attributes +composing a class. It can *roughly* be represented as: + +.. tab-set:: + + .. tab-item:: C / C++ + + .. code-block:: c + + struct { + uint8_t type; + void* value; + } + + .. tab-item:: Python + + .. code-block:: python + + class Event: + type: int + value: object + +Types +^^^^^ + +There are basically 4 kinds of events depending on the range of ``type``: + ++----------+--------------------+-----------------+ +| Event ID | Size of ``value`` | Total event size | ++==========+===================+==================+ +| 0-63 | 1 byte | 1 + 1 = **2** | ++----------+-------------------+------------------+ +| 64-127 | 2 bytes | 1 + 2 = **3** | ++----------+-------------------+------------------+ +| 128-191 | 4 bytes | 1 + 4 = **5** | ++----------+-------------------+------------------+ +| 192-255 | Length prefixed | >= 2 | ++----------+-------------------+------------------+ + +.. note:: Length prefixed events + + These events store the length of the ``value`` they contain after ``type`` + in a varint. It can be considered as the only true TLV encoded event type. + + .. code-block:: c + + struct { + uint8_t type; // 192-255 + uint8_t* length; // varint + void* value; // string, struct or subevent + } + +It should be clearer by now how the FLP format is a misfit for the data it +represents. + +Representation +^^^^^^^^^^^^^^ + +Event IDs 0-191 are used for storing fixed size data like integers, floats and +booleans. IDs from 192-255 are used for storing structs, subevents and strings. diff --git a/docs/architecture/how-it-works.rst b/docs/architecture/how-it-works.rst new file mode 100644 index 0000000..d8f8e52 --- /dev/null +++ b/docs/architecture/how-it-works.rst @@ -0,0 +1,110 @@ +Part II: How PyFLP works +======================== + + ๐Ÿ’ก You should read Part I before this. + +PyFLP's entry-point :meth:`pyflp.parse` verifies the headers and parses all the +events. These events are collected into an :class:`pyflp._events.EventTree`. + +Schematic diagram +----------------- + +.. svgbob:: + :align: center + + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ Events - binary representation - low level API - Stage 1 parser โ”‚ + โ”‚ โ”‚ + โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ + โ”‚ โ”‚ Project-wide / 1-time โ”‚ โ”‚ Per-instance โ”‚ โ”‚ Shared โ”‚ โ”‚ + โ”‚ โ”‚โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”โ”‚ โ”‚โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚ + โ”‚ โ”‚โ”‚ Event 1 โ”‚ โ”‚ Event 2 โ”‚โ”‚ โ”‚โ”‚ Event 3 โ”‚ โ”‚ Event 4 โ”‚โ”‚ โ”‚ โ”‚ Event 5 โ”‚ โ”‚ โ”‚ + โ”‚ โ”‚โ”‚ id: 199 โ”‚ โ†’ โ”‚ id: 159 โ”‚โ”‚โ†’โ”‚โ”‚ id: 64 โ”‚ โ†’ โ”‚ id: 215 โ”‚โ”‚โ†’โ”‚ โ”‚ id: 225 โ”‚ โ”‚ โ”‚ + โ”‚ โ”‚โ”‚ string โ”‚ โ”‚ integer โ”‚โ”‚ โ”‚โ”‚ integer โ”‚ โ”‚ struct โ”‚โ”‚ โ”‚ โ”‚ AoS โ”‚ โ”‚ โ”‚ + โ”‚ โ”‚โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜โ”‚ โ”‚โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ + โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”‚โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”‚โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”‚โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”‚โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”‚โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ + โ”‚ โ””โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏโ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ โ”‚ โ”‚ + โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ + โ”‚ โ”‚ Model A โ”‚ โ”‚ Model โ”‚ Model B1 โ”‚ attr_64: int โ”‚ โ”‚ Model C1: e[0] โ”‚ โ”‚ + โ”‚ โ”‚ attr_199: str โ”‚ โ”‚ list โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ + โ”‚ โ”‚ attr_159: int โ”‚ โ”‚ of B โ”‚ Model B2 โ”‚ attr_215: X โ”‚ โ”‚ Model C2: e[1] โ”‚ โ”‚ + โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ + โ”‚ โ”‚ + โ”‚ Models - PyFLP's representation - high level API - Stage 2 parser โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + +PyFLP provides a high-level and a low-level API. Normally the high-level API +should get your work done. However, it might be possible that due to a bug or +super old versions of FLPs the high level API fails to parse. In that case, +one can use the low-level API. Using it requires a deeper understanding of +the FLP format and how the GUI hierarchies relate to their underlying events. + +What it does? +------------- + +In a nutshell, PyFLP parses the events and creates a better semantic structure +from it (as shown in the above diagram; stage 2 parser). I call this a "model". + +.. _architecture-model: + +Model +----- + +A model acts like a "view" or alternate representation of the event data. It +has no state of its own and its composed of descriptors which get and set +values from the events directly. A model is essentially stateless. + +This has some advantages as compared to stateful models: + +1. The underlying event data and the values returned from the model descriptors + *i.e. its attributes or properties* always remain in sync with each other. +2. Since modifying the event data at a binary level means conforming to the + various size and range limits imposed by C's data types, it can act as basic + validation for no extra cost or implementation. +3. Avoid the use of private members in the models itself. Private members maybe + a good idea in languages which have better implementation of such concepts, + but in Python its quite as good as shooting yourself in the foot. Due to + Python's do-whatever-you-want nature, it can lead to some very bad coding + practices. This is one of the big reasons why PyFLP underwent a rewrite. +4. Nothing is done in class constructor, so if a particular set of events are + out of order or follow a sequence not yet understood by PyFLP, they will + fail only for the attributes which use them. Hence, what is *parseable* can + still be parsed. This lazy evaluation can be good and bad both, but with + adequate unit tests its more good than it is bad. + +Creating a model involves a good amount of reverse engineering and insight. The +models PyFLP has are based as close to the GUI objects inside FL Studio. For +e.g. a pattern is represented by :class:`pyflp.pattern.Pattern`. + +A model is constructed with events it requires and additional information (like +PPQ) its descriptors might need. + +Disadvantages +------------- + +Difficult to make ports +^^^^^^^^^^^^^^^^^^^^^^^ + +The current working of PyFLP is non-replicable in most other languages. +Descriptors are a Python specific feature I have yet to find anywhere else. +Therefore, the possibility of a port that's as clean (and featured) as PyFLP is +less. Most languages however have some sort of 3rd party Python interop library +available, so its not like PyFLP is completely unuseable from other languages. + +A quick search on Github will return some FLP parsers available for other +languages, but almost all of them are pretty much unmaintained or archived. + +Unit-testing is paramount +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Due to the lazy nature of models and their descriptors, each of them should be +tested so as to ensure that no changes in the event handling affect or break. + +For a long time, I used only a single FLP to test all of PyFLP's API. Things +have changed now and I use presets exported from FL Studio itself for the +testing of a huge chunk of API to ensure isolation of test results. + +The problem is that all the test data comes from FL Studio itself and can +be only really validated in the same. That's the reason I usually don't +raise any errors event if I know quite surely that, for example a value out of +range is set for some property. diff --git a/docs/architecture/reference.rst b/docs/architecture/reference.rst new file mode 100644 index 0000000..ff4f900 --- /dev/null +++ b/docs/architecture/reference.rst @@ -0,0 +1,105 @@ +Developer Reference +=================== + +This page documents PyFLP's internals which consists of :mod:`pyflp._events`, +:mod:`pyflp._descriptors` and :mod:`pyflp._models`. + + The content below assumes you have fairly good knowledge of the following: + + - OOP and descriptors, especially + - Type annotations + - Binary data types and streams + +Events +------ + +.. automodule:: pyflp._events + +If you have read Part I, you know how events use a TLV encoding scheme. + +Type +^^^^ + +The ``type`` represents the event ID. A custom enum class (and a metaclass) +supporting unknown IDs and member check using Python's ``... in ...`` syntax is +used. + +.. autoclass:: _EventEnumMeta + :members: +.. autoclass:: EventEnum + :members: + +Length +^^^^^^ + +The ``length`` is a field prefixed for IDs in the range of 192-255. It is the +size of ``value`` and is encoded as a VLQ128 (variable length quantity base-128). + +Value +^^^^^ + +Below are the list of classes PyFLP has, grouped w.r.t the ID range. + +.. dropdown:: 0-63 + + .. autoclass:: ByteEventBase + .. autoclass:: U8Event + .. autoclass:: BoolEvent + .. autoclass:: I8Event + +.. dropdown:: 64-127 + + .. autoclass:: WordEventBase + .. autoclass:: U16Event + .. autoclass:: I16Event + +.. dropdown:: 128-191 + + .. autoclass:: DWordEventBase + .. autoclass:: U32Event + .. autoclass:: I32Event + .. autoclass:: ColorEvent + .. autoclass:: U16TupleEvent + +.. dropdown:: 192-255 + + .. autoclass:: VarintEventBase + .. autoclass:: StrEventBase + .. autoclass:: AsciiEvent + .. autoclass:: UnicodeEvent + .. autoclass:: DataEventBase + .. autoclass:: UnknownDataEvent + .. autoclass:: StructEventBase + .. autoclass:: ListEventBase + +EventTree +^^^^^^^^^ + +.. autoclass:: EventTree + :members: + +Models +------ + +.. automodule:: pyflp._models + +Implementing a model +^^^^^^^^^^^^^^^^^^^^ + +A look at the **source code** will definitely help, although these are a few +points that must be kept in mind when Implementing a model: + +1. Does the model mimic the hierarchy exposed by FL Studio's GUI? + + .. tip:: + + Browse through the hierarchies of :class:`pyflp.channel.Channel` + subclasses to get a very good idea of this. + +2. Are ``__dunder__`` methods provided by Python used whenever possible? +3. Is either :class:`ModelReprMixin` subclassed or ``__repr__`` implemented? + +Descriptors +----------- + +.. automodule:: pyflp._descriptors diff --git a/docs/conf.py b/docs/conf.py index 5e399f9..b76543e 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -84,6 +84,7 @@ r"https://pyflp.rtfd.io.*": r"https://pyflp.readthedocs.io/en/latest/.*", r"https://www.python.org/dev/peps/.*": r"https://peps.python.org/.*", r"https://github.com/demberto/PyFLP/files/.*": r"https://objects.githubusercontent.com/.*", + r"https://https://stackoverflow.com/a/.*": r"https://stackoverflow.com/questions/.*", } diff --git a/docs/features.rst b/docs/features.rst index a262e4f..481f8b8 100644 --- a/docs/features.rst +++ b/docs/features.rst @@ -1,5 +1,5 @@ -Features -======== +โœจ Features +============ Non-destructive editing ----------------------- diff --git a/docs/guides.rst b/docs/guides.rst index 874451f..04feb29 100644 --- a/docs/guides.rst +++ b/docs/guides.rst @@ -1,5 +1,5 @@ -Developer guides -================ +๐Ÿ“– Developer guides +==================== Want to be a **contributor**? Interested in the internals of the FLP format? This is the perfect place to begin. Reading :doc:`architecture` is also diff --git a/docs/guides/reversing.rst b/docs/guides/reversing.rst index 3e88a79..e45ea0f 100644 --- a/docs/guides/reversing.rst +++ b/docs/guides/reversing.rst @@ -1,5 +1,5 @@ -Reversing FLP format -==================== +๐Ÿค“ Reversing FLP format +======================== You should first take a look at :doc:`what events are <../architecture>`. A decent knowledge of the topics mentioned there as well as Python itself diff --git a/docs/limitations.rst b/docs/limitations.rst index 457ff6f..1c2a408 100644 --- a/docs/limitations.rst +++ b/docs/limitations.rst @@ -56,19 +56,3 @@ a few.) which is implemented in pure Python. I am all ears if anyone has any suggestions on improving PyFLP's performance. - -Testing can never be truly enough -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -For non dev audiences, testing is a process to check the integrity of code. -Its always done as new features can break existing ones. - -For a long time, I used only a single FLP to test all of PyFLP's API. -Things have changed now and I use presets exported from FL Studio -itself for the testing of a huge chunk of API to ensure **isolation** of test -results. - -The problem is that all the test data comes from FL Studio itself and can -be only really validated in the same. That's the reason I usually don't -raise any errors event if I know quite surely that, for example a value out of -range is set for some property. diff --git a/docs/reference.rst b/docs/reference.rst index 4c0e0f0..591a43c 100644 --- a/docs/reference.rst +++ b/docs/reference.rst @@ -1,5 +1,5 @@ -Reference -========= +๐Ÿงพ Reference +============= .. toctree:: :maxdepth: 2