WIP: removing empty FileInstance

[switowski]

No description provided.

[switowski]

For whoever will take over this code, I put some logging (check for logger.warning) in this function:
https://github.com/CERNDocumentServer/cds-videos/blob/cdslabs_qa/cds/modules/webhooks/tasks.py#L567
Like that:

with db.session.begin_nested():
            # Create FileInstance
            file_instance = FileInstance.create()
            logger.warning('FileInstance created')

            # Create ObjectVersion
            obj_key = self._build_subformat_key(preset_quality=preset_quality)
            obj = ObjectVersion.create(bucket=bucket_id, key=obj_key)
            logger.warning('ObjectVersion created')

            # Extract new location
            storage = file_instance.storage(default_location=bucket_location)
            directory, filename = storage._get_fs()

            input_file = self.object.file.uri
            # XRootDPyFS doesn't implement root_path
            try:
                # XRootD Safe
                output_file = os.path.join(
                    directory.root_url + directory.base_path, filename)
            except AttributeError:
                output_file = os.path.join(directory.root_path, filename)

            try:
                # Start Sorenson
                job_id, ar, preset_config = start_encoding(input_file,
                                                           output_file,
                                                           preset_quality,
                                                           aspect_ratio,
                                                           max_height=height,
                                                           max_width=width)
            except (InvalidResolutionError, TooHighResolutionError) as e:
                exception = self._meta_exception_envelope(exc=e)
                self.update_state(state=REVOKED, meta=exception)
                raise Ignore()

            # Set revoke handler, in case of an abrupt execution halt.
            self.set_revoke_handler(partial(stop_encoding, job_id))

            # Create ObjectVersionTags
            ObjectVersionTag.create(obj, 'master', self.obj_id)
            ObjectVersionTag.create(obj, '_sorenson_job_id', job_id)
            ObjectVersionTag.create(obj, 'preset_quality', preset_quality)
            ObjectVersionTag.create(obj, 'media_type', 'video')
            ObjectVersionTag.create(obj, 'context_type', 'subformat')
            ObjectVersionTag.create(obj, 'display_aspect_ratio', ar)
            for key, value in preset_config.items():
                ObjectVersionTag.create(obj, key, value)

            # Information necessary for monitoring
            job_info = dict(
                preset_quality=preset_quality,
                job_id=job_id,
                file_instance=str(file_instance.id),
                uri=output_file,
                version_id=str(obj.version_id),
                key=obj_key,
                tags=obj.get_tags(),
                percentage=0,
            )

        logger.warning('DB session commited')
        db.session.commit()

and this was the output in the celery-transcoding.log:

[2018-07-20 11:34:36,383: WARNING/MainProcess] [email protected] ready.
[2018-07-20 11:37:12,478: WARNING/Worker-1] cds.modules.webhooks.tasks.TranscodeVideoTask[7103e558-2d97-4c09-b103-8abff7f595fa]: FileInstance created
[2018-07-20 11:37:12,478: WARNING/Worker-1] FileInstance created
[2018-07-20 11:37:12,503: WARNING/Worker-3] cds.modules.webhooks.tasks.TranscodeVideoTask[d383041f-41ce-4720-94c9-21ee2689bd30]: FileInstance created
[2018-07-20 11:37:12,503: WARNING/Worker-3] FileInstance created
[2018-07-20 11:37:12,507: WARNING/Worker-4] cds.modules.webhooks.tasks.TranscodeVideoTask[0467d59c-1ef7-4837-a6fd-c3ee79608af8]: FileInstance created
[2018-07-20 11:37:12,507: WARNING/Worker-4] FileInstance created
[2018-07-20 11:37:12,517: WARNING/Worker-5] cds.modules.webhooks.tasks.TranscodeVideoTask[1a2382c3-73a8-4d62-8bbf-c7de23596f6f]: FileInstance created
[2018-07-20 11:37:12,517: WARNING/Worker-5] FileInstance created
[2018-07-20 11:37:12,558: WARNING/Worker-2] cds.modules.webhooks.tasks.TranscodeVideoTask[885ebe8d-1cec-456b-b3df-217baecccccf]: FileInstance created
[2018-07-20 11:37:12,558: WARNING/Worker-2] FileInstance created
[2018-07-20 11:37:12,604: WARNING/Worker-3] cds.modules.webhooks.tasks.TranscodeVideoTask[d383041f-41ce-4720-94c9-21ee2689bd30]: ObjectVersion created
[2018-07-20 11:37:12,604: WARNING/Worker-3] ObjectVersion created
[2018-07-20 11:37:12,659: WARNING/Worker-3] [2018-07-20 11:37:12,658] DEBUG in api: Transcoding file://media-smb.cern.ch/mediacds/test/videos/files/30/b3/d57b-280f-43bc-8bac-c834cd4e835f/data to quality 1080p and aspect ratio 16:9
[2018-07-20 11:37:12,658: DEBUG/Worker-3] Transcoding file://media-smb.cern.ch/mediacds/test/videos/files/30/b3/d57b-280f-43bc-8bac-c834cd4e835f/data to quality 1080p and aspect ratio 16:9
[2018-07-20 11:37:12,879: WARNING/Worker-3] cds.modules.webhooks.tasks.TranscodeVideoTask[d383041f-41ce-4720-94c9-21ee2689bd30]: DB session commited
[2018-07-20 11:37:12,879: WARNING/Worker-3] DB session commited
[2018-07-20 11:37:12,894: WARNING/Worker-1] cds.modules.webhooks.tasks.TranscodeVideoTask[7103e558-2d97-4c09-b103-8abff7f595fa]: ObjectVersion created
[2018-07-20 11:37:12,894: WARNING/Worker-1] ObjectVersion created
[2018-07-20 11:37:12,929: WARNING/Worker-1] [2018-07-20 11:37:12,928] DEBUG in api: Transcoding file://media-smb.cern.ch/mediacds/test/videos/files/30/b3/d57b-280f-43bc-8bac-c834cd4e835f/data to quality 360p and aspect ratio 16:9
[2018-07-20 11:37:12,928: DEBUG/Worker-1] Transcoding file://media-smb.cern.ch/mediacds/test/videos/files/30/b3/d57b-280f-43bc-8bac-c834cd4e835f/data to quality 360p and aspect ratio 16:9
[2018-07-20 11:37:13,067: WARNING/Worker-1] cds.modules.webhooks.tasks.TranscodeVideoTask[7103e558-2d97-4c09-b103-8abff7f595fa]: DB session commited
[2018-07-20 11:37:13,067: WARNING/Worker-1] DB session commited
[2018-07-20 11:37:13,077: WARNING/Worker-4] cds.modules.webhooks.tasks.TranscodeVideoTask[0467d59c-1ef7-4837-a6fd-c3ee79608af8]: ObjectVersion created
[2018-07-20 11:37:13,077: WARNING/Worker-4] ObjectVersion created
[2018-07-20 11:37:13,124: WARNING/Worker-4] [2018-07-20 11:37:13,123] DEBUG in api: Transcoding file://media-smb.cern.ch/mediacds/test/videos/files/30/b3/d57b-280f-43bc-8bac-c834cd4e835f/data to quality 480p and aspect ratio 16:9
[2018-07-20 11:37:13,123: DEBUG/Worker-4] Transcoding file://media-smb.cern.ch/mediacds/test/videos/files/30/b3/d57b-280f-43bc-8bac-c834cd4e835f/data to quality 480p and aspect ratio 16:9
[2018-07-20 11:37:13,271: WARNING/Worker-4] cds.modules.webhooks.tasks.TranscodeVideoTask[0467d59c-1ef7-4837-a6fd-c3ee79608af8]: DB session commited
[2018-07-20 11:37:13,271: WARNING/Worker-4] DB session commited
[2018-07-20 11:37:13,288: WARNING/Worker-5] cds.modules.webhooks.tasks.TranscodeVideoTask[1a2382c3-73a8-4d62-8bbf-c7de23596f6f]: ObjectVersion created
[2018-07-20 11:37:13,288: WARNING/Worker-5] ObjectVersion created
[2018-07-20 11:37:13,335: WARNING/Worker-5] [2018-07-20 11:37:13,334] DEBUG in api: Transcoding file://media-smb.cern.ch/mediacds/test/videos/files/30/b3/d57b-280f-43bc-8bac-c834cd4e835f/data to quality 240p and aspect ratio 16:9
[2018-07-20 11:37:13,334: DEBUG/Worker-5] Transcoding file://media-smb.cern.ch/mediacds/test/videos/files/30/b3/d57b-280f-43bc-8bac-c834cd4e835f/data to quality 240p and aspect ratio 16:9
[2018-07-20 11:37:13,491: WARNING/Worker-5] cds.modules.webhooks.tasks.TranscodeVideoTask[1a2382c3-73a8-4d62-8bbf-c7de23596f6f]: DB session commited
[2018-07-20 11:37:13,491: WARNING/Worker-5] DB session commited
[2018-07-20 11:37:13,504: WARNING/Worker-2] cds.modules.webhooks.tasks.TranscodeVideoTask[885ebe8d-1cec-456b-b3df-217baecccccf]: ObjectVersion created
[2018-07-20 11:37:13,504: WARNING/Worker-2] ObjectVersion created
[2018-07-20 11:37:13,542: WARNING/Worker-2] [2018-07-20 11:37:13,541] DEBUG in api: Transcoding file://media-smb.cern.ch/mediacds/test/videos/files/30/b3/d57b-280f-43bc-8bac-c834cd4e835f/data to quality 720p and aspect ratio 16:9
[2018-07-20 11:37:13,541: DEBUG/Worker-2] Transcoding file://media-smb.cern.ch/mediacds/test/videos/files/30/b3/d57b-280f-43bc-8bac-c834cd4e835f/data to quality 720p and aspect ratio 16:9
[2018-07-20 11:37:13,680: WARNING/Worker-2] cds.modules.webhooks.tasks.TranscodeVideoTask[885ebe8d-1cec-456b-b3df-217baecccccf]: DB session commited
[2018-07-20 11:37:13,680: WARNING/Worker-2] DB session commited

As you can see, the first 5 FileInstance objects are crated, then as soon as the first ObjectVersion finishes, we are commiting changes to the DB (while the FileInstance objects have 0 size, no URL and no checksum). Then if someone deletes the video, the FileInstance objects will stay like that and they won't be removed (I think my PR should fix this).
But also, if the transcoding will get stuck (there is an error in Sorenson), the ObjectVersion is never removed so the empty FileInstance stays as well.