Upload issues with S3 - unfinished job queues, broken links ...

[vedadkajtaz]

Lychee version

v5.5.1

Which PHP version are you using?

PHP 8.3

Detailed description of the problem

Hello,

I can't get Lychee to work correctly with self-hosted S3 backend.

Issue description

When using the "Upload photos" feature in the browser, although (as per S3 logs) all uploads were successfull, and all objects actually uploaded to the bucket (as per rclone ls), I'm facing the following issues:

• Always: The "Jobs queue" remains open after the upload has finished, and the upload status modal has closed. It always only holds original/... paths. I have to use "Fix job history" to make it go away.

• Frequent: usually when uploading a large number of photos: in the upload status modal, there are always photos (first in queue always among them) then go back from green "Finished" to blue "100%". Also, other photos remain (or revert back from "Finished") at "100%", before the upload modal closes.

When upload starts:

A few moments later:

Despite these, most of the time everything works alright, photos can be browsed, etc. However:

• Almost always, when uploading large (empirically, i'd say over 50) number of photos, and the photos selected for upload contain multiple duplicates (eg. IMG_5575.JPG and IMG_5575 (1).JPG, etc): Some photos (always some of the duplicates) appear as black squares in gallery. The page inspector shows that the urlset uses the APP_URL/uploads/small/... url, instead of using AWS_URL as for the working thumbnails. Moreover, find public/uploads/small/ -type f shows no files on disk (ie. whatever was meant to be uploaded to S3, was uploaded). So, there is definitely some race condition occurring here.

All these problems appear both with sqlite and mysql database backends, both with chrome and firefox. None of them seem to appear when not using S3 backend.

I'm not sure it's related, but the following errors always appear in logs, when uploading (even a single) photos:

[2024-08-17 13:29:59] production.ERROR: League\Flysystem\Filesystem::readStream:79 League\Flysystem\Filesystem::readStream(): Argument #1 ($location) must be of type string, null given, called in /home/lychee/Lychee/vendor/laravel/framework/src/Illuminate/Filesystem/FilesystemAdapter.php on line 627
App\Jobs\UploadSizeVariantToS3Job::handleVideoPartner:91 League\Flysystem\Filesystem::readStream(): Argument #1 ($location) must be of type string, null given, called in /home/lychee/Lychee/vendor/laravel/framework/src/Illuminate/Filesystem/FilesystemAdapter.php on line 627; caused by

I've inserted a log in the readStream($path) method, and there is indeed always a call with $path being null. Here's what gets logged during a single photo upload:

"livewire-tmp/81MRzfQKXZSaN1xhJR61GdPPtzBg05-metaYmxvYg==-."
"thumb/c8/86/a18cde3f9f2ba77189b7d71a3764.jpeg"
"thumb2x/c8/86/a18cde3f9f2ba77189b7d71a3764.jpeg"
"small/c8/86/a18cde3f9f2ba77189b7d71a3764.JPG"
"small2x/c8/86/a18cde3f9f2ba77189b7d71a3764.JPG"
"medium/c8/86/a18cde3f9f2ba77189b7d71a3764.JPG"
"medium2x/c8/86/a18cde3f9f2ba77189b7d71a3764.JPG"
"original/c8/86/a18cde3f9f2ba77189b7d71a3764.JPG"
null

Other

I have a few other minor issues / questions regarding S3 support, I'll create separate issues for them, once this is figured out (as they might be related).

S3 bucket for lychee devs

Please note that I'm willing to provide a permanent access to a S3 bucket to lychee devs for dev/testing, let me know if you're interested.

Steps to reproduce the issue

No response

Diagnostics [REQUIRED]

Diagnostics
-----------
    Warning: /h*********************lic/uploads/ has permissions 0775, but should have 2770
    Warning: /h*********************lic/uploads/medium2x has permissions 0770, but should have 2770
    Warning: /h*********************lic/uploads/medium2x/8b has permissions 0770, but should have 2770
    Warning: /h*********************lic/uploads/medium2x/8b/6f has permissions 0770, but should have 2770
    Warning: /h*********************lic/uploads/medium2x/8b/ca has permissions 0770, but should have 2770
    Warning: 9029 more directories with wrong permissions
    Warning: Dropbox import not working. dropbox_key is empty.


System Information
------------------
    Lychee Version (release):                5.5.1
    DB Version:                              5.5.1
    
    composer install:                        --no-dev
    APP_ENV:                                 production
    APP_DEBUG:                               false
    APP_URL:                                 set
    APP_DIR:                                 default
    LOG_VIEWER_ENABLED:                      true
    LIVEWIRE_ENABLED:                        true
    
    System:                                  FreeBSD
    PHP Version:                             8.3.8
    PHP User agent:                          Lychee/4 (https://lycheeorg.github.io/)
    Timezone:                                Europe/Paris
    Max uploaded file size:                  100M
    Max post size:                           120M
    Livewire chunk size:                     12.00 MB
    Max execution time:                      300
    MySQL Version:                           5.7.42-46-log
    
    exec() Available:                        yes
    Imagick Available:                       1
    Imagick Enabled:                         1
    Imagick Version:                         1809
    GD Version:                              2.3.3
    Number of foreign key:                   12 found.


Config Information
------------------
    version:                                 050501
    check_for_updates:                       0
    sorting_photos_col:                      taken_at
    sorting_photos_order:                    ASC
    sorting_albums_col:                      max_taken_at
    sorting_albums_order:                    ASC
    imagick:                                 1
    skip_duplicates:                         1
    small_max_width:                         0
    small_max_height:                        360
    medium_max_width:                        1920
    medium_max_height:                       1080
    lang:                                    en
    image_overlay_type:                      desc
    default_license:                         none
    compression_quality:                     90
    grants_full_photo_access:                1
    delete_imported:                         0
    mod_frame_enabled:                       1
    mod_frame_refresh:                       30
    thumb_2x:                                1
    small_2x:                                1
    medium_2x:                               1
    landing_page_enable:                     0
    force_migration_in_production:           0
    search_public:                           0
    hide_version_number:                     0
    SL_enable:                               0
    SL_for_admin:                            0
    recent_age:                              1
    grants_download:                         0
    photos_wraparound:                       1
    map_display:                             1
    zip64:                                   1
    map_display_public:                      1
    map_provider:                            RRZE
    force_32bit_ids:                         0
    map_include_subalbums:                   1
    update_check_every_days:                 3
    has_exiftool:                            1
    share_button_visible:                    0
    import_via_symlink:                      0
    has_ffmpeg:                              1
    location_decoding:                       1
    location_decoding_timeout:               30
    location_show:                           1
    location_show_public:                    0
    rss_enable:                              0
    rss_recent_days:                         7
    rss_max_items:                           100
    prefer_available_xmp_metadata:           0
    editor_enabled:                          1
    lossless_optimization:                   0
    swipe_tolerance_x:                       150
    swipe_tolerance_y:                       250
    local_takestamp_video_formats:           .avi|.mov
    log_max_num_line:                        1000
    unlock_password_photos_with_url_param:   0
    nsfw_visible:                            1
    nsfw_blur:                               0
    nsfw_warning:                            0
    nsfw_warning_admin:                      0
    nsfw_banner_override:                    
    map_display_direction:                   1
    album_subtitle_type:                     oldstyle
    upload_processing_limit:                 4
    new_photos_notification:                 0
    legacy_id_redirection:                   1
    zip_deflate_level:                       6
    default_album_protection:                1
    album_decoration:                        layers
    album_decoration_orientation:            row
    allow_username_change:                   0
    auto_fix_orientation:                    1
    use_job_queues:                          0
    random_album_id:                         starred
    use_last_modified_date_when_no_exif_date: 0
    layout:                                  grid
    date_format_photo_thumb:                 M j, Y, g:i:s A e
    date_format_photo_overlay:               M j, Y, g:i:s A e
    date_format_sidebar_uploaded:            M j, Y, g:i:s A e
    date_format_sidebar_taken_at:            M j, Y, g:i:s A e
    date_format_hero_min_max:                F Y
    date_format_hero_created_at:             M j, Y, g:i:s A T
    date_format_album_thumb:                 M Y
    upload_chunk_size:                       0
    nsfw_banner_blur_backdrop:               0
    search_pagination_limit:                 1000
    search_minimum_length_required:          4
    photo_layout_justified_row_height:       320
    photo_layout_masonry_column_width:       300
    photo_layout_grid_column_width:          250
    photo_layout_square_column_width:        200
    photo_layout_gap:                        12
    display_thumb_album_overlay:             always
    display_thumb_photo_overlay:             hover
    default_album_thumb_aspect_ratio:        1/1
    use_album_compact_header:                0
    login_button_position:                   left
    back_button_enabled:                     0
    back_button_text:                        Return to Home
    current_job_processing_visible:          1
    SA_random_thumbs:                        0
    job_processing_queue_visible:            1
    login_required:                          0
    enable_unsorted:                         1
    enable_starred:                          1
    enable_recent:                           1
    enable_on_this_day:                      1
    login_required_root_only:                1

Browser & System [REQUIRED]

System: FreeBSD 14.0, installed from the release v5.5.1 Lychee.zip file.
Browsers: Chrome and Firefox.

Please confirm (incomplete submissions will not be addressed)

• I have provided easy and step-by-step instructions to reproduce the bug.
• I understand my bug report will be removed if I haven't met the criteria above.

[ildyria]

Frequent: usually when uploading a large number of photos: in the upload status modal, there are always photos (first in queue always among them) then go back from green "Finished" to blue "100%". Also, other photos remain (or revert back from "Finished") at "100%", before the upload modal closes.

That one will be fixed on the v6.

Always: The "Jobs queue" remains open after the upload has finished, and the upload status modal has closed. It always only holds original/... paths. I have to use "Fix job history" to make it go away.

This will be reworked in the v6.

Some photos (always some of the duplicates) appear as black squares in gallery. The page inspector shows that the urlset uses the APP_URL/uploads/small/... url, instead of using AWS_URL as for the working thumbnails. Moreover, find public/uploads/small/ -type f shows no files on disk (ie. whatever was meant to be uploaded to S3, was uploaded). So, there is definitely some race condition occurring here.

Seems indeed like it is to the wrong size variants.

By the way, the S3 implementation has been done by @Kovah and to be brutally honest, I have 0 clue how it really works.
#2379

I did try setting up an AWS S3 bucket but coulnd't get it to work. :(

[kiancross]

Always: The "Jobs queue" remains open after the upload has finished, and the upload status modal has closed. It always only holds original/... paths. I have to use "Fix job history" to make it go away.

This issue is indeed caused by the error below:

I'm not sure it's related, but the following errors always appear in logs, when uploading (even a single) photos:

[2024-08-17 13:29:59] production.ERROR: League\Flysystem\Filesystem::readStream:79 ?>League\Flysystem\Filesystem::readStream(): Argument #1 ($location) must be of type string, null given, called in /home/lychee/Lychee/vendor/laravel/framework/src/Illuminate/Filesystem/FilesystemAdapter.php on line 627
App\Jobs\UploadSizeVariantToS3Job::handleVideoPartner:91 League\Flysystem\Filesystem::readStream(): Argument #1 ($location) must be of type string, null given, called in /home/lychee/Lychee/vendor/laravel/framework/src/Illuminate/Filesystem/FilesystemAdapter.php on line 627; caused by

It should be fixed by this PR: #2849.

Almost always, when uploading large (empirically, i'd say over 50) number of photos, and the photos selected for upload contain multiple duplicates (eg. IMG_5575.JPG and IMG_5575 (1).JPG, etc): Some photos (always some of the duplicates) appear as black squares in gallery. The page inspector shows that the urlset uses the APP_URL/uploads/small/... url, instead of using AWS_URL as for the working thumbnails. Moreover, find public/uploads/small/ -type f shows no files on disk (ie. whatever was meant to be uploaded to S3, was uploaded). So, there is definitely some race condition occurring here.

I was unable to reproduce this. In fact, all my duplicates are marked as 'skipped' and not uploaded.

Do you have any more information about this?

[ildyria]

I guess fixed by @kiancross :)