diff --git a/CHANGELOG.md b/CHANGELOG.md index a0b9c37..6a21059 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,41 +1,203 @@ -# CHANGELOG for v1.0.0 +# CHANGELOG for v2.2.2 #### This changelog consists the bug & security fixes and new features being included in the releases listed below. -## **v1.0.1 (17th of January 2024)** - *Release* -* Virtual Product Order Place Issue. -* Add Address issue while checkout. -* Bundle product price details not updating. -* User unable to add virtual product from homepage. -* Push notifications issue fixed -* User Profile image Update issue. +## **v2.2.2 (23rd of October 2024)** - *Release* -## **v1.0.0 (8th of January 2024)** - *Release* +* [Feature] Compatible with Bagisto version 2.2.2 -* [Feature] Multi-locale support +* [Feature] Reorder support -* [Feature] Product share +* [Feature] Subcategories Support -* [Feature] Product Search +* [Feature] Default address Support -* [Feature] Wishlist +* [Feature] Same billing and shipping address support -* [Feature] Compare Product +* [Feature] Inclusive and exclusive tax support -* [Feature] All Type Product Supported +* [Feature] Quantity option in wishlist add to cart support -* [Feature] Multi Currency Support +* [Feature] Subscribe and unsubscribe newsletter support + +* [Feature] Filter option for downloadable products support + +* [Feature] Configurable product details support + +* [Feature] Contact us page support + +## **Bug Fixes** + +* [Fixed] - Getting product in the compare page when new customer register. + +* [Fixed] - Downloadable product sample file is not downloading from the product page. + +* [Fixed] - Show "Please add address" warning message on the address page while checkout if address already added. + +* [Fixed] - Default product and quantity should be select on the bundle product page and total amount and selected products should be visible as per selected options. + +* [Fixed] - Admin added logo and banner image should visible for the category page. + +* [Fixed] - Sorting from a-z or from z-a is not working properly on the catalog page. + +* [Fixed] - Filter is not working properly on the catalog page. + +* [Fixed] - Need to improve the warning message if user trying to register account with already registered email address. + +* [Fixed] - Customer and Guest user is not able to checkout due to shipping methods not coming. + +* [Fixed] - Show warning message if user send the forget password email. + +* [Fixed] - Customer is not able to place order with downloadable product. + +* [Fixed] - Customer is able to add configurable product to cart without selecting size on the product page. + +* [Fixed] - Dark Mode issue on the search page. + +* [Fixed] - #20 Compatibility with latest graphql version + + +## **v2.0.0 (31st of January 2024)** - *Release* + +* [Feature] Compatible with Bagisto version 2.0.0 + +* [Feature] Push Notification + +* [Feature] Multi-locale support * [Feature] Dark Mode Supported -* [Feature] Product Review +* [Feature] Guest Checkout + +* [Feature] Multi Currency Support + +* [Feature] All Type Product Supported * [Feature] Coupons Supported -* [Feature] Guest Checkout +## **Bug Fixes** -* [Feature] Push Notification +* [Fixed] - Show "null review" and product name, price will hide when user refresh the product page. + +* [Fixed] - Show extra products under the unselected category from admin end on the catalog category page. + +* [Fixed] - User should be able to apply any price filter not multiple of 50 on the catalog product filter page. + +* [Fixed] - Need to improve the text and manage space for success message when guest user save address on the address page. + +* [Fixed] - Getting warning message " Null check operator user on a null value" if user use "empty spaces" as coupon code and apply on the cart page. + +* [Fixed] - Need to show the message if shipping methods are not available for particular location on the shipping page. + +* [Fixed] - Product price and subtotal are not visible on the payment page. + +* [Fixed] - Getting empty page with message "Null check operator used on a null value" if guest user click on the "Your order id" button the order confirmation page. + +* [Fixed] - Need to improve the success message when user add the review to the product. + +* [Fixed] - Remove All button is not removing after remove all products from the wishlist. + +* [Fixed] - Need to improve the success message when user remove the coupon code from the cart page. + +* [Fixed] - Applied coupon amount value is not reflecting on the price details on the payment page. + +* [Fixed] - If user place order after adding first time address on the address page then click proceed button then getting warning message. + +* [Fixed] - User added review on product is not visible on the admin end. + +* [Fixed] - Show wrong data at place of email field on the reviews page. + +* [Fixed] - After click "Continue Shopping" button if user again visit the cart page the product is removed. + +* [Fixed] - User is not able to remove the already added products on the compare product page. + +* [Fixed] - Homepage refresh API is not working properly as wishlist status is not updating on the homepage. + +* [Fixed] - User is not able to place order with virtual product getting "Oops server error.Please try again." on the shipping methods page. + +* [Fixed] - If user click on recent products then product page is not open and recent product name is not visible on the homepage. + +* [Fixed] - After order cancel user redirect to the order page then after some time orders are not visible orders page. + +* [Fixed] - Add/Edit address on the address page is not updating immediately on the change address page. + + +## **v1.4.5 (5th of June 2023)** - *Release* + +* [Feature] Compatible with Bagisto version 1.4.5 + +* [Feature] App Performance Enhanced + +* [Feature] voice search + +* [Feature] Add Filters on Order list + +* [Feature] implemented dashboard view + +* [Feature] implement fingerprint login + +* [Feature] implemented Product share + +* [Feature] implement wishlist sharing + +* [Feature] implement sorting on products + +* [Feature] Address filling via google map + +## **Bug Fixes** + +* [Fixed] - When the user removes the coupon from the "Review and checkout" page, the Total amount should get updated. + +* [Fixed] - App is not responding, The menu bar is not responding. + +* [Fixed] - Orders || order quantity is not correct in app. + +* [Fixed] - When the user creates a new account and opens the account information page, some already saved profile picture is visible. + +* [Fixed] - Cart|| After adding the product into the cart when refreshing home page at that time cart is getting empty. + +* [Fixed] - When the user set the profile image, Without clicking on the save button, Profile pic gets saved. + +* [Fixed] - when user create their new account, at that time success message is not correct in app. + +* [Fixed] - Category|| filters ||need to implement "apply" button in filter. + +* [Fixed] - Add Address || When we add an address through the live location the text is shown in English. + +* [Fixed] - guest user|| after added complete address by fetching current location,"country" is not showing correct on review and checkout page in app. + + +## **v1.3.3 (23rd November 2021)** - *Release* + +* [Feature] Compatible with Bagisto version 1.3.3 + +* [Fixed] - Guest user should not be able to add product to the wishlist. + +* [Fixed] - After order placed,the particular product is not visible on order page. + +* [Fixed] - Cross button is not visible on Compare product page. + +* [Fixed] - User click on product on catalog page that product page is not opened. + +* [Fixed] - Order date is showing wrong in the order-list + +* [Fixed] - User is not able to complete order from shipping page. + + +## **v1.3.2 (30th April 2021)** - *Release* + +* [Feature] Compatible with Bagisto version 1.3.2 + +* [Fixed] - If user edit the address then app will add brackets with street field every time. + +* [Fixed] - As a guest user-unable to add a wishlist -getting something went wrong message + +* [Fixed] - Quantity increase and decrease button on product page is not working. + +* [Fixed] - Not able to remove address from address book page. + +* [Fixed] - Unable to show user profile information -* [Feature] Complete Customer end features of ecommerce app +* [Fixed] - Share button is not working on product page. -* [Feature] Compatible with Bagisto v2.0.0 +* [Fixed] - Price details are not correct and showing without a currency symbol on shopping cart page. \ No newline at end of file diff --git a/Configuration_guide.md b/Configuration_guide.md new file mode 100644 index 0000000..a1d7f69 --- /dev/null +++ b/Configuration_guide.md @@ -0,0 +1,85 @@ + +# Configuration Guide:- + +**//--------------------Configuration Guide Bagisto app----------------------//** + +### For setup the application: +- Path-lib/utils/server_configuration.dart +- Change baseUrl, defaultAppTitle, demo login email & password. +- To disable preFetching and caching change isPreFetchingEnable to false. + +``` +static const String baseUrl = "***********"; +static const String demoEmail = "***********"; +static const String demoPassword = "***********"; +static const String defaultAppTitle = "***********"; +``` + +- Path-lib/utils/mobikul_theme.dart +- Change primary & accent color. + +``` +static const Color primaryColor = Color(***********); +static const Color accentColor = Color(***********); +``` + +- Path-assets/language/ +- Change builderAppName with new app name in language string json files (en.json, ar.json, es.json etc) + +  + +## For Application Title: +- **Android** + * Path-android/app/src/main/AndroidManifest.xml + * change app name - **android:label=```"***********"```**. +- **iOS** + * Go to the ios then Runner and open the info.plist file + * Find the key named as **CFBundleDisplayName** and replace the string value to reflect the new app name. + +  + +## For splash screen: +- **Change splash screen with lottie json** + * Path-assets/lottie/ + * replace the **splash_screen**. +- **Or change splash screen with image** + * Path-assets/images/ + * replace the **splash.png**. + * Go to lib/screens/splash_screen/view/splash_screen.dart + * comment the lottie asset splash code and uncomment stack code for asset image splash. + +   + +## For App icon: +- **Android** - open **android** folder right click **app > new > Image Asset** set Image. +- **iOS** - ios/Runner/Assets.xcassets/AppIcon.appiconset + +   + +## For push notification service: +- **Android** - Replace **"google-services.json"**. +- **iOS** - Replace **"GoogleService-Info.plist"**. + +   + +## For Google maps: +- **Android** + * Path-Project/Project name/android/app/src/main/AndroidManifest.xml + * Replace Your Google Map Key For android. +- **iOS** + * Path-ios/Runner/AppDelegate.swift + * GMSServices provideAPIKey:@"Your Google Map Key For ios". + +   + +## For deeplink: +- **Android** + * Path-android/app/src/main/AndroidManifest.xml + * Change host - **android:host=```"***********"```**. + * Change path - **android:pathPrefix=```"***********"```**. +- **iOS** + * Path-ios/Runner/Info.plist + * Find the key named as **CFBundleURLName** and replace with the host name. + * Open ios project in Xcode and go to signing and capabilities expand associated domains and from here replace - **applinks:```***********```** + +   diff --git a/README.md b/README.md index befbb2a..dd8c87d 100644 --- a/README.md +++ b/README.md @@ -21,41 +21,41 @@ Android: https://play.google.com/store/apps/details?id=com.webkul.bagisto.mobiku iOS: https://apps.apple.com/us/app/mobikul-bagisto-laravel-app/id6447519140 # Features -The open-source ecommerce mobile app comes with an array of features to improve your customers' shopping experience. +The open-source ecommerce mobile app comes with an array of features to improve your customers' shopping experience. ## Interactive Home Page and Search -![enter image description here](https://raw.githubusercontent.com/bagisto/temp-media/master/img1.png) +![enter image description here](https://raw.githubusercontent.com/bagisto/temp-media/master/interactive-homepage-and-search.png) ## All Type Product Supported -![enter image description here](https://raw.githubusercontent.com/bagisto/temp-media/master/img2%20(1).png) +![enter image description here](https://raw.githubusercontent.com/bagisto/temp-media/master/product-details.png) ## Dark Mode and Push Notification -![enter image description here](https://raw.githubusercontent.com/bagisto/temp-media/master/img6.png) +![enter image description here](https://raw.githubusercontent.com/bagisto/temp-media/master/dark-theme-and-push-notifications.png) ## Discount Coupons and Guest Checkout -![enter image description here](https://raw.githubusercontent.com/bagisto/temp-media/master/img5.png) +![enter image description here](https://raw.githubusercontent.com/bagisto/temp-media/master/coupon-and-guest-checkout.png) -## Wishlist and Compare Product +## Wishlist and Product Category -![enter image description here](https://raw.githubusercontent.com/bagisto/temp-media/master/img4.png) +![enter image description here](https://raw.githubusercontent.com/bagisto/temp-media/master/category%3Dpage-and-wishlist.png) -## Share and Product Reviews +## Order Details and Product Reviews -![enter image description here](https://raw.githubusercontent.com/bagisto/temp-media/master/img3.png) +![enter image description here](https://raw.githubusercontent.com/bagisto/temp-media/master/order-details-and-product-reviews.png) ## Installation Guide Before beginning with the installation, you will need the following with the mentioned versions -- Bagisto Version - v2.0.0 -- Android Studio Version - Flamingo | 2022.2.1 -- Flutter Version - 3.10.1 -- Dart - 3.0.1 -- Xcode - 14.3 +- Bagisto Version - v2.2.2 +- Android Studio Version - Flamingo | 2022.2.1 +- Flutter Version - 3.19.2 +- Dart - 3.3.0 +- Xcode - 15.2 - Swift - 5 Make sure you have installed the API module and set this up properly on your bagisto. @@ -80,7 +80,7 @@ git clone https://github.com/bagisto/opensource-ecommerce-mobile-app.git ```sh cd ``` - + - Run the following command to install the required packages ```sh @@ -110,7 +110,7 @@ flutter pub run build_runner build --delete-conflicting-outputs * Emulator 1. Start an Android or iOS emulator using your preferred IDE or tools. - + ### Run the Project - Use the following command to build and run the project @@ -120,8 +120,8 @@ flutter run ``` ## Minimum Versions -- Android: 21 -- iOS: 12 +- Android: 22 +- iOS: 14 ## Configurations Steps @@ -134,7 +134,7 @@ Change the baseUrl as per your store ```sh static const String baseUrl = ‘....’; ``` -> Note: Add the value of the complete URL ending with the GraphQL API endpoint. E.g - https://example.com/graphql +> Note: Add the value of the complete URL ending with the GraphQL API endpoint. E.g - https://example.com/graphql ### For Theme @@ -149,10 +149,10 @@ static const Color accentColor = Color(***********); ### For Push Notification Service -- Android +- Android Replace "google-services.json". -- iOS +- iOS Replace "GoogleService-Info.plist". @@ -172,7 +172,7 @@ Replace "GoogleService-Info.plist". * iOS 1. Go to the general tab and identity change the display name to your app name - + > For Homepage Header Title - Go to ‘assets/language/en.json’ > (Note: Here, “en” in en.json refers to the languages that would be supported within the application) @@ -186,7 +186,7 @@ Replace "GoogleService-Info.plist". ```sh static const String splashLottie = "assets/lottie/splash_screen.json"; ``` - + * For adding an Image as a Splash Screen 1. **Path:** assets/images/splash.png @@ -222,4 +222,3 @@ Contributions are welcome! Follow the contribution guidelines to get started. Bagisto is open-sourced software licensed under the MIT license. - diff --git a/android/app/build.gradle b/android/app/build.gradle index 983ee05..59e1104 100644 --- a/android/app/build.gradle +++ b/android/app/build.gradle @@ -28,8 +28,17 @@ apply plugin: 'kotlin-kapt' apply from: "$flutterRoot/packages/flutter_tools/gradle/flutter.gradle" android { + signingConfigs { + release { + storeFile file('jks') + storePassword 'admin' + keyAlias 'admin' + keyPassword 'admin' + } + } + namespace "com.example.bagisto_app_demo" - compileSdkVersion flutter.compileSdkVersion + compileSdkVersion 34 ndkVersion flutter.ndkVersion compileOptions { @@ -41,27 +50,34 @@ android { jvmTarget = '1.8' } + buildFeatures { + dataBinding = true + } + sourceSets { main.java.srcDirs += 'src/main/kotlin' } defaultConfig { - // TODO: Specify your own unique Application ID (https://developer.android.com/studio/build/application-id.html). applicationId "com.webkul.bagisto.mobikul" // You can update the following values to match your application needs. // For more information, see: https://docs.flutter.dev/deployment/android#reviewing-the-gradle-build-configuration. minSdkVersion 22 - targetSdkVersion 33 - versionCode flutterVersionCode.toInteger() - versionName flutterVersionName + targetSdkVersion 34 + versionCode 118 + versionName "2.2.2" } - buildTypes { - release { - // TODO: Add your own signing config for the release build. - // Signing with the debug keys for now, so `flutter run --release` works. + debug { + debuggable true + minifyEnabled false signingConfig signingConfigs.debug } + release { + signingConfig signingConfigs.release + minifyEnabled true + shrinkResources true + } } } @@ -72,8 +88,6 @@ flutter { dependencies { implementation "org.jetbrains.kotlin:kotlin-stdlib-jdk7:$kotlin_version" implementation platform('com.google.firebase:firebase-bom:29.1.0') - - implementation 'com.facebook.android:facebook-android-sdk:8.2.0' implementation 'com.google.ar.sceneform.ux:sceneform-ux:1.17.1' // ML KIT api 'com.google.android.gms:play-services-mlkit-text-recognition:18.0.0' @@ -82,7 +96,6 @@ dependencies { api 'com.google.mlkit:image-labeling-custom:17.0.1' //===Ar Core implementation "com.google.ar.sceneform.ux:sceneform-ux:1.17.1" - // implementation "com.google.ar.sceneform:core:1.17.1" implementation 'com.google.ar.sceneform:assets:1.17.1' implementation "com.google.ar:core:1.30.0" //---------------------------------------// @@ -90,5 +103,4 @@ dependencies { implementation 'androidx.appcompat:appcompat:1.4.1' implementation 'com.github.bumptech.glide:glide:4.12.0' implementation 'com.google.android.material:material:1.5.0' - implementation 'androidx.databinding:databinding-runtime:8.1.2' } diff --git a/android/app/google-services.json b/android/app/google-services.json index 1ce9b97..0f3acfb 100644 --- a/android/app/google-services.json +++ b/android/app/google-services.json @@ -1,43 +1,43 @@ { "project_info": { - "project_number": "1076699867652", - "project_id": "bagisto-mobikul-app", - "storage_bucket": "bagisto-mobikul-app.appspot.com" + "project_number": "10787652", + "project_id": "bagisto", + "storage_bucket": "bagisto" }, "client": [ { "client_info": { - "mobilesdk_app_id": "1:1076699867652:android:7627738ab9478170f06351", + "mobilesdk_app_id": "d:7627738ab9478170f06351", "android_client_info": { - "package_name": "com.webkul.bagisto.marketplace" + "package_name": "bagisto" } }, "oauth_client": [ { - "client_id": "1076699867652-6nijb4tlissq470kebt6ij3bh4drm3fb.apps.googleusercontent.com", + "client_id": "rm3fb.apps.googleusercontent.com", "client_type": 3 } ], "api_key": [ { - "current_key": "AIzaSyD9oLzksLYsYpJnPvW9-5oXYo2jTKzGt_A" + "current_key": "ksLYsYpJnPvW9-5oXYo2jTKzGt_A" }, { - "current_key": "AIzaSyDL5pg-OdSNY31ekS9CjVEbk9_pK0j1avU" + "current_key": "A-OdSNY31ekS9CjVEbk9_pK0j1avU" } ], "services": { "appinvite_service": { "other_platform_oauth_client": [ { - "client_id": "1076699867652-6nijb4tlissq470kebt6ij3bh4drm3fb.apps.googleusercontent.com", + "client_id": "107nijb4tlissq470kebt6ij3bh4drm3fb.apps.googleusercontent.com", "client_type": 3 }, { - "client_id": "1076699867652-cofvnvlt7i0bfbp9h7n66jsvkllpmuqq.apps.googleusercontent.com", + "client_id": "1076699t7i0bfbp9hsvkllpmuqq.apps.googleusercontent.com", "client_type": 2, "ios_info": { - "bundle_id": "com.webkul.bagisto.iOS" + "bundle_id": "bagisto.iOS" } } ] @@ -46,53 +46,53 @@ }, { "client_info": { - "mobilesdk_app_id": "1:1076699867652:android:dc081e4de9130a48f06351", + "mobilesdk_app_id": "1:10766:dc081e4de9130a48f06351", "android_client_info": { - "package_name": "com.webkul.bagisto.mobikul" + "package_name": "bagisto" } }, "oauth_client": [ { - "client_id": "1076699867652-ggevn4e5hp8qvcd8d64f9r1rv3h0ecva.apps.googleusercontent.com", + "client_id": "107669988qvcd8d64f9.apps.googleusercontent.com", "client_type": 1, "android_info": { - "package_name": "com.webkul.bagisto.mobikul", - "certificate_hash": "742ec4289323a0b2cc7e731b1fc7011a46869290" + "package_name": "bagisto", + "certificate_hash": "742ec421b1fc7011a46869290" } }, { - "client_id": "1076699867652-o456sfuqsbcan4svdhpt6rg12loq9jbq.apps.googleusercontent.com", + "client_id": "1076699sbcan4svdhpt6rgq.apps.googent.com", "client_type": 1, "android_info": { - "package_name": "com.webkul.bagisto.mobikul", - "certificate_hash": "61a253a049223dac296ae6ac3a733f744db12235" + "package_name": "bagisto", + "certificate_hash": "61a3a733f744db12235" } }, { - "client_id": "1076699867652-6nijb4tlissq470kebt6ij3bh4drm3fb.apps.googleusercontent.com", + "client_id": "107669986kebt6ij3bh4drm3fb.apps.googleusercontent.com", "client_type": 3 } ], "api_key": [ { - "current_key": "AIzaSyD9oLzksLYsYpJnPvW9-5oXYo2jTKzGt_A" + "current_key": "AIzaSyDnPvW9-5oXYo2jTKzGt_A" }, { - "current_key": "AIzaSyDL5pg-OdSNY31ekS9CjVEbk9_pK0j1avU" + "current_key": "AIzaSyDCjVEbk9_pK0j1avU" } ], "services": { "appinvite_service": { "other_platform_oauth_client": [ { - "client_id": "1076699867652-6nijb4tlissq470kebt6ij3bh4drm3fb.apps.googleusercontent.com", + "client_id": "1076699860kebt6ij3bh4drm3fb.apps.googl", "client_type": 3 }, { - "client_id": "1076699867652-cofvnvlt7i0bfbp9h7n66jsvkllpmuqq.apps.googleusercontent.com", + "client_id": "1076699867n66jsvkllpmuqq.apps.gooontent.com", "client_type": 2, "ios_info": { - "bundle_id": "com.webkul.bagisto.iOS" + "bundle_id": "bagisto" } } ] diff --git a/android/app/src/debug/AndroidManifest.xml b/android/app/src/debug/AndroidManifest.xml index 399f698..ef7ddb9 100644 --- a/android/app/src/debug/AndroidManifest.xml +++ b/android/app/src/debug/AndroidManifest.xml @@ -1,3 +1,13 @@ + + + @@ -16,6 +26,7 @@ + + + + + + + + + + + android:host="" + android:pathPrefix="" + android:scheme="https" /> + + + + + + + + - methodChannelResult = result; - if (call.method == "initialLink") { - val initialUrl = initialLink() - if (initialUrl != null) { - result.success(initialLink()) - } else { - result.error("UNAVAILABLE", "No deep link found", null) + try { + methodChannelResult = result; + if (call.method == "initialLink") { + val initialUrl = initialLink() + if (initialUrl != null) { + result.success(initialLink()) + } else { + result.error("UNAVAILABLE", "No deep link found", null) + } } - } - else if (call.method.equals("fileviewer")) { - var path: String = call.arguments()!! - var file = File(path) - val extension = FilenameUtils.getExtension(path) + else if (call.method.equals("fileviewer")) { + var path: String = call.arguments()!! + var file = File(path) + val extension = FilenameUtils.getExtension(path) - val photoURI: Uri = FileProvider.getUriForFile( - this.applicationContext, - "com.webkul.bagisto.mobikul.provider", - file - ) - val intent = Intent(Intent.ACTION_VIEW) - intent.setFlags(Intent.FLAG_ACTIVITY_NO_HISTORY) - intent.addFlags(Intent.FLAG_GRANT_READ_URI_PERMISSION) - intent.setDataAndType(photoURI, "application/" + extension) - if (intent.resolveActivity(getPackageManager()) != null) { - //if device have requested extension app then respective app will open - println("File extension app found in the device") - startActivity(intent); - } - else { - //if device don't have requested extension app then all app option will be visible. - println("File extension app Not found in the device") + val photoURI: Uri = FileProvider.getUriForFile( + this.applicationContext, + "com.webkul.bagisto.mobikul.provider", + file + ) + val intent = Intent(Intent.ACTION_VIEW) + intent.setFlags(Intent.FLAG_ACTIVITY_NO_HISTORY) + intent.addFlags(Intent.FLAG_GRANT_READ_URI_PERMISSION) + intent.setDataAndType(photoURI, "application/" + extension) + if (intent.resolveActivity(getPackageManager()) != null) { + //if device have requested extension app then respective app will open + println("File extension app found in the device") + startActivity(intent); + } + else { + //if device don't have requested extension app then all app option will be visible. + println("File extension app Not found in the device") + + intent.setDataAndType(photoURI, "*/*") + startActivity(intent); + } + result.success(true) - intent.setDataAndType(photoURI, "*/*") - startActivity(intent); } - result.success(true) + else if (call.method == "imageSearch") { + startImageFinding() + } else if (call.method == "textSearch") { + startTextFinding() + } else if (call.method == "showAr") { + if (call.hasArgument("url")) { + Log.d("qdaasdas", call.argument("url") ?: "No Name") + } + showArActivity(call.argument("name"), call.argument("url")) + } + }catch (e: Exception) { + print(e) } - } } -// override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) { -// super.onActivityResult(requestCode, resultCode, data) -// if (requestCode == 101 && resultCode == Activity.RESULT_OK) { -// methodChannelResult?.success(data?.getStringExtra(CameraSearchActivity.CAMERA_SEARCH_HELPER)) -// } -// } + override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) { + super.onActivityResult(requestCode, resultCode, data) + if (requestCode == 101 && resultCode == Activity.RESULT_OK) { + methodChannelResult?.success(data?.getStringExtra(CameraSearchActivity.CAMERA_SEARCH_HELPER)) + } + } fun initialLink(): String? { val uri = intent.data @@ -82,4 +100,35 @@ class MainActivity : FlutterFragmentActivity() { } } + private fun startImageFinding() { + val intent = Intent(this, CameraSearchActivity::class.java) + intent.putExtra( + CameraSearchActivity.CAMERA_SELECTED_MODEL, + CameraSearchActivity.IMAGE_LABELING + ) + startActivityForResult(intent, 101) + } + + private fun startTextFinding() { + val intent = Intent(this, CameraSearchActivity::class.java) + intent.putExtra( + CameraSearchActivity.CAMERA_SELECTED_MODEL, + CameraSearchActivity.TEXT_RECOGNITION + ) + startActivityForResult(intent, 101) + } + private fun showArActivity(name: String?, url: String?){ + val availability = ArCoreApk.getInstance().checkAvailability(this) + if (availability.isSupported) { + Log.d("TEST_LOG", "${name}----${url}") + val intent = Intent(this, ArActivity::class.java) + intent.putExtra("name", name) + intent.putExtra("link", url) + startActivity(intent) + // methodChannelResult?.success("Supported "); + } else { + methodChannelResult?.error("401", "Your device does not support AR feature",""); + } + } + } diff --git a/android/app/src/main/kotlin/webkul/bagisto_app_demo/arcore/activities/ArActivity.kt b/android/app/src/main/kotlin/webkul/bagisto_app_demo/arcore/activities/ArActivity.kt new file mode 100644 index 0000000..973074d --- /dev/null +++ b/android/app/src/main/kotlin/webkul/bagisto_app_demo/arcore/activities/ArActivity.kt @@ -0,0 +1,199 @@ +/* + * Webkul Software. + * + * Kotlin + * + * @author Webkul + * @category Webkul + * @package com.webkul.mobikul + * @copyright 2010-2018 Webkul Software Private Limited (https://webkul.com) + * @license https://store.webkul.com/license.html ASL Licence + * @link https://store.webkul.com/license.html + */ + +package webkul.bagisto_app_demo.arcore.activities + +import android.Manifest +import android.app.Activity +import android.app.ActivityManager +import android.content.Context +import android.content.pm.PackageManager +import android.net.Uri +import android.os.Build +import android.os.Bundle +import android.text.SpannableString +import androidx.core.content.ContextCompat +import androidx.appcompat.app.AppCompatActivity +import android.util.Log +import android.view.MotionEvent +import android.widget.Toast +import androidx.databinding.DataBindingUtil +import com.example.bagisto_app_demo.databinding.ActivityArBinding +import com.example.bagisto_app_demo.R +import com.google.android.material.snackbar.Snackbar +import com.google.ar.core.HitResult +import com.google.ar.core.Plane +import com.google.ar.core.Session +import com.google.ar.sceneform.AnchorNode +import com.google.ar.sceneform.assets.RenderableSource +import com.google.ar.sceneform.rendering.ModelRenderable +import com.google.ar.sceneform.ux.ArFragment +import com.google.ar.sceneform.ux.TransformableNode +import java.util.concurrent.CompletableFuture + + +class ArActivity : AppCompatActivity() { + private val MIN_OPENGL_VERSION = 3.1 + + val RC_AR = 1011 + private var arModel: String? = null + private lateinit var mContentViewBinding: ActivityArBinding + private var arFragment: ArFragment? = null + private var objectRenderable: ModelRenderable? = null + private var mUserRequestedInstall = false + private var mSession: Session? = null + + var anchorNode: AnchorNode? = null + private var mModelStateSnackBar: Snackbar? = null + private lateinit var mModelCompletableFuture: CompletableFuture + + override fun onCreate(savedInstanceState: Bundle?) { + super.onCreate(savedInstanceState) + mContentViewBinding = DataBindingUtil.setContentView(this, R.layout.activity_ar) + arModel = intent.getStringExtra("link") + initSupportActionBar() + if (checkIsSupportedDeviceOrFinish(this)) { + + startInitialization() + } else { + Toast.makeText( + this, + getString(R.string.the_ar_feature_is_not_supported_by_your_device), + Toast.LENGTH_SHORT + ).show() + this.finish() + } + } + + fun initSupportActionBar() { + setSupportActionBar(mContentViewBinding.toolbar) + val title = SpannableString(intent.getStringExtra("name") ?: "") + supportActionBar?.title = title + supportActionBar?.elevation = 4f + } + private fun startInitialization() { + try { + if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.N) { + if (ContextCompat.checkSelfPermission(this, Manifest.permission.WRITE_EXTERNAL_STORAGE) == PackageManager.PERMISSION_GRANTED && ContextCompat.checkSelfPermission(this, Manifest.permission.READ_EXTERNAL_STORAGE) == PackageManager.PERMISSION_GRANTED) { + try { + + arFragment = supportFragmentManager.findFragmentById(R.id.ux_fragment) as ArFragment + + + Toast.makeText(this@ArActivity, getString(R.string.downloading_model), Toast.LENGTH_SHORT).show() + + // Init renderable + loadModel() + + // Set tap listener + arFragment!!.setOnTapArPlaneListener { hitResult: HitResult, plane: Plane?, motionEvent: MotionEvent? -> + val anchor = hitResult.createAnchor() + if (anchorNode == null) { + anchorNode = AnchorNode(anchor) + anchorNode?.setParent(arFragment!!.arSceneView.scene) + createModel() + } + } + } catch (e: Exception) { + Log.d("TAG", e.printStackTrace().toString() + e.message.toString()) + e.printStackTrace() + } + } else { + val permissions = arrayOf(Manifest.permission.WRITE_EXTERNAL_STORAGE, Manifest.permission.READ_EXTERNAL_STORAGE) + requestPermissions(permissions, RC_AR) + } + } + } catch (e: java.lang.Exception) { + Log.d("TAG", e.printStackTrace().toString() + e.message.toString()) + } + } + + private fun createModel() { + try { + if (anchorNode != null) { + val node = TransformableNode(arFragment!!.transformationSystem) + node.scaleController.maxScale = 1.0f + node.scaleController.minScale = 0.01f + node.scaleController.sensitivity = 0.1f + node.setParent(anchorNode) + node.renderable = objectRenderable + + node.select() + mModelStateSnackBar = Snackbar.make(mContentViewBinding.arLayout, getString(R.string.model_ready), Snackbar.LENGTH_INDEFINITE).setAction(getString(R.string.dismiss)) { + mModelStateSnackBar?.dismiss() + } + } + } catch (e: Exception) { + Toast.makeText(this@ArActivity, getString(R.string.something_went_wrong), Toast.LENGTH_SHORT).show() + + } + } + + private fun loadModel() { + try { + if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.N) { + ModelRenderable.builder() + .setSource(this, + RenderableSource.builder().setSource( + this, + Uri.parse(arModel), + RenderableSource.SourceType.GLB) + .setScale(0.75f) + .setRecenterMode(RenderableSource.RecenterMode.ROOT) + .build()) + .setRegistryId(arModel) + .build() + .thenAccept { renderable: ModelRenderable -> + objectRenderable = renderable + Toast.makeText(this@ArActivity, getString(R.string.model_ready), Toast.LENGTH_SHORT).show() + + } + .exceptionally { throwable: Throwable? -> + // Log.i("Model", "cant load") + Toast.makeText(this@ArActivity, getString(R.string.model_error)+throwable?.message, Toast.LENGTH_SHORT).show() + mModelStateSnackBar = Snackbar.make(mContentViewBinding.arLayout, getString(R.string.model_error), Snackbar.LENGTH_INDEFINITE).setAction(getString(R.string.try_again)) { + mModelStateSnackBar?.dismiss() + } + null + } + } + } catch (e: Exception) { + Toast.makeText(this@ArActivity, getString(R.string.model_error)+e?.message, Toast.LENGTH_SHORT).show() + e.printStackTrace() + } + } + + + + private fun checkIsSupportedDeviceOrFinish(activity: Activity): Boolean { + if (Build.VERSION.SDK_INT < Build.VERSION_CODES.N) { + Log.e("error", "Sceneform requires Android N or later") + return false + } + val openGlVersionString = + (activity.getSystemService(Context.ACTIVITY_SERVICE) as ActivityManager) + .deviceConfigurationInfo + .glEsVersion + if (java.lang.Double.parseDouble(openGlVersionString) < MIN_OPENGL_VERSION) { + Log.e("error", "Sceneform requires OpenGL ES 3.1 later") + return false + } + return true + } + + override fun onDestroy() { + super.onDestroy() + if (::mModelCompletableFuture.isInitialized) + mModelCompletableFuture.cancel(true) + } +} \ No newline at end of file diff --git a/android/app/src/main/kotlin/webkul/bagisto_app_demo/arcore/helper/CameraPermissionHelper.kt b/android/app/src/main/kotlin/webkul/bagisto_app_demo/arcore/helper/CameraPermissionHelper.kt new file mode 100644 index 0000000..d6f2ddb --- /dev/null +++ b/android/app/src/main/kotlin/webkul/bagisto_app_demo/arcore/helper/CameraPermissionHelper.kt @@ -0,0 +1,69 @@ +/* + * Webkul Software. + * + * Kotlin + * + * @author Webkul + * @category Webkul + * @package com.webkul.mobikul + * @copyright 2010-2018 Webkul Software Private Limited (https://webkul.com) + * @license https://store.webkul.com/license.html ASL Licence + * @link https://store.webkul.com/license.html + */ + +package webkul.bagisto_app_demo.arcore.helper + +import android.Manifest +import android.app.Activity +import android.content.Intent +import android.content.pm.PackageManager +import android.net.Uri +import android.provider.Settings +import androidx.core.app.ActivityCompat +import androidx.core.content.ContextCompat + + +class CameraPermissionHelper { + + companion object { + + private val CAMERA_PERMISSION_CODE = 0 + private val CAMERA_PERMISSION = Manifest.permission.CAMERA + + /** + * Check to see we have the necessary permissions for this app. + */ + fun hasCameraPermission(activity: Activity): Boolean { + return ContextCompat.checkSelfPermission( + activity, + CAMERA_PERMISSION + ) == PackageManager.PERMISSION_GRANTED + } + + /** + * Check to see we have the necessary permissions for this app, and ask for them if we don't. + */ + fun requestCameraPermission(activity: Activity) { + ActivityCompat.requestPermissions( + activity, arrayOf(CAMERA_PERMISSION), CAMERA_PERMISSION_CODE + ) + } + } + + /** + * Check to see if we need to show the rationale for this permission. + */ + fun shouldShowRequestPermissionRationale(activity: Activity): Boolean { + return ActivityCompat.shouldShowRequestPermissionRationale(activity, CAMERA_PERMISSION) + } + + /** + * Launch Application Setting to grant permission. + */ + fun launchPermissionSettings(activity: Activity) { + val intent = Intent() + intent.action = Settings.ACTION_APPLICATION_DETAILS_SETTINGS + intent.data = Uri.fromParts("package", activity.packageName, null) + activity.startActivity(intent) + } +} \ No newline at end of file diff --git a/android/app/src/main/kotlin/webkul/bagisto_app_demo/arcore/helper/CameraPreview.kt b/android/app/src/main/kotlin/webkul/bagisto_app_demo/arcore/helper/CameraPreview.kt new file mode 100644 index 0000000..682e32f --- /dev/null +++ b/android/app/src/main/kotlin/webkul/bagisto_app_demo/arcore/helper/CameraPreview.kt @@ -0,0 +1,96 @@ +/* + * Webkul Software. + * + * Kotlin + * + * @author Webkul + * @category Webkul + * @package com.webkul.mobikul + * @copyright 2010-2018 Webkul Software Private Limited (https://webkul.com) + * @license https://store.webkul.com/license.html ASL Licence + * @link https://store.webkul.com/license.html + */ + +package webkul.bagisto_app_demo.arcore.helper + +import android.content.Context +import android.hardware.Camera +import android.util.AttributeSet +import android.util.Log +import android.view.SurfaceHolder +import android.view.SurfaceView +import java.io.IOException + + +class CameraPreview : SurfaceView, SurfaceHolder.Callback { + + private lateinit var mHolder: SurfaceHolder + private lateinit var mCamera: Camera + + constructor(context: Context) : this(context, null) + + constructor(context: Context, attrs: AttributeSet?) : this(context, attrs, 0) + + constructor(context: Context, camera: Camera) : this(context, null, 0) { + mCamera = camera + mCamera.setDisplayOrientation(90) + // Install a SurfaceHolder.Callback so we get notified when the + // underlying surface is created and destroyed. + mHolder = holder + mHolder.addCallback(this) + // deprecated setting, but required on Android versions prior to 3.0 + mHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS) + } + + constructor(context: Context, attrs: AttributeSet?, defStyleAttr: Int) : super( + context, + attrs, + defStyleAttr + ) + + override fun surfaceCreated(holder: SurfaceHolder) { + // The Surface has been created, now tell the camera where to draw the preview. + try { + mCamera.setPreviewDisplay(holder) + mCamera.startPreview() + } catch (e: IOException) { + e.printStackTrace() + Log.d("error", "Error setting camera preview: " + e.message) + } catch (e: java.lang.Exception) { + e.printStackTrace() + } + } + + override fun surfaceDestroyed(holder: SurfaceHolder) { + // empty. Take care of releasing the Camera preview in your activity. + } + + override fun surfaceChanged(holder: SurfaceHolder, format: Int, w: Int, h: Int) { + // If your preview can change or rotate, take care of those events here. + // Make sure to stop the preview before resizing or reformatting it. + + if (mHolder.surface == null) { + // preview surface does not exist + return + } + + // stop preview before making changes + try { + mCamera.stopPreview() + } catch (e: Exception) { + // ignore: tried to stop a non-existent preview + } + + // set preview size and make any resize, rotate or + // reformatting changes here + + // start preview with new settings + try { + mCamera.setPreviewDisplay(mHolder) + mCamera.startPreview() + + } catch (e: Exception) { + Log.d("error", "Error starting camera preview: " + e.message) + } + } +} \ No newline at end of file diff --git a/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/activities/CameraSearchActivity.kt b/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/activities/CameraSearchActivity.kt new file mode 100644 index 0000000..f6c75b8 --- /dev/null +++ b/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/activities/CameraSearchActivity.kt @@ -0,0 +1,401 @@ +/* + * Copyright 2020 Google LLC. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package webkul.bagisto_app_demo.mlkit.activities + +import android.Manifest +import android.content.Context +import android.content.Intent +import android.content.pm.PackageManager +import android.hardware.Camera +import android.hardware.Camera.CameraInfo +import android.os.Bundle +import android.util.Log +import android.view.Menu +import android.view.View +import android.widget.* +import androidx.appcompat.app.AppCompatActivity +import androidx.core.app.ActivityCompat +import androidx.core.content.ContextCompat +import androidx.databinding.DataBindingUtil +import androidx.recyclerview.widget.LinearLayoutManager +import com.example.bagisto_app_demo.R +import com.example.bagisto_app_demo.databinding.ActivityCameraSearchBinding +import com.google.android.gms.common.annotation.KeepName +import com.google.mlkit.vision.label.ImageLabel +import com.google.mlkit.vision.label.defaults.ImageLabelerOptions +import com.google.mlkit.vision.text.Text +import webkul.bagisto_app_demo.mlkit.adapters.CameraSearchResultAdapter +import webkul.bagisto_app_demo.mlkit.customviews.CameraSource +import webkul.bagisto_app_demo.mlkit.labeldetector.LabelDetectorProcessor +import webkul.bagisto_app_demo.mlkit.textdetector.TextRecognitionProcessor +import java.io.IOException + +/** Live preview demo for ML Kit APIs. */ +@KeepName +class CameraSearchActivity : AppCompatActivity(), ActivityCompat.OnRequestPermissionsResultCallback, + CompoundButton.OnCheckedChangeListener { + lateinit var mContentViewBinding: ActivityCameraSearchBinding + private var cameraSource: CameraSource? = null + private var resultNumberTv: TextView? = null + private var displayAdapter: CameraSearchResultAdapter? = null + private var selectedModel = TEXT_RECOGNITION + lateinit var displayList: ArrayList + private var hasFlash = false + var TAG = "CameraActivity" + + companion object { + const val CAMERA_SEARCH_HELPER = "searchHelper" + const val TEXT_RECOGNITION = "Text Detection" + const val IMAGE_LABELING = "Product Detection" + private const val TAG = "LivePreviewActivity" + private const val PERMISSION_REQUESTS = 1 + private fun isPermissionGranted( + context: Context, + permission: String?, + ): Boolean { + if (ContextCompat.checkSelfPermission( + context, + permission!! + ) == PackageManager.PERMISSION_GRANTED + ) { + Log.i(TAG, "Permission granted: $permission") + return true + } + Log.i(TAG, "Permission NOT granted: $permission") + return false + } + + var CAMERA_SELECTED_MODEL = "CAMERA_SELECTED_MODEL" + } + + override fun onCreate(savedInstanceState: Bundle?) { + super.onCreate(savedInstanceState) + mContentViewBinding = DataBindingUtil.setContentView(this, R.layout.activity_camera_search) + displayList = ArrayList() + + if (intent.hasExtra(CAMERA_SELECTED_MODEL)) { + selectedModel = intent.getStringExtra(CAMERA_SELECTED_MODEL)!! + } + if (ContextCompat.checkSelfPermission(this, Manifest.permission.CAMERA) + == PackageManager.PERMISSION_DENIED + ) { + ActivityCompat.requestPermissions(this, arrayOf(Manifest.permission.CAMERA), 200) + onBackPressed() + } + + mContentViewBinding.resultsMessageTv.text = getString(R.string.x_results_found, displayList.size) + mContentViewBinding.resultRv.layoutManager = + LinearLayoutManager(this, LinearLayoutManager.VERTICAL, false) + displayAdapter = CameraSearchResultAdapter(this, displayList) + mContentViewBinding.resultRv.adapter = displayAdapter + + mContentViewBinding.resultRv.bringToFront() + + initSupportActionBar() + + cameraSwitchSetup() + + flashSetup() + + + if (allPermissionsGranted()) { + mContentViewBinding.previewView.stop() + createCameraSource(selectedModel) + startCameraSource() + } else { + runtimePermissions + } + + + } + + override fun onCreateOptionsMenu(menu: Menu): Boolean { + return true + } + + fun initSupportActionBar() { + if (supportActionBar != null) { + supportActionBar!!.setDisplayHomeAsUpEnabled(false) + supportActionBar!!.setHomeButtonEnabled(false) + supportActionBar!!.title = selectedModel + } + } + + private fun cameraSwitchSetup() { + if (hasFrontCamera()) { + mContentViewBinding.facingSwitch.setOnCheckedChangeListener { buttonView, isChecked -> + if (cameraSource != null) { + if (isChecked) { + mContentViewBinding.flashSwitch.visibility = View.GONE + cameraSource!!.setFacing(CameraSource.CAMERA_FACING_FRONT) + } else { + flashSetup() + cameraSource!!.setFacing(CameraSource.CAMERA_FACING_BACK) + } + + } + mContentViewBinding.previewView.stop() + displayList.clear() + mContentViewBinding.resultRv.adapter?.notifyDataSetChanged() + startCameraSource() + } + } else { + mContentViewBinding.facingSwitch.isEnabled = false + mContentViewBinding.facingSwitch.isChecked = false + mContentViewBinding.facingSwitch.visibility = View.GONE + } + } + + private fun flashSetup() { + hasFlash = + applicationContext.packageManager.hasSystemFeature(PackageManager.FEATURE_CAMERA_FLASH) + + if (hasFlash) { + mContentViewBinding.flashSwitch.visibility = View.VISIBLE + mContentViewBinding.flashSwitch.setOnCheckedChangeListener { buttonView, isChecked -> + if (cameraSource != null) { + val camera = cameraSource!!.camera + if (camera != null) { + val parameters = camera!!.parameters + if (isChecked) { + parameters.flashMode = Camera.Parameters.FLASH_MODE_TORCH + } else { + parameters.flashMode = Camera.Parameters.FLASH_MODE_OFF + } + + camera.parameters = parameters + } + } else { + Toast.makeText( + this@CameraSearchActivity, + getString(R.string.error_while_using_flash), + Toast.LENGTH_LONG + ).show() + } + } + } else { + mContentViewBinding.flashSwitch.visibility = View.GONE + } + } + + + private fun hasFrontCamera(): Boolean { + val cameraInfo = CameraInfo() + val numberOfCameras = Camera.getNumberOfCameras() + for (i in 0 until numberOfCameras) { + Camera.getCameraInfo(i, cameraInfo) + if (cameraInfo.facing == CameraInfo.CAMERA_FACING_FRONT) { + return true + } + } + return false + } + + override fun onCheckedChanged(buttonView: CompoundButton, isChecked: Boolean) { + Log.d(TAG, "Set facing") + if (cameraSource != null) { + if (isChecked) { + cameraSource?.setFacing(CameraSource.CAMERA_FACING_FRONT) + } else { + cameraSource?.setFacing(CameraSource.CAMERA_FACING_BACK) + } + } + mContentViewBinding.previewView.stop() + startCameraSource() + } + + private fun createCameraSource(model: String) { + // If there's no existing cameraSource, create one. + if (cameraSource == null) { + cameraSource = CameraSource(this, mContentViewBinding.graphicOverlay) + } + try { + when (model) { + TEXT_RECOGNITION -> { + Log.i( + TAG, + "Using on-device Text recognition Processor" + ) + cameraSource!!.setMachineLearningFrameProcessor(TextRecognitionProcessor(this)) + displayList.clear() + } + IMAGE_LABELING -> { + Log.i( + TAG, + "Using Image Label Detector Processor" + ) + cameraSource!!.setMachineLearningFrameProcessor( + LabelDetectorProcessor( + this, + ImageLabelerOptions.DEFAULT_OPTIONS + ) + ) + displayList.clear() + } + else -> Log.e(TAG, "Unknown model: $model") + } + } catch (e: Exception) { + Log.e(TAG, "Can not create image processor: $model", e) + Toast.makeText( + applicationContext, "Can not create image processor: " + e.message, + Toast.LENGTH_LONG + ).show() + } + } + + /** + * Starts or restarts the camera source, if it exists. If the camera source doesn't exist yet + * (e.g., because onResume was called before the camera source was created), this will be called + * again when the camera source is created. + */ + private fun startCameraSource() { + if (cameraSource != null) { + try { + mContentViewBinding.previewView.start( + cameraSource!!, + mContentViewBinding.graphicOverlay + ) + } catch (e: IOException) { + Log.e(TAG, "Unable to start camera source.", e) + cameraSource!!.release() + cameraSource = null + } + } + } + + public override fun onResume() { + super.onResume() + Log.d(TAG, "onResume") + createCameraSource(selectedModel) + startCameraSource() + } + + /** Stops the camera. */ + override fun onPause() { + super.onPause() + mContentViewBinding.previewView.stop() + } + + public override fun onDestroy() { + super.onDestroy() + if (cameraSource != null) { + cameraSource?.release() + } + } + + private val requiredPermissions: Array + get() = try { + val info = this.packageManager + .getPackageInfo(this.packageName, PackageManager.GET_PERMISSIONS) + val ps = info.requestedPermissions + if (ps != null && ps.isNotEmpty()) { + ps + } else { + arrayOfNulls(0) + } + } catch (e: Exception) { + arrayOfNulls(0) + } + + private fun allPermissionsGranted(): Boolean { + for (permission in requiredPermissions) { + if (!isPermissionGranted(this, permission)) { + return false + } + } + return true + } + + private val runtimePermissions: Unit + get() { + val allNeededPermissions: MutableList = ArrayList() + for (permission in requiredPermissions) { + if (!isPermissionGranted(this, permission)) { + allNeededPermissions.add(permission) + } + } + if (allNeededPermissions.isNotEmpty()) { + ActivityCompat.requestPermissions( + this, + allNeededPermissions.toTypedArray(), + PERMISSION_REQUESTS + ) + } + } + + + override fun onRequestPermissionsResult( + requestCode: Int, + permissions: Array, + grantResults: IntArray + ) { + if (allPermissionsGranted()) { + createCameraSource(selectedModel) + startCameraSource() + } + super.onRequestPermissionsResult(requestCode, permissions, grantResults) + } + + + fun sendResultBack(position: Int) { + val resultIntent = Intent() + resultIntent.putExtra(CAMERA_SEARCH_HELPER, displayList[position]) + this.setResult(RESULT_OK, resultIntent) + this.finish() + } + + var contentChange = true + fun updateSpinnerFromTextResults(textResults: Text) { + val blocks: List = textResults.textBlocks + for (eachBlock in blocks) { + for (eachLine in eachBlock.lines) { + for (eachElement in eachLine.elements) { + if (!displayList.contains(eachElement.text) && displayList.size <= 9) { + displayList.add(eachElement.text) + contentChange = true + } + } + } + } + if (contentChange) { + mContentViewBinding.resultsMessageTv.text = + getString(R.string.x_results_found, displayList.size) + mContentViewBinding.resultRv.adapter?.notifyDataSetChanged() + contentChange = false + } + + } + + fun updateSpinnerFromResults(labelList: List) { + for (visionLabel in labelList) { + if (!displayList.contains(visionLabel.text) && visionLabel.confidence > 0.5f && displayList.size <= 9) { + displayList.add(visionLabel.text) + contentChange = true + // resultList?.add(visionLabel) + } + } + if (contentChange) { + mContentViewBinding.resultsMessageTv.text = + getString(R.string.x_results_found, displayList.size) + mContentViewBinding.resultRv.adapter?.notifyDataSetChanged() + contentChange = false + } + // mContentViewBinding.resultRv.adapter = CameraSearchResultAdapter(this, displayList) + + } + +} diff --git a/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/adapters/CameraSearchResultAdapter.kt b/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/adapters/CameraSearchResultAdapter.kt new file mode 100644 index 0000000..eb9e7e1 --- /dev/null +++ b/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/adapters/CameraSearchResultAdapter.kt @@ -0,0 +1,51 @@ +/** + * Webkul Software. + * + * Kotlin + * + * @author Webkul @webkul.com> + * @category Webkul + * @package com.webkul.mobikul + * @copyright 2010-2018 Webkul Software Private Limited (https://webkul.com) + * @license https://store.webkul.com/license.html ASL Licence + * @link https://store.webkul.com/license.html + */ + +package webkul.bagisto_app_demo.mlkit.adapters + +import android.view.LayoutInflater +import android.view.View +import android.view.ViewGroup +import androidx.databinding.DataBindingUtil +import androidx.recyclerview.widget.RecyclerView +import com.example.bagisto_app_demo.R +import com.example.bagisto_app_demo.databinding.CameraSimpleSpinnerItemBinding +import webkul.bagisto_app_demo.mlkit.activities.CameraSearchActivity + +class CameraSearchResultAdapter( + private val context: CameraSearchActivity, + private val labelList: List +) : RecyclerView.Adapter() { + + override fun onCreateViewHolder(parent: ViewGroup, viewType: Int): ViewHolder { + val view = + LayoutInflater.from(context).inflate(R.layout.camera_simple_spinner_item, parent, false) + return ViewHolder(view) + } + + override fun onBindViewHolder(holder: ViewHolder, position: Int) { + holder.mBinding?.labelTv?.text = labelList[position] + holder.mBinding?.labelTv?.setOnClickListener { context.sendResultBack(position) } + + } + + override fun getItemCount() = labelList.size + + + inner class ViewHolder(itemView: View) : RecyclerView.ViewHolder(itemView) { + val mBinding: CameraSimpleSpinnerItemBinding? = DataBindingUtil.bind(itemView) + + } + + +} diff --git a/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/customviews/CameraSource.kt b/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/customviews/CameraSource.kt new file mode 100755 index 0000000..3f489c8 --- /dev/null +++ b/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/customviews/CameraSource.kt @@ -0,0 +1,657 @@ +/* + * Copyright 2020 Google LLC. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package webkul.bagisto_app_demo.mlkit.customviews + +import android.Manifest +import android.annotation.SuppressLint +import android.app.Activity +import android.content.Context +import android.graphics.ImageFormat +import android.graphics.SurfaceTexture +import android.hardware.Camera +import android.hardware.Camera.CameraInfo +import android.hardware.Camera.PreviewCallback +import android.util.Log +import android.view.Surface +import android.view.SurfaceHolder +import android.view.WindowManager +import androidx.annotation.RequiresPermission +import com.google.android.gms.common.images.Size +import java.io.IOException +import java.nio.ByteBuffer +import java.util.* + +/** + * Manages the camera and allows UI updates on top of it (e.g. overlaying extra Graphics or + * displaying extra information). This receives preview frames from the camera at a specified rate, + * sending those frames to child classes' detectors / classifiers as fast as it is able to process. + */ +class CameraSource(protected var activity: Activity, private val graphicOverlay: GraphicOverlay) { + private val processingRunnable: FrameProcessingRunnable = FrameProcessingRunnable() + private val processorLock = Object() + + /** + * Map to convert between a byte array, received from the camera, and its associated byte buffer. + * We use byte buffers internally because this is a more efficient way to call into native code + * later (avoids a potential copy). + * + * + * **Note:** uses IdentityHashMap here instead of HashMap because the behavior of an array's + * equals, hashCode and toString methods is both useless and unexpected. IdentityHashMap enforces + * identity ('==') check on the keys. + */ + private val bytesToByteBuffer = IdentityHashMap() + var camera: Camera? = null + private set + + /** + * Returns the selected camera; one of [.CAMERA_FACING_BACK] or [ ][.CAMERA_FACING_FRONT]. + */ + var cameraFacing = CAMERA_FACING_BACK + private set + + /** + * Rotation of the device, and thus the associated preview images captured from the device. + */ + private var rotationDegrees = 0 + + /** + * Returns the preview size that is currently in use by the underlying camera. + */ + var previewSize: Size? = null + private set + + // This instance needs to be held onto to avoid GC of its underlying resources. Even though it + // isn't used outside of the method that creates it, it still must have hard references maintained + // to it. + private var dummySurfaceTexture: SurfaceTexture? = null + + /** + * Dedicated thread and associated runnable for calling into the detector with frames, as the + * frames become available from the camera. + */ + private var processingThread: Thread? = null + private var frameProcessor: VisionImageProcessor? = null + + /** + * Stops the camera and releases the resources of the camera and underlying detector. + */ + fun release() { + synchronized(processorLock) { + stop() + if (frameProcessor != null) { + frameProcessor!!.stop() + } + } + } + + /** + * Opens the camera and starts sending preview frames to the underlying detector. The preview + * frames are not displayed. + * + * @throws IOException if the camera's preview texture or display could not be initialized + */ + @RequiresPermission(Manifest.permission.CAMERA) + @Synchronized + @Throws(IOException::class) + fun start(): CameraSource { + if (camera != null) { + return this + } + camera = createCamera() + dummySurfaceTexture = SurfaceTexture(DUMMY_TEXTURE_NAME) + camera!!.setPreviewTexture(dummySurfaceTexture) + camera!!.startPreview() + processingThread = Thread(processingRunnable) + processingRunnable.setActive(true) + processingThread!!.start() + return this + } + + /** + * Opens the camera and starts sending preview frames to the underlying detector. The supplied + * surface holder is used for the preview so frames can be displayed to the user. + * + * @param surfaceHolder the surface holder to use for the preview frames + * @throws IOException if the supplied surface holder could not be used as the preview display + */ + @RequiresPermission(Manifest.permission.CAMERA) + @Synchronized + @Throws(IOException::class) + fun start(surfaceHolder: SurfaceHolder?): CameraSource { + if (camera != null) { + return this + } + camera = createCamera() + camera!!.setPreviewDisplay(surfaceHolder) + camera!!.startPreview() + processingThread = Thread(processingRunnable) + processingRunnable.setActive(true) + processingThread!!.start() + return this + } + + /** + * Closes the camera and stops sending frames to the underlying frame detector. + * + * + * This camera source may be restarted again by calling [.start] or [ ][.start]. + * + * + * Call [.release] instead to completely shut down this camera source and release the + * resources of the underlying detector. + */ + @Synchronized + fun stop() { + processingRunnable.setActive(false) + if (processingThread != null) { + try { + // Wait for the thread to complete to ensure that we can't have multiple threads + // executing at the same time (i.e., which would happen if we called start too + // quickly after stop). + processingThread!!.join() + } catch (e: InterruptedException) { + Log.d(TAG, "Frame processing thread interrupted on release.") + } + processingThread = null + } + if (camera != null) { + camera!!.stopPreview() + camera!!.setPreviewCallbackWithBuffer(null) + try { + camera!!.setPreviewTexture(null) + dummySurfaceTexture = null + camera!!.setPreviewDisplay(null) + } catch (e: Exception) { + Log.e(TAG, "Failed to clear camera preview: $e") + } + camera!!.release() + camera = null + } + + // Release the reference to any image buffers, since these will no longer be in use. + bytesToByteBuffer.clear() + } + + /** + * Changes the facing of the camera. + */ + @Synchronized + fun setFacing(facing: Int) { + require(!(facing != CAMERA_FACING_BACK && facing != CAMERA_FACING_FRONT)) { "Invalid camera: $facing" } + cameraFacing = facing + } + + /** + * Opens the camera and applies the user settings. + * + * @throws IOException if camera cannot be found or preview cannot be processed + */ + @SuppressLint("InlinedApi") + @Throws(IOException::class) + private fun createCamera(): Camera { + val requestedCameraId = getIdForRequestedCamera(cameraFacing) + if (requestedCameraId == -1) { + throw IOException("Could not find requested camera.") + } + val camera = Camera.open(requestedCameraId) + var sizePair: SizePair? = null + if (sizePair == null) { + sizePair = selectSizePair( + camera, + DEFAULT_REQUESTED_CAMERA_PREVIEW_WIDTH, + DEFAULT_REQUESTED_CAMERA_PREVIEW_HEIGHT + ) + } + if (sizePair == null) { + throw IOException("Could not find suitable preview size.") + } + previewSize = sizePair.preview + Log.v(TAG, "Camera preview size: $previewSize") + val previewFpsRange = selectPreviewFpsRange(camera, REQUESTED_FPS) + ?: throw IOException("Could not find suitable preview frames per second range.") + val parameters = camera.parameters + val pictureSize = sizePair.picture + if (pictureSize != null) { + Log.v(TAG, "Camera picture size: $pictureSize") + parameters.setPictureSize(pictureSize.width, pictureSize.height) + } + parameters.setPreviewSize(previewSize!!.width, previewSize!!.height) + parameters.setPreviewFpsRange( + previewFpsRange[Camera.Parameters.PREVIEW_FPS_MIN_INDEX], + previewFpsRange[Camera.Parameters.PREVIEW_FPS_MAX_INDEX] + ) + // Use YV12 so that we can exercise YV12->NV21 auto-conversion logic for OCR detection + parameters.previewFormat = IMAGE_FORMAT + setRotation(camera, parameters, requestedCameraId) + if (REQUESTED_AUTO_FOCUS) { + if (parameters + .supportedFocusModes + .contains(Camera.Parameters.FOCUS_MODE_CONTINUOUS_VIDEO) + ) { + parameters.focusMode = Camera.Parameters.FOCUS_MODE_CONTINUOUS_VIDEO + } else { + Log.i(TAG, "Camera auto focus is not supported on this device.") + } + } + camera.parameters = parameters + + // Four frame buffers are needed for working with the camera: + // + // one for the frame that is currently being executed upon in doing detection + // one for the next pending frame to process immediately upon completing detection + // two for the frames that the camera uses to populate future preview images + // + // Through trial and error it appears that two free buffers, in addition to the two buffers + // used in this code, are needed for the camera to work properly. Perhaps the camera has + // one thread for acquiring images, and another thread for calling into user code. If only + // three buffers are used, then the camera will spew thousands of warning messages when + // detection takes a non-trivial amount of time. + camera.setPreviewCallbackWithBuffer(CameraPreviewCallback()) + camera.addCallbackBuffer(createPreviewBuffer(previewSize)) + camera.addCallbackBuffer(createPreviewBuffer(previewSize)) + camera.addCallbackBuffer(createPreviewBuffer(previewSize)) + camera.addCallbackBuffer(createPreviewBuffer(previewSize)) + return camera + } + + /** + * Calculates the correct rotation for the given camera id and sets the rotation in the + * parameters. It also sets the camera's display orientation and rotation. + * + * @param parameters the camera parameters for which to set the rotation + * @param cameraId the camera id to set rotation based on + */ + private fun setRotation(camera: Camera, parameters: Camera.Parameters, cameraId: Int) { + val windowManager = activity.getSystemService(Context.WINDOW_SERVICE) as WindowManager + var degrees = 0 + val rotation = windowManager.defaultDisplay.rotation + when (rotation) { + Surface.ROTATION_0 -> degrees = 0 + Surface.ROTATION_90 -> degrees = 90 + Surface.ROTATION_180 -> degrees = 180 + Surface.ROTATION_270 -> degrees = 270 + else -> Log.e(TAG, "Bad rotation value: $rotation") + } + val cameraInfo = CameraInfo() + Camera.getCameraInfo(cameraId, cameraInfo) + val displayAngle: Int + if (cameraInfo.facing == CameraInfo.CAMERA_FACING_FRONT) { + rotationDegrees = (cameraInfo.orientation + degrees) % 360 + displayAngle = (360 - rotationDegrees) % 360 // compensate for it being mirrored + } else { // back-facing + rotationDegrees = (cameraInfo.orientation - degrees + 360) % 360 + displayAngle = rotationDegrees + } + Log.d(TAG, "Display rotation is: $rotation") + Log.d(TAG, "Camera face is: " + cameraInfo.facing) + Log.d(TAG, "Camera rotation is: " + cameraInfo.orientation) + // This value should be one of the degrees that ImageMetadata accepts: 0, 90, 180 or 270. + Log.d(TAG, "RotationDegrees is: " + rotationDegrees) + camera.setDisplayOrientation(displayAngle) + parameters.setRotation(rotationDegrees) + } + + /** + * Creates one buffer for the camera preview callback. The size of the buffer is based off of the + * camera preview size and the format of the camera image. + * + * @return a new preview buffer of the appropriate size for the current camera settings + */ + @SuppressLint("InlinedApi") + private fun createPreviewBuffer(previewSize: Size?): ByteArray { + val bitsPerPixel = ImageFormat.getBitsPerPixel(IMAGE_FORMAT) + val sizeInBits = previewSize!!.height.toLong() * previewSize.width * bitsPerPixel + val bufferSize = Math.ceil(sizeInBits / 8.0).toInt() + 1 + + // Creating the byte array this way and wrapping it, as opposed to using .allocate(), + // should guarantee that there will be an array to work with. + val byteArray = ByteArray(bufferSize) + val buffer = ByteBuffer.wrap(byteArray) + check(!(!buffer.hasArray() || buffer.array() != byteArray)) { + // I don't think that this will ever happen. But if it does, then we wouldn't be + // passing the preview content to the underlying detector later. + "Failed to create valid buffer for camera source." + } + bytesToByteBuffer[byteArray] = buffer + return byteArray + } + + fun setMachineLearningFrameProcessor(processor: VisionImageProcessor?) { + synchronized(processorLock) { + + if (frameProcessor != null) { + frameProcessor!!.stop() + } + frameProcessor = processor + } + } + // ============================================================================================== + // Frame processing + // ============================================================================================== + + + /** + * Stores a preview size and a corresponding same-aspect-ratio picture size. To avoid distorted + * preview images on some devices, the picture size must be set to a size that is the same aspect + * ratio as the preview size or the preview may end up being distorted. If the picture size is + * null, then there is no picture size with the same aspect ratio as the preview size. + */ + class SizePair { + val preview: Size + val picture: Size? + + internal constructor(previewSize: Camera.Size, pictureSize: Camera.Size?) { + preview = Size(previewSize.width, previewSize.height) + picture = if (pictureSize != null) Size(pictureSize.width, pictureSize.height) else null + } + + } + + /** + * Called when the camera has a new preview frame. + */ + private inner class CameraPreviewCallback : PreviewCallback { + override fun onPreviewFrame(data: ByteArray, camera: Camera) { + processingRunnable.setNextFrame(data, camera) + } + } + + /** + * This runnable controls access to the underlying receiver, calling it to process frames when + * available from the camera. This is designed to run detection on frames as fast as possible + * (i.e., without unnecessary context switching or waiting on the next frame). + * + * + * While detection is running on a frame, new frames may be received from the camera. As these + * frames come in, the most recent frame is held onto as pending. As soon as detection and its + * associated processing is done for the previous frame, detection on the mostly recently received + * frame will immediately start on the same thread. + */ + private inner class FrameProcessingRunnable() : Runnable { + // This lock guards all of the member variables below. + private val lock = Object() + private var active = true + + // These pending variables hold the state associated with the new frame awaiting processing. + private var pendingFrameData: ByteBuffer? = null + + /** + * Marks the runnable as active/not active. Signals any blocked threads to continue. + */ + fun setActive(active: Boolean) { + synchronized(lock) { + this.active = active + lock.notifyAll() + } + } + + /** + * Sets the frame data received from the camera. This adds the previous unused frame buffer (if + * present) back to the camera, and keeps a pending reference to the frame data for future use. + */ + fun setNextFrame(data: ByteArray, camera: Camera) { + synchronized(lock) { + if (pendingFrameData != null) { + camera.addCallbackBuffer(pendingFrameData!!.array()) + pendingFrameData = null + } + if (!bytesToByteBuffer.containsKey(data)) { + Log.d( + TAG, "Skipping frame. Could not find ByteBuffer associated with the image " + + "data from the camera." + ) + return + } + pendingFrameData = bytesToByteBuffer[data] + + // Notify the processor thread if it is waiting on the next frame (see below). + lock.notifyAll() + } + } + + /** + * As long as the processing thread is active, this executes detection on frames continuously. + * The next pending frame is either immediately available or hasn't been received yet. Once it + * is available, we transfer the frame info to local variables and run detection on that frame. + * It immediately loops back for the next frame without pausing. + * + * + * If detection takes longer than the time in between new frames from the camera, this will + * mean that this loop will run without ever waiting on a frame, avoiding any context switching + * or frame acquisition time latency. + * + * + * If you find that this is using more CPU than you'd like, you should probably decrease the + * FPS setting above to allow for some idle time in between frames. + */ + @SuppressLint("InlinedApi") + override fun run() { + var data: ByteBuffer? + while (true) { + synchronized(lock) { + while (active && pendingFrameData == null) { + try { + // Wait for the next frame to be received from the camera, since we + // don't have it yet. + lock.wait() + } catch (e: InterruptedException) { + Log.d(TAG, "Frame processing loop terminated.", e) + return + } + } + if (!active) { + // Exit the loop once this camera source is stopped or released. We check + // this here, immediately after the wait() above, to handle the case where + // setActive(false) had been called, triggering the termination of this + // loop. + return + } + + // Hold onto the frame data locally, so that we can use this for detection + // below. We need to clear pendingFrameData to ensure that this buffer isn't + // recycled back to the camera before we are done using that data. + data = pendingFrameData + pendingFrameData = null + } + + // The code below needs to run outside of synchronization, because this will allow + // the camera to add pending frame(s) while we are running detection on the current + // frame. + try { + synchronized(processorLock) { + frameProcessor!!.processByteBuffer( + data, + FrameMetadata.Builder() + .setWidth(previewSize!!.width) + .setHeight(previewSize!!.height) + .setRotation(rotationDegrees) + .build(), + graphicOverlay + ) + } + } catch (t: Exception) { + Log.e(TAG, "Exception thrown from receiver.", t) + } finally { + camera!!.addCallbackBuffer(data!!.array()) + } + } + } + } + + companion object { + @SuppressLint("InlinedApi") + val CAMERA_FACING_BACK = CameraInfo.CAMERA_FACING_BACK + + @SuppressLint("InlinedApi") + val CAMERA_FACING_FRONT = CameraInfo.CAMERA_FACING_FRONT + const val IMAGE_FORMAT = ImageFormat.NV21 + const val DEFAULT_REQUESTED_CAMERA_PREVIEW_WIDTH = 480 + const val DEFAULT_REQUESTED_CAMERA_PREVIEW_HEIGHT = 360 + private const val TAG = "MIDemoApp:CameraSource" + + /** + * The dummy surface texture must be assigned a chosen name. Since we never use an OpenGL context, + * we can choose any ID we want here. The dummy surface texture is not a crazy hack - it is + * actually how the camera team recommends using the camera without a preview. + */ + private const val DUMMY_TEXTURE_NAME = 100 + + /** + * If the absolute difference between a preview size aspect ratio and a picture size aspect ratio + * is less than this tolerance, they are considered to be the same aspect ratio. + */ + private const val ASPECT_RATIO_TOLERANCE = 0.01f + private const val REQUESTED_FPS = 30.0f + private const val REQUESTED_AUTO_FOCUS = true + // ============================================================================================== + // Public + // ============================================================================================== + /** + * Gets the id for the camera specified by the direction it is facing. Returns -1 if no such + * camera was found. + * + * @param facing the desired camera (front-facing or rear-facing) + */ + private fun getIdForRequestedCamera(facing: Int): Int { + val cameraInfo = CameraInfo() + for (i in 0 until Camera.getNumberOfCameras()) { + Camera.getCameraInfo(i, cameraInfo) + if (cameraInfo.facing == facing) { + return i + } + } + return -1 + } + + /** + * Selects the most suitable preview and picture size, given the desired width and height. + * + * + * Even though we only need to find the preview size, it's necessary to find both the preview + * size and the picture size of the camera together, because these need to have the same aspect + * ratio. On some hardware, if you would only set the preview size, you will get a distorted + * image. + * + * @param camera the camera to select a preview size from + * @param desiredWidth the desired width of the camera preview frames + * @param desiredHeight the desired height of the camera preview frames + * @return the selected preview and picture size pair + */ + fun selectSizePair(camera: Camera, desiredWidth: Int, desiredHeight: Int): SizePair? { + val validPreviewSizes = generateValidPreviewSizeList(camera) + + // The method for selecting the best size is to minimize the sum of the differences between + // the desired values and the actual values for width and height. This is certainly not the + // only way to select the best size, but it provides a decent tradeoff between using the + // closest aspect ratio vs. using the closest pixel area. + var selectedPair: SizePair? = null + var minDiff = Int.MAX_VALUE + for (sizePair in validPreviewSizes) { + val size = sizePair.preview + val diff = + Math.abs(size.width - desiredWidth) + Math.abs(size.height - desiredHeight) + if (diff < minDiff) { + selectedPair = sizePair + minDiff = diff + } + } + return selectedPair + } + + /** + * Generates a list of acceptable preview sizes. Preview sizes are not acceptable if there is not + * a corresponding picture size of the same aspect ratio. If there is a corresponding picture size + * of the same aspect ratio, the picture size is paired up with the preview size. + * + * + * This is necessary because even if we don't use still pictures, the still picture size must + * be set to a size that is the same aspect ratio as the preview size we choose. Otherwise, the + * preview images may be distorted on some devices. + */ + fun generateValidPreviewSizeList(camera: Camera): List { + val parameters = camera.parameters + val supportedPreviewSizes = parameters.supportedPreviewSizes + val supportedPictureSizes = parameters.supportedPictureSizes + val validPreviewSizes: MutableList = ArrayList() + for (previewSize in supportedPreviewSizes) { + val previewAspectRatio = previewSize.width.toFloat() / previewSize.height.toFloat() + + // By looping through the picture sizes in order, we favor the higher resolutions. + // We choose the highest resolution in order to support taking the full resolution + // picture later. + for (pictureSize in supportedPictureSizes) { + val pictureAspectRatio = + pictureSize.width.toFloat() / pictureSize.height.toFloat() + if (Math.abs(previewAspectRatio - pictureAspectRatio) < ASPECT_RATIO_TOLERANCE) { + validPreviewSizes.add(SizePair(previewSize, pictureSize)) + break + } + } + } + + // If there are no picture sizes with the same aspect ratio as any preview sizes, allow all + // of the preview sizes and hope that the camera can handle it. Probably unlikely, but we + // still account for it. + if (validPreviewSizes.size == 0) { + Log.w(TAG, "No preview sizes have a corresponding same-aspect-ratio picture size") + for (previewSize in supportedPreviewSizes) { + // The null picture size will let us know that we shouldn't set a picture size. + validPreviewSizes.add(SizePair(previewSize, null)) + } + } + return validPreviewSizes + } + + /** + * Selects the most suitable preview frames per second range, given the desired frames per second. + * + * @param camera the camera to select a frames per second range from + * @param desiredPreviewFps the desired frames per second for the camera preview frames + * @return the selected preview frames per second range + */ + @SuppressLint("InlinedApi") + private fun selectPreviewFpsRange(camera: Camera, desiredPreviewFps: Float): IntArray? { + // The camera API uses integers scaled by a factor of 1000 instead of floating-point frame + // rates. + val desiredPreviewFpsScaled = (desiredPreviewFps * 1000.0f).toInt() + + // Selects a range with whose upper bound is as close as possible to the desired fps while its + // lower bound is as small as possible to properly expose frames in low light conditions. Note + // that this may select a range that the desired value is outside of. For example, if the + // desired frame rate is 30.5, the range (30, 30) is probably more desirable than (30, 40). + var selectedFpsRange: IntArray? = null + var minUpperBoundDiff = Int.MAX_VALUE + var minLowerBound = Int.MAX_VALUE + val previewFpsRangeList = camera.parameters.supportedPreviewFpsRange + for (range in previewFpsRangeList) { + val upperBoundDiff = + Math.abs(desiredPreviewFpsScaled - range[Camera.Parameters.PREVIEW_FPS_MAX_INDEX]) + val lowerBound = range[Camera.Parameters.PREVIEW_FPS_MIN_INDEX] + if (upperBoundDiff <= minUpperBoundDiff && lowerBound <= minLowerBound) { + selectedFpsRange = range + minUpperBoundDiff = upperBoundDiff + minLowerBound = lowerBound + } + } + return selectedFpsRange + } + } + + init { + //graphicOverlay.clear() + } +} \ No newline at end of file diff --git a/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/customviews/CameraSourcePreview.kt b/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/customviews/CameraSourcePreview.kt new file mode 100755 index 0000000..3a50cc6 --- /dev/null +++ b/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/customviews/CameraSourcePreview.kt @@ -0,0 +1,171 @@ +/* + * Copyright 2020 Google LLC. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package webkul.bagisto_app_demo.mlkit.customviews + +import android.content.Context +import android.content.res.Configuration +import android.util.AttributeSet +import android.util.Log +import android.view.SurfaceHolder +import android.view.SurfaceView +import android.view.ViewGroup +import java.io.IOException + +/** + * Preview the camera image in the screen. + */ +class CameraSourcePreview @JvmOverloads constructor( + context: Context, + attrs: AttributeSet? = null, + defStyle: Int = 0 +) : ViewGroup(context, attrs, defStyle) { + private val surfaceView: SurfaceView + private var startRequested = false + private var surfaceAvailable = false + private var cameraSource: CameraSource? = null + private var overlay: GraphicOverlay? = null + + @Throws(IOException::class) + private fun start(cameraSource: CameraSource) { + this.cameraSource = cameraSource + if (this.cameraSource != null) { + startRequested = true + startIfReady() + } + } + + @Throws(IOException::class) + fun start(cameraSource: CameraSource, overlay: GraphicOverlay?) { + this.overlay = overlay + start(cameraSource) + } + + fun stop() { + if (cameraSource != null) { + cameraSource!!.stop() + } + } + + fun release() { + if (cameraSource != null) { + cameraSource!!.release() + cameraSource = null + } + surfaceView.holder.surface.release() + } + + @Throws(IOException::class, SecurityException::class) + private fun startIfReady() { + try { + if (startRequested && surfaceAvailable) { + cameraSource!!.start(surfaceView.holder) + requestLayout() + if (overlay != null) { + val size = cameraSource!!.previewSize + val min = Math.min(size!!.width, size.height) + val max = Math.max(size!!.width, size.height) + val isImageFlipped = + cameraSource!!.cameraFacing == CameraSource.CAMERA_FACING_FRONT + if (isPortraitMode) { + // Swap width and height sizes when in portrait, since it will be rotated by 90 degrees. + // The camera preview and the image being processed have the same size. + overlay!!.setImageSourceInfo(min, max, isImageFlipped) + } else { + overlay!!.setImageSourceInfo(max, min, isImageFlipped) + } + + } + startRequested = false + } + } catch (e: Exception) { + e.printStackTrace() + } + } + + override fun onLayout(changed: Boolean, left: Int, top: Int, right: Int, bottom: Int) { + var width = 320 + var height = 240 + if (cameraSource != null) { + val size = cameraSource!!.previewSize + if (size != null) { + width = size.width + height = size.height + } + } + + // Swap width and height sizes when in portrait, since it will be rotated 90 degrees + if (isPortraitMode) { + val tmp = width + width = height + height = tmp + } + val previewAspectRatio = width.toFloat() / height + val layoutWidth = right - left + val layoutHeight = bottom - top + val layoutAspectRatio = layoutWidth.toFloat() / layoutHeight + if (previewAspectRatio > layoutAspectRatio) { + // The preview input is wider than the layout area. Fit the layout height and crop + // the preview input horizontally while keep the center. + val horizontalOffset = (previewAspectRatio * layoutHeight - layoutWidth).toInt() / 2 + surfaceView.layout(-horizontalOffset, 0, layoutWidth + horizontalOffset, layoutHeight) + } else { + // The preview input is taller than the layout area. Fit the layout width and crop the preview + // input vertically while keep the center. + val verticalOffset = (layoutWidth / previewAspectRatio - layoutHeight).toInt() / 2 + surfaceView.layout(0, -verticalOffset, layoutWidth, layoutHeight + verticalOffset) + } + } + + private val isPortraitMode: Boolean + get() { + val orientation = context.resources.configuration.orientation + if (orientation == Configuration.ORIENTATION_LANDSCAPE) { + return false + } + if (orientation == Configuration.ORIENTATION_PORTRAIT) { + return true + } + Log.d(TAG, "isPortraitMode returning false by default") + return false + } + + private inner class SurfaceCallback : SurfaceHolder.Callback { + override fun surfaceCreated(surface: SurfaceHolder) { + surfaceAvailable = true + try { + startIfReady() + } catch (e: IOException) { + Log.e(TAG, "Could not start camera source.", e) + } + } + + override fun surfaceDestroyed(surface: SurfaceHolder) { + surfaceAvailable = false + } + + override fun surfaceChanged(holder: SurfaceHolder, format: Int, width: Int, height: Int) {} + } + + companion object { + private const val TAG = "MIDemoApp:Preview" + } + + init { + surfaceView = SurfaceView(context) + surfaceView.holder.addCallback(SurfaceCallback()) + addView(surfaceView) + } +} \ No newline at end of file diff --git a/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/customviews/FrameMetadata.kt b/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/customviews/FrameMetadata.kt new file mode 100755 index 0000000..12a29b3 --- /dev/null +++ b/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/customviews/FrameMetadata.kt @@ -0,0 +1,32 @@ +package webkul.bagisto_app_demo.mlkit.customviews + +/** + * Describing a frame info. + */ +class FrameMetadata private constructor(val width: Int, val height: Int, val rotation: Int) { + + + class Builder { + private var width = 0 + private var height = 0 + private var rotation = 0 + fun setWidth(width: Int): Builder { + this.width = width + return this + } + + fun setHeight(height: Int): Builder { + this.height = height + return this + } + + fun setRotation(rotation: Int): Builder { + this.rotation = rotation + return this + } + + fun build(): FrameMetadata { + return FrameMetadata(width, height, rotation) + } + } +} \ No newline at end of file diff --git a/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/customviews/GraphicOverlay.kt b/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/customviews/GraphicOverlay.kt new file mode 100755 index 0000000..4155a55 --- /dev/null +++ b/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/customviews/GraphicOverlay.kt @@ -0,0 +1,142 @@ +/* + * Copyright 2020 Google LLC. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package webkul.bagisto_app_demo.mlkit.customviews + +import android.content.Context +import android.graphics.Canvas +import android.graphics.Matrix +import android.util.AttributeSet +import android.view.View +import java.util.* + +/** + * A view which renders a series of custom graphics to be overlayed on top of an associated preview + * (i.e., the camera preview). The creator can add graphics objects, update the objects, and remove + * them, triggering the appropriate drawing and invalidation within the view. + * + * + * Supports scaling and mirroring of the graphics relative the camera's preview properties. The + * idea is that detection items are expressed in terms of an image size, but need to be scaled up + * to the full view size, and also mirrored in the case of the front-facing camera. + * + * + * Associated [Graphic] items should use the following methods to convert to view + * coordinates for the graphics that are drawn: + * + * + * 1. [Graphic.scale] adjusts the size of the supplied value from the image scale + * to the view scale. + * 1. [Graphic.translateX] and [Graphic.translateY] adjust the + * coordinate from the image's coordinate system to the view coordinate system. + * + */ +class GraphicOverlay(context: Context?, attrs: AttributeSet?) : View(context, attrs) { + private val lock = Object() + + // Matrix for transforming from image coordinates to overlay view coordinates. + private val transformationMatrix = Matrix() + var imageWidth = 0 + private set + var imageHeight = 0 + private set + + // The factor of overlay View size to image size. Anything in the image coordinates need to be + // scaled by this amount to fit with the area of overlay View. + private var scaleFactor = 1.0f + + // The number of horizontal pixels needed to be cropped on each side to fit the image with the + // area of overlay View after scaling. + private var postScaleWidthOffset = 0f + + // The number of vertical pixels needed to be cropped on each side to fit the image with the + // area of overlay View after scaling. + private var postScaleHeightOffset = 0f + private var isImageFlipped = false + private var needUpdateTransformation = true + + fun setImageSourceInfo(imageWidth: Int, imageHeight: Int, isFlipped: Boolean) { + /* Preconditions.checkState(imageWidth > 0, "image width must be positive"); + Preconditions.checkState(imageHeight > 0, "image height must be positive");*/ + synchronized(lock) { + this.imageWidth = imageWidth + this.imageHeight = imageHeight + isImageFlipped = isFlipped + needUpdateTransformation = true + } + postInvalidate() + } + + + override fun onDraw(canvas: Canvas) { + super.onDraw(canvas) + + } + + /** + * Base class for a custom graphics object to be rendered within the graphic overlay. Subclass + * this and implement the [Graphic.draw] method to define the graphics element. Add + * instances to the overlay using [GraphicOverlay.add]. + */ + abstract class Graphic(private val overlay: GraphicOverlay) { + + + /** + * Adjusts the supplied value from the image scale to the view scale. + */ + fun scale(imagePixel: Float): Float { + return imagePixel * overlay.scaleFactor + } + + /** + * Returns the application context of the app. + */ + val applicationContext: Context + get() = overlay.context.applicationContext + + fun isImageFlipped(): Boolean { + return overlay.isImageFlipped + } + + /** + * Adjusts the x coordinate from the image's coordinate system to the view coordinate system. + */ + fun translateX(x: Float): Float { + return if (overlay.isImageFlipped) { + overlay.width - (scale(x) - overlay.postScaleWidthOffset) + } else { + scale(x) - overlay.postScaleWidthOffset + } + } + + /** + * Adjusts the y coordinate from the image's coordinate system to the view coordinate system. + */ + fun translateY(y: Float): Float { + return scale(y) - overlay.postScaleHeightOffset + } + + /** + * Returns a [Matrix] for transforming from image coordinates to overlay view coordinates. + */ + fun getTransformationMatrix(): Matrix { + return overlay.transformationMatrix + } + + fun postInvalidate() { + overlay.postInvalidate() + } + } +} \ No newline at end of file diff --git a/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/customviews/GrapicOverlay.kt b/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/customviews/GrapicOverlay.kt new file mode 100644 index 0000000..ed2eb25 --- /dev/null +++ b/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/customviews/GrapicOverlay.kt @@ -0,0 +1,143 @@ +///* +// * Copyright 2020 Google LLC. All rights reserved. +// * +// * Licensed under the Apache License, Version 2.0 (the "License"); +// * you may not use this file except in compliance with the License. +// * You may obtain a copy of the License at +// * +// * http://www.apache.org/licenses/LICENSE-2.0 +// * +// * Unless required by applicable law or agreed to in writing, software +// * distributed under the License is distributed on an "AS IS" BASIS, +// * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// * See the License for the specific language governing permissions and +// * limitations under the License. +// */ +//package webkul.opencart.mobikul.mlkit.customviews +// +//import android.content.Context +//import android.graphics.Canvas +//import android.graphics.Matrix +//import android.util.AttributeSet +//import android.view.View +//import webkul.opencart.mobikul.mlkit.customviews.GraphicOverlay.Graphic +//import java.util.* +// +///** +// * A view which renders a series of custom graphics to be overlayed on top of an associated preview +// * (i.e., the camera preview). The creator can add graphics objects, update the objects, and remove +// * them, triggering the appropriate drawing and invalidation within the view. +// * +// * +// * Supports scaling and mirroring of the graphics relative the camera's preview properties. The +// * idea is that detection items are expressed in terms of an image size, but need to be scaled up +// * to the full view size, and also mirrored in the case of the front-facing camera. +// * +// * +// * Associated [Graphic] items should use the following methods to convert to view +// * coordinates for the graphics that are drawn: +// * +// * +// * 1. [Graphic.scale] adjusts the size of the supplied value from the image scale +// * to the view scale. +// * 1. [Graphic.translateX] and [Graphic.translateY] adjust the +// * coordinate from the image's coordinate system to the view coordinate system. +// * +// */ +//class GraphicOverlay(context: Context?, attrs: AttributeSet?) : View(context, attrs) { +// private val lock = Object() +// +// // Matrix for transforming from image coordinates to overlay view coordinates. +// private val transformationMatrix = Matrix() +// var imageWidth = 0 +// private set +// var imageHeight = 0 +// private set +// +// // The factor of overlay View size to image size. Anything in the image coordinates need to be +// // scaled by this amount to fit with the area of overlay View. +// private var scaleFactor = 1.0f +// +// // The number of horizontal pixels needed to be cropped on each side to fit the image with the +// // area of overlay View after scaling. +// private var postScaleWidthOffset = 0f +// +// // The number of vertical pixels needed to be cropped on each side to fit the image with the +// // area of overlay View after scaling. +// private var postScaleHeightOffset = 0f +// private var isImageFlipped = false +// private var needUpdateTransformation = true +// +// fun setImageSourceInfo(imageWidth: Int, imageHeight: Int, isFlipped: Boolean) { +// /* Preconditions.checkState(imageWidth > 0, "image width must be positive"); +// Preconditions.checkState(imageHeight > 0, "image height must be positive");*/ +// synchronized(lock) { +// this.imageWidth = imageWidth +// this.imageHeight = imageHeight +// isImageFlipped = isFlipped +// needUpdateTransformation = true +// } +// postInvalidate() +// } +// +// +// override fun onDraw(canvas: Canvas) { +// super.onDraw(canvas) +// +// } +// +// /** +// * Base class for a custom graphics object to be rendered within the graphic overlay. Subclass +// * this and implement the [Graphic.draw] method to define the graphics element. Add +// * instances to the overlay using [GraphicOverlay.add]. +// */ +// abstract class Graphic(private val overlay: GraphicOverlay) { +// +// +// /** +// * Adjusts the supplied value from the image scale to the view scale. +// */ +// fun scale(imagePixel: Float): Float { +// return imagePixel * overlay.scaleFactor +// } +// +// /** +// * Returns the application context of the app. +// */ +// val applicationContext: Context +// get() = overlay.context.applicationContext +// +// fun isImageFlipped(): Boolean { +// return overlay.isImageFlipped +// } +// +// /** +// * Adjusts the x coordinate from the image's coordinate system to the view coordinate system. +// */ +// fun translateX(x: Float): Float { +// return if (overlay.isImageFlipped) { +// overlay.width - (scale(x) - overlay.postScaleWidthOffset) +// } else { +// scale(x) - overlay.postScaleWidthOffset +// } +// } +// +// /** +// * Adjusts the y coordinate from the image's coordinate system to the view coordinate system. +// */ +// fun translateY(y: Float): Float { +// return scale(y) - overlay.postScaleHeightOffset +// } +// +// /** +// * Returns a [Matrix] for transforming from image coordinates to overlay view coordinates. +// */ +// fun getTransformationMatrix(): Matrix { +// return overlay.transformationMatrix +// } +// +// fun postInvalidate() { +// overlay.postInvalidate() +// } +// } +//} \ No newline at end of file diff --git a/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/customviews/ScopedExecutor.kt b/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/customviews/ScopedExecutor.kt new file mode 100755 index 0000000..dc804ad --- /dev/null +++ b/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/customviews/ScopedExecutor.kt @@ -0,0 +1,52 @@ +/* + * Copyright 2020 Google LLC. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package webkul.bagisto_app_demo.mlkit.customviews + +import java.util.concurrent.Executor +import java.util.concurrent.atomic.AtomicBoolean + +/** + * Wraps an existing executor to provide a [.shutdown] method that allows subsequent + * cancellation of submitted runnables. + */ +class ScopedExecutor(private val executor: Executor) : Executor { + private val shutdown = AtomicBoolean() + override fun execute(command: Runnable) { + // Return early if this object has been shut down. + if (shutdown.get()) { + return + } + executor.execute { + + // Check again in case it has been shut down in the mean time. + if (shutdown.get()) { + return@execute + } + command.run() + } + } + + /** + * After this method is called, no runnables that have been submitted or are subsequently + * submitted will start to execute, turning this executor into a no-op. + * + * + * Runnables that have already started to execute will continue. + */ + fun shutdown() { + shutdown.set(true) + } +} \ No newline at end of file diff --git a/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/customviews/VisionImageProcessor.kt b/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/customviews/VisionImageProcessor.kt new file mode 100755 index 0000000..b087749 --- /dev/null +++ b/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/customviews/VisionImageProcessor.kt @@ -0,0 +1,42 @@ +/* + * Copyright 2020 Google LLC. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package webkul.bagisto_app_demo.mlkit.customviews + +import com.google.mlkit.common.MlKitException +import java.nio.ByteBuffer + +/** + * An interface to process the images with different vision detectors and custom image models. + */ +interface VisionImageProcessor { + /** + * Processes a bitmap image. + */ + /* fun processBitmap(bitmap: Bitmap?, graphicOverlay: GraphicOverlay) + */ + /** + * Processes ByteBuffer image data, e.g. used for Camera1 live preview case. + */ + @Throws(MlKitException::class) + fun processByteBuffer( + data: ByteBuffer?, frameMetadata: FrameMetadata?, graphicOverlay: GraphicOverlay + ) + + /** + * Stops the underlying machine learning model and release resources. + */ + fun stop() +} \ No newline at end of file diff --git a/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/customviews/VisionProcessorBase.kt b/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/customviews/VisionProcessorBase.kt new file mode 100644 index 0000000..26c9fab --- /dev/null +++ b/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/customviews/VisionProcessorBase.kt @@ -0,0 +1,190 @@ +package webkul.bagisto_app_demo.mlkit.customviews + +import android.app.ActivityManager +import android.content.Context +import android.os.SystemClock +import android.util.Log +import android.widget.Toast +import androidx.annotation.GuardedBy +import com.google.android.gms.tasks.Task +import com.google.android.gms.tasks.TaskExecutors +import com.google.mlkit.vision.common.InputImage +import webkul.bagisto_app_demo.mlkit.activities.CameraSearchActivity +import java.nio.ByteBuffer +import java.util.* + +/** + * Abstract base class for ML Kit frame processors. Subclasses need to implement {@link + * #onSuccess(T, FrameMetadata, GraphicOverlay)} to define what they want to with the detection + * results and {@link #detectInImage(VisionImage)} to specify the detector object. + * + * @param The type of the detected feature. + */ +abstract class VisionProcessorBase(context: CameraSearchActivity) : VisionImageProcessor { + + companion object { + const val MANUAL_TESTING_LOG = "LogTagForTest" + private const val TAG = "VisionProcessorBase" + } + + private var activityManager: ActivityManager = + context.getSystemService(Context.ACTIVITY_SERVICE) as ActivityManager + private val fpsTimer = Timer() + private val executor = ScopedExecutor(TaskExecutors.MAIN_THREAD) + + // Whether this processor is already shut down + private var isShutdown = false + + // Used to calculate latency, running in the same thread, no sync needed. + private var numRuns = 0 + private var totalFrameMs = 0L + private var maxFrameMs = 0L + private var minFrameMs = Long.MAX_VALUE + private var totalDetectorMs = 0L + private var maxDetectorMs = 0L + private var minDetectorMs = Long.MAX_VALUE + + // Frame count that have been processed so far in an one second interval to calculate FPS. + private var frameProcessedInOneSecondInterval = 0 + private var framesPerSecond = 0 + + // To keep the latest images and its metadata. + @GuardedBy("this") + private var latestImage: ByteBuffer? = null + + @GuardedBy("this") + private var latestImageMetaData: FrameMetadata? = null + + // To keep the images and metadata in process. + @GuardedBy("this") + private var processingImage: ByteBuffer? = null + + @GuardedBy("this") + private var processingMetaData: FrameMetadata? = null + + init { + fpsTimer.scheduleAtFixedRate( + object : TimerTask() { + override fun run() { + framesPerSecond = frameProcessedInOneSecondInterval + frameProcessedInOneSecondInterval = 0 + } + }, + 0, + 1000 + ) + } + + + // -----------------Code for processing live preview frame from Camera1 API----------------------- + @Synchronized + override fun processByteBuffer( + data: ByteBuffer?, + frameMetadata: FrameMetadata?, + graphicOverlay: GraphicOverlay, + ) { + latestImage = data + latestImageMetaData = frameMetadata + if (processingImage == null && processingMetaData == null) { + processLatestImage(graphicOverlay) + } + } + + @Synchronized + private fun processLatestImage(graphicOverlay: GraphicOverlay) { + processingImage = latestImage + processingMetaData = latestImageMetaData + latestImage = null + latestImageMetaData = null + if (processingImage != null && processingMetaData != null && !isShutdown) { + processImage(processingImage!!, processingMetaData!!, graphicOverlay) + } + } + + private fun processImage( + data: ByteBuffer, + frameMetadata: FrameMetadata, + graphicOverlay: GraphicOverlay, + ) { + val frameStartMs = SystemClock.elapsedRealtime() + // If live viewport is on (that is the underneath surface view takes care of the camera preview + // drawing), skip the unnecessary bitmap creation that used for the manual preview drawing. + requestDetectInImage( + InputImage.fromByteBuffer( + data, + frameMetadata.width, + frameMetadata.height, + frameMetadata.rotation, + InputImage.IMAGE_FORMAT_NV21 + ), + graphicOverlay, + frameStartMs + ) + .addOnSuccessListener(executor) { processLatestImage(graphicOverlay) } + } + + + // -----------------Common processing logic------------------------------------------------------- + private fun requestDetectInImage( + image: InputImage, + graphicOverlay: GraphicOverlay, + frameStartMs: Long, + ): Task { + val detectorStartMs = SystemClock.elapsedRealtime() + return detectInImage(image).addOnSuccessListener(executor) { results: T -> + val endMs = SystemClock.elapsedRealtime() + val currentFrameLatencyMs = endMs - frameStartMs + val currentDetectorLatencyMs = endMs - detectorStartMs + numRuns++ + frameProcessedInOneSecondInterval++ + totalFrameMs += currentFrameLatencyMs + maxFrameMs = Math.max(currentFrameLatencyMs, maxFrameMs) + minFrameMs = Math.min(currentFrameLatencyMs, minFrameMs) + totalDetectorMs += currentDetectorLatencyMs + maxDetectorMs = Math.max(currentDetectorLatencyMs, maxDetectorMs) + minDetectorMs = Math.min(currentDetectorLatencyMs, minDetectorMs) + + // Only log inference info once per second. When frameProcessedInOneSecondInterval is + // equal to 1, it means this is the first frame processed during the current second. + if (frameProcessedInOneSecondInterval == 1) { + + val mi = ActivityManager.MemoryInfo() + activityManager.getMemoryInfo(mi) + val availableMegs = mi.availMem / 0x100000L + Log.d(TAG, "Memory available in system: $availableMegs MB") + } + this@VisionProcessorBase.onSuccess(results, graphicOverlay) + graphicOverlay.postInvalidate() + } + .addOnFailureListener(executor) { e: Exception -> + /* graphicOverlay.clear()*/ + graphicOverlay.postInvalidate() + Toast.makeText( + graphicOverlay.context, + "Failed to process.\nError: " + + e.localizedMessage + + "\nCause: " + + e.cause, + Toast.LENGTH_LONG + ) + .show() + e.printStackTrace() + this@VisionProcessorBase.onFailure(e) + } + } + + override fun stop() { + executor.shutdown() + isShutdown = true + numRuns = 0 + totalFrameMs = 0 + totalDetectorMs = 0 + fpsTimer.cancel() + } + + protected abstract fun detectInImage(image: InputImage): Task + + protected abstract fun onSuccess(results: T, graphicOverlay: GraphicOverlay) + + protected abstract fun onFailure(e: Exception) +} diff --git a/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/labeldetector/LabelDetectorProcessor.kt b/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/labeldetector/LabelDetectorProcessor.kt new file mode 100644 index 0000000..128ac01 --- /dev/null +++ b/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/labeldetector/LabelDetectorProcessor.kt @@ -0,0 +1,68 @@ +package webkul.bagisto_app_demo.mlkit.labeldetector + +import android.util.Log +import com.google.android.gms.tasks.Task +import com.google.mlkit.vision.common.InputImage +import com.google.mlkit.vision.label.ImageLabel +import com.google.mlkit.vision.label.ImageLabeler +import com.google.mlkit.vision.label.ImageLabelerOptionsBase +import com.google.mlkit.vision.label.ImageLabeling +import webkul.bagisto_app_demo.mlkit.activities.CameraSearchActivity +import webkul.bagisto_app_demo.mlkit.customviews.GraphicOverlay +import webkul.bagisto_app_demo.mlkit.customviews.VisionProcessorBase +import java.io.IOException + + +class LabelDetectorProcessor( + private val activityInstance: CameraSearchActivity, + private val options: ImageLabelerOptionsBase +) : + VisionProcessorBase>(activityInstance) { + + private val imageLabeler: ImageLabeler = ImageLabeling.getClient(options) + + override fun stop() { + super.stop() + try { + imageLabeler.close() + } catch (e: IOException) { + Log.e( + TAG, + "Exception thrown while trying to close ImageLabelerClient: $e" + ) + } + } + + override fun detectInImage(image: InputImage): Task> { + return imageLabeler.process(image) + } + + override fun onSuccess(labels: List, graphicOverlay: GraphicOverlay) { + activityInstance.updateSpinnerFromResults(labels) + // graphicOverlay.add(LabelGraphic(graphicOverlay, labels)) + // graphicOverlay.clear() + logExtrasForTesting(labels) + + } + + override fun onFailure(e: Exception) { + Log.w(TAG, "Label detection failed.$e") + } + + companion object { + private const val TAG = "LabelDetectorProcessor" + + private fun logExtrasForTesting(labels: List?) { + if (labels == null) { + Log.v(MANUAL_TESTING_LOG, "No labels detected") + } else { + for (label in labels) { + Log.v( + MANUAL_TESTING_LOG, + String.format("Label %s, confidence %f", label.text, label.confidence) + ) + } + } + } + } +} diff --git a/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/textdetector/TextRecognitionProcessor.kt b/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/textdetector/TextRecognitionProcessor.kt new file mode 100644 index 0000000..c1e969e --- /dev/null +++ b/android/app/src/main/kotlin/webkul/bagisto_app_demo/mlkit/textdetector/TextRecognitionProcessor.kt @@ -0,0 +1,97 @@ +package webkul.bagisto_app_demo.mlkit.textdetector + +import android.util.Log +import com.google.android.gms.tasks.Task +import com.google.mlkit.vision.common.InputImage +import com.google.mlkit.vision.text.Text +import com.google.mlkit.vision.text.TextRecognition +import com.google.mlkit.vision.text.TextRecognizer +import com.google.mlkit.vision.text.latin.TextRecognizerOptions +import webkul.bagisto_app_demo.mlkit.activities.CameraSearchActivity +import webkul.bagisto_app_demo.mlkit.customviews.GraphicOverlay +import webkul.bagisto_app_demo.mlkit.customviews.VisionProcessorBase + + +/** Processor for the text detector demo. */ +class TextRecognitionProcessor(private val activityInstance: CameraSearchActivity) : + VisionProcessorBase(activityInstance) { + private val textRecognizer: TextRecognizer = + TextRecognition.getClient(TextRecognizerOptions.DEFAULT_OPTIONS) + + override fun stop() { + super.stop() + textRecognizer.close() + } + + override fun detectInImage(image: InputImage): Task { + return textRecognizer.process(image) + } + + override fun onSuccess(text: Text, graphicOverlay: GraphicOverlay) { + Log.d(TAG, "On-device Text detection successful") + logExtrasForTesting(text) + activityInstance.updateSpinnerFromTextResults(text) + + } + + override fun onFailure(e: Exception) { + Log.w(TAG, "Text detection failed.$e") + } + + companion object { + private const val TAG = "TextRecProcessor" + private fun logExtrasForTesting(text: Text?) { + if (text != null) { + Log.v( + MANUAL_TESTING_LOG, + "Detected text has : " + text.textBlocks.size + " blocks" + ) + for (i in text.textBlocks.indices) { + val lines = text.textBlocks[i].lines + Log.v( + MANUAL_TESTING_LOG, + String.format("Detected text block %d has %d lines", i, lines.size) + ) + for (j in lines.indices) { + val elements = + lines[j].elements + Log.v( + MANUAL_TESTING_LOG, + String.format("Detected text line %d has %d elements", j, elements.size) + ) + for (k in elements.indices) { + val element = elements[k] + Log.v( + MANUAL_TESTING_LOG, + String.format("Detected text element %d says: %s", k, element.text) + ) + Log.v( + MANUAL_TESTING_LOG, + String.format( + "Detected text element %d has a bounding box: %s", + k, element.boundingBox!!.flattenToString() + ) + ) + Log.v( + MANUAL_TESTING_LOG, + String.format( + "Expected corner point size is 4, get %d", + element.cornerPoints!!.size + ) + ) + for (point in element.cornerPoints!!) { + Log.v( + MANUAL_TESTING_LOG, + String.format( + "Corner point for element %d is located at: x - %d, y = %d", + k, point.x, point.y + ) + ) + } + } + } + } + } + } + } +} diff --git a/android/app/src/main/res/drawable-anydpi/cart.xml b/android/app/src/main/res/drawable-anydpi/cart.xml index 040a5a3..077609b 100644 --- a/android/app/src/main/res/drawable-anydpi/cart.xml +++ b/android/app/src/main/res/drawable-anydpi/cart.xml @@ -1,11 +1,11 @@ - ~ @Copyright (c) Webkul Software Private Limited (https://webkul.com) - ~ @license https://store.webkul.com/license.html - ~ @link https://store.webkul.com/license.html + ~ Webkul Software. + ~ @package Mobikul Application Code. + ~ @Category Mobikul + ~ @author Webkul + ~ @Copyright (c) Webkul Software Private Limited (https://webkul.com) + ~ @license https://store.webkul.com/license.html + ~ @link https://store.webkul.com/license.html --> - ~ @Copyright (c) Webkul Software Private Limited (https://webkul.com) - ~ @license https://store.webkul.com/license.html - ~ @link https://store.webkul.com/license.html + ~ Webkul Software. + ~ @package Mobikul Application Code. + ~ @Category Mobikul + ~ @author Webkul + ~ @Copyright (c) Webkul Software Private Limited (https://webkul.com) + ~ @license https://store.webkul.com/license.html + ~ @link https://store.webkul.com/license.html --> - ~ @Copyright (c) Webkul Software Private Limited (https://webkul.com) - ~ @license https://store.webkul.com/license.html - ~ @link https://store.webkul.com/license.html + ~ Webkul Software. + ~ @package Mobikul Application Code. + ~ @Category Mobikul + ~ @author Webkul + ~ @Copyright (c) Webkul Software Private Limited (https://webkul.com) + ~ @license https://store.webkul.com/license.html + ~ @link https://store.webkul.com/license.html --> diff --git a/android/app/src/main/res/drawable/flash_toggle_bg.xml b/android/app/src/main/res/drawable/flash_toggle_bg.xml index 5c871d8..7e5cb42 100644 --- a/android/app/src/main/res/drawable/flash_toggle_bg.xml +++ b/android/app/src/main/res/drawable/flash_toggle_bg.xml @@ -1,4 +1,14 @@ + + diff --git a/android/app/src/main/res/drawable/ic_flash_off.xml b/android/app/src/main/res/drawable/ic_flash_off.xml index bf52de3..fd61053 100644 --- a/android/app/src/main/res/drawable/ic_flash_off.xml +++ b/android/app/src/main/res/drawable/ic_flash_off.xml @@ -1,3 +1,13 @@ + + + ~ @Copyright (c) Webkul Software Private Limited (https://webkul.com) + ~ @license https://store.webkul.com/license.html + ~ @link https://store.webkul.com/license.html + --> + + ~ @Copyright (c) Webkul Software Private Limited (https://webkul.com) + ~ @license https://store.webkul.com/license.html + ~ @link https://store.webkul.com/license.html + --> + + + diff --git a/android/app/src/main/res/drawable/opening_screen.xml b/android/app/src/main/res/drawable/opening_screen.xml index f6e4544..2d26bc9 100755 --- a/android/app/src/main/res/drawable/opening_screen.xml +++ b/android/app/src/main/res/drawable/opening_screen.xml @@ -1,4 +1,14 @@ + + diff --git a/android/app/src/main/res/drawable/toggle_style.xml b/android/app/src/main/res/drawable/toggle_style.xml index 4527589..dfd66df 100644 --- a/android/app/src/main/res/drawable/toggle_style.xml +++ b/android/app/src/main/res/drawable/toggle_style.xml @@ -1,4 +1,14 @@ + + diff --git a/android/app/src/main/res/layout/activity_ar.xml b/android/app/src/main/res/layout/activity_ar.xml index 8320c9a..0bca8c7 100644 --- a/android/app/src/main/res/layout/activity_ar.xml +++ b/android/app/src/main/res/layout/activity_ar.xml @@ -1,14 +1,11 @@ + + + + diff --git a/android/app/src/main/res/mipmap-anydpi-v26/ic_launcher_round.xml b/android/app/src/main/res/mipmap-anydpi-v26/ic_launcher_round.xml index 2f2d992..7dcd3ae 100644 --- a/android/app/src/main/res/mipmap-anydpi-v26/ic_launcher_round.xml +++ b/android/app/src/main/res/mipmap-anydpi-v26/ic_launcher_round.xml @@ -1,12 +1,12 @@ diff --git a/android/app/src/main/res/values-night/colors.xml b/android/app/src/main/res/values-night/colors.xml index 9b8d16f..0d516e4 100644 --- a/android/app/src/main/res/values-night/colors.xml +++ b/android/app/src/main/res/values-night/colors.xml @@ -1,4 +1,14 @@ + + #ffffff #ffffff diff --git a/android/app/src/main/res/values-night/styles.xml b/android/app/src/main/res/values-night/styles.xml index 66df308..f52b26a 100644 --- a/android/app/src/main/res/values-night/styles.xml +++ b/android/app/src/main/res/values-night/styles.xml @@ -1,12 +1,12 @@ diff --git a/android/app/src/main/res/values/colors.xml b/android/app/src/main/res/values/colors.xml index d7ad1b6..ebe3108 100644 --- a/android/app/src/main/res/values/colors.xml +++ b/android/app/src/main/res/values/colors.xml @@ -1,4 +1,14 @@ + + #ffffff #ffffff diff --git a/android/app/src/main/res/values/strings.xml b/android/app/src/main/res/values/strings.xml index 4ae235e..13bc4d7 100644 --- a/android/app/src/main/res/values/strings.xml +++ b/android/app/src/main/res/values/strings.xml @@ -1,8 +1,18 @@ + + - AIzaSyBAlqVDV_6ec8DKG3yJPAE29HV4f-GOsdk + AIzaSywelqVDV_6ec8DKG3yJPAE29HV4f-GOsdk Results Found %d Results Found diff --git a/android/app/src/main/res/values/styles.xml b/android/app/src/main/res/values/styles.xml index 5b88b49..03de489 100644 --- a/android/app/src/main/res/values/styles.xml +++ b/android/app/src/main/res/values/styles.xml @@ -1,19 +1,16 @@ - 390844401858575 - 215374c862d4f89d04995cfaafca7e15 -