Fixing the build of GRDB on Linux #1821
Replies: 6 comments 30 replies
-
|
Thank you @thinkpractice. I don't remember now why we do not use a plain |
Beta Was this translation helpful? Give feedback.
-
|
This would indeed make it much more easy. Also to provide a custom Linux build in the future. And for now such a flag could be disable by default on Linux. Let me know when you made the changes and I will have a look if it builds on Linux :) |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I will ping you, thank you for your help. I don't quite know how to check the linux builds, actually. |
Beta Was this translation helpful? Give feedback.
-
|
One way we could do this is by adding a CI pipeline for Linux and making the pipeline run the unit tests. To run the unit test the main project will already need to be linked. If the linker fails, the test project will not build and the tests will not run. I have made such pipelines before for |
Beta Was this translation helpful? Give feedback.
-
|
Running SPM tests on Linux from What I'm really looking for is a quick way to check debug Linux builds on my machine, with minimum to zero maintenance job (because I'm lazy and I do not like friction). For now, your help checking the future changes will be enough 😅 |
Beta Was this translation helpful? Give feedback.
-
|
In that case I think it's easier for me to try and build the changes you make indeed. Then I will focus on the unit tests instead :) |
Beta Was this translation helpful? Give feedback.
-
|
Small question: what |
Beta Was this translation helpful? Give feedback.
-
|
It is currently Swift 6.0. Bumping to Swift 6.1 is, strictly speaking, a breaking change. But it would be fine: unless I'm mistaken one must use Xcode 16.4 or 26 to submit apps on the App Store (i.e. Swift 6.1+). |
Beta Was this translation helpful? Give feedback.
-
|
Then I will make it work first with Swift 6.1/6.2 and after that investigate what's happening with Swift 6.0. I'm happy it's not swift 5.x because most of the |
Beta Was this translation helpful? Give feedback.
-
You can actually still use as low as Xcode 16.0 to submit to the App Store; requirements currently are Xcode 16+ and built against the iOS 18 SDK (or equivalent for other platforms). So Xcode 16.0 would still indicate Swift 6.0 support if you are trying to support all versions that you can currently submit builds with. |
Beta Was this translation helpful? Give feedback.
-
|
Good to know. Actually the build error in swift 6.0 looks like it's related to swift structured concurrency. So it's different than the errors I already fixed. But will have to investigate this further. |
Beta Was this translation helpful? Give feedback.
-
|
I've been working now on getting the tests to compile. It seems some compiler directives are not necessary any more on Linux or can easily be changed to make the tests compile and succeed on Linux. There are a couple of CGFloat
#if canImport(CoreGraphics)to: #if canImport(CoreGraphics)
import CoreGraphics
#elseif os(Linux)
import Foundation
#endifDecimal
I tested these changes with swift 6.1 and swift 6.2. Note: swift 6.0 doesn't build. Which versions would be relevant to support? NSString, NSDate, DateSome foundation Removed
NSNumberThere's a problem with the following code in public static func fromDatabaseValue(_ dbValue: DatabaseValue) -> Self? {
switch dbValue.storage {
case .int64(let int64):
return self.init(value: int64)
case .double(let double):
return self.init(value: double)
case let .string(string):
// Must match Decimal.fromDatabaseValue(_:)
guard let decimal = Decimal(string: string, locale: posixLocale) else { return nil }
return NSDecimalNumber(decimal: decimal) as? Self
default:
return nil
}
}both the not sure yet what to do here, tried replacing it by UPDATE I was able to fix the above by a kind of hackish approach like this: public static func fromDatabaseValue(_ dbValue: DatabaseValue) -> Self? {
switch dbValue.storage {
case .int64(let int64):
let number = NSNumber(value: int64)
return number as? Self
case .double(let double):
let number = NSNumber(value: double)
return number as? Self
case .string(let string):
// Must match Decimal.fromDatabaseValue(_:)
guard let decimal = Decimal(string: string, locale: posixLocale) else { return nil }
return NSDecimalNumber(decimal: decimal) as? Self
default:
return nil
}
}The It looks like the main issue to begin with is that All tests in
However, in:
There are several failures:
These fail because the
but then:
fail. The last assertion: XCTAssertTrue(try roundTrip(NSNumber(value: Double(10000000.01))))in both test cases then fails, due to a precision error ( UPDATE 2 Ok could solve it: public static func fromDatabaseValue(_ dbValue: DatabaseValue) -> Self? {
switch dbValue.storage {
case .int64(let int64) where self is NSDecimalNumber.Type:
let number = NSDecimalNumber(value: int64)
return number as? Self
case .int64(let int64):
let number = NSNumber(value: int64)
return number as? Self
case .double(let double) where self is NSDecimalNumber.Type:
let number = NSDecimalNumber(value: double)
return number as? Self
case .double(let double):
let number = NSNumber(value: double)
return number as? Self
case .string(let string):
// Must match Decimal.fromDatabaseValue(_:)
guard let decimal = Decimal(string: string, locale: posixLocale) else { return nil }
return NSDecimalNumber(decimal: decimal) as? Self
default:
return nil
}
}
}But now I feel like I need a shower 😉 NSURLThe /// Returns a TEXT database value containing the absolute URL.
public var databaseValue: DatabaseValue {
#if os(Linux)
absoluteString.databaseValue
#else
absoluteString?.databaseValue ?? .null
#endif
}UUID
public static func fromDatabaseValue(_ dbValue: DatabaseValue) -> Self? {
switch dbValue.storage {
case .blob(let data) where data.count == 16:
return data.withUnsafeBytes {
self.init(uuidBytes: $0.bindMemory(to: UInt8.self).baseAddress)
}
case .string(let string):
return self.init(uuidString: string)
default:
return nil
}
}The This could be fixed with a guard statement. However, similarly to the case for UPDATE 3 Fixed it and its unittests with: public static func fromDatabaseValue(_ dbValue: DatabaseValue) -> Self? {
switch dbValue.storage {
case .blob(let data) where data.count == 16:
return data.withUnsafeBytes {
#if canImport(Darwin)
self.init(uuidBytes: $0.bindMemory(to: UInt8.self).baseAddress)
#else
guard let uuidBytes = $0.bindMemory(to: UInt8.self).baseAddress else {
return nil as Self?
}
return NSUUID(uuidBytes: uuidBytes) as? Self
#endif
}
case .string(let string):
return NSUUID(uuidString: string) as? Self
default:
return nil
}
}TODOs
With this we: |
Beta Was this translation helpful? Give feedback.
-
|
I really appreciate this level of care 🤩 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks! Yes I want to really port it to Linux and be sure that it will work. Tests will be essential for that 😄 |
Beta Was this translation helpful? Give feedback.
-
|
@groue I've been looking into the next issue with the tests on Linux and I found one that is harder to fix. Mostly, because I'm not very familiar with the func testDefaultLabel() throws {
let dbPool = try makeDatabasePool()
dbPool.writeWithoutTransaction { db in
XCTAssertEqual(db.configuration.label, nil)
XCTAssertEqual(db.description, "GRDB.DatabasePool.writer")
// This test CAN break in future releases: the dispatch queue labels
// are documented to be a debug-only tool.
let label = String(utf8String: __dispatch_queue_get_label(nil))
XCTAssertEqual(label, "GRDB.DatabasePool.writer")
Do I understand correctly that |
Beta Was this translation helpful? Give feedback.
-
|
Oh yeah, it looks like I did in fact punt on these tests 😟 GRDB.swift/Tests/GRDBTests/DatabasePoolConcurrencyTests.swift Lines 1272 to 1275 in cbc1464 I agree that it would be nice to find a solution for testing this on Linux without using |
Beta Was this translation helpful? Give feedback.
-
|
All tests that use NSFileCoordinator aim at testing the sample codes in the Sharing a Database guide. It's for applications that open a database from multiple processes, in which case NSFileCoordinator helps preventing concurrent setup performed by the app (early GRDB code, app migrations, etc - after this setup phase, SQLite handles concurrency on its own). I suppose that Linux/Android apps would need to go down to I think those tests can safely be skipped or disabled on platforms that do not have NSFileCoordinator. One day we'll extend the Sharing a Database guide for Linux and Android. |
Beta Was this translation helpful? Give feedback.
-
|
@groue Do I understand correctly that you lock the file using I agree we can ignore it for now, so I can create a PR with the bulk of my changes and tackle more difficult issues one by one in a later stage. I didn't have much time today but hope to continue tomorrow. I will have a look at the work of @marcprux and integrate them with my changes. @marcprux is there an easy way I can test building the package on Android? Would I need skip.tools for that on my mac or can I test it on Linux? This would make it easier to test if my changes for Linux work cross-platform. |
Beta Was this translation helpful? Give feedback.
-
You don't need Skip to build and test against Android — you can install the (soon-to-be-official) cross-compilation SDK from https://github.com/swift-android-sdk/swift-android-sdk/releases on either macOS or Linux. Or, if you use GitHub actions, you can use https://github.com/marketplace/actions/swift-android-action to run the build and test as part of a CI workflow. But it won't work, since there's no (publicly available) SQLite library that is available for linking on Android. This was a large part of the impetus for the SQLCipher effort (as mentioned at #1708 (comment)), in order to facilitate using a source SQLite build with GRDB. So I'd recommend just ignoring Android for the time being. Once I get back to working on the solution there, I'm sure your Linux work will cover 90% of the Android-specific build concerns. |
Beta Was this translation helpful? Give feedback.
-
We're talking about (We could say that those tests should be removed, arguing that they are just asserting the behavior of NSFileCoordinator and that "testing the framework" is a bad practice - but we're also asserting that DatabasePool evolution must be compatible with user code that uses NSFileCoordinator and that any deadlock must be prevented.) Arguably |
Beta Was this translation helpful? Give feedback.
-
NSErrorBridging of #if !os(Darwin)
func testNSErrorBridging() throws {
let dbQueue = try makeDatabaseQueue()
try dbQueue.inDatabase { db in
try db.create(table: "parents") { $0.column("id", .integer).primaryKey() }
try db.create(table: "children") { $0.belongsTo("parent") }
do {
try db.execute(sql: "INSERT INTO children (parentId) VALUES (1)")
} catch let error as NSError {
XCTAssertEqual(DatabaseError.errorDomain, "GRDB.DatabaseError")
XCTAssertEqual(error.domain, DatabaseError.errorDomain)
XCTAssertEqual(error.code, 787)
XCTAssertNotNil(error.localizedFailureReason)
}
}
}
#endif |
Beta Was this translation helpful? Give feedback.
-
|
@groue @marcprux Just a short update. I'm preparing a document with all the changes that I made so far. This will serve as the documentation for the PR I'm going to submit. I need to create a new fork as I think I didn't fork the code properly the first time. This will also allow me to check whether all changes I've made are properly documented. I hope to finish this before next week as I will be travelling the next two weeks (SwiftLeeds 🥳) @groue did you make any progress with adding a non-Darwin/Linux clause to @marcprux just a small question, when would you use |
Beta Was this translation helpful? Give feedback.
-
I was planning to have a look at it on the next Saturday (the day after tomorrow) - or Sunday, depending on the nature of my Friday evening ;-) |
Beta Was this translation helpful? Give feedback.
-
You are right, they are basically the same. I had picked |
Beta Was this translation helpful? Give feedback.
-
|
@groue 🥳 🤣 I'm almost there. I applied the fixes above and more and adapted them using the examples and remarks by @marcprux. The code builds and links on Linux now and apart from some tests that are skipped, the tests all succeed. Hopefully, my changes will also help the Android build. The only thing that I made Linux specific is this compiler directive around the WAL code: #if (SQLITE_ENABLE_SNAPSHOT || (!GRDBCUSTOMSQLITE && !GRDBCIPHER)) && !os(Linux)If I understand correctly, @marcprux is going to add a custom build for There's one issue remaining though. The I will see if I can find an alternative swift-lint package or if I can fix the build. @marcprux are you using UPDATE I found this in the SwiftLint repo: On Linux, SwiftLint is not self-contained and requires libsourcekitdInProc.so as a runtime dependency. It's contained in all Swift toolchains. Citing from the README: On Linux, SourceKit is expected to be located in /usr/lib/libsourcekitdInProc.so or specified by the LINUX_SOURCEKIT_LIB_PATH environment variable Will try to apply it later and update the PR (if necessary). |
Beta Was this translation helpful? Give feedback.
-
|
@groue @marcprux you can find my patches for Linux here I created a new PR with some upstream changes integrated here: #1825 |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi there,
I really like
GRDBand theSQLiteDatapackage PointFree has built on top of it and I would like to use them both on Linux. I had already made some quick changes to GRDB to make it work on Linux a while back, but now I'm looking into making these changes in a "proper" way so they can contributed back toGRDB.Most of
GRDBactually builds on Linux just fine. The only change is that I needed to do to make it build was adding ato handle non-Sendable protocol conformance in
GRDB/Core/DispatchQueueActor.swift. Not sure if that's the right approach but it made everything build!The problem after that is linking. The standard
libsqlite3-devinstalled withaptordnfdoesn't have WAL enabled and when trying to useGRDBin another swift package or app in Linux, you get a linker error. Functions likesqlite3_snapshot_get,sqlite3_snapshot_openare not found.There would be two ways around it I think, either:
GRDBon Linux, orI chose the latter for now as it is a more easy fix. I updated the following conditional compilation directive:
#if SQLITE_ENABLE_SNAPSHOT || (!GRDBCUSTOMSQLITE && !GRDBCIPHER)to:
#if (SQLITE_ENABLE_SNAPSHOT || (!GRDBCUSTOMSQLITE && !GRDBCIPHER)) && !os(Linux)This compiler directive is present (and updated) in the following files:
GRDB/Core/DatabasePool.swiftGRDB/Core/DatabaseSnapshotPool.swiftGRDB/Core/WALSnapshot.swiftGRDB/Core/WALSnapshotTransaction.swiftGRDB/ValueObservation/Observers/ValueConcurrentObserver.swiftUpdating the compiler directive as above will fix the build with the standard
libsqlite3-devon Linux. If that's fine we can keep it at that. I'm not completely sure yet what WAL does but if I understand correctly, using it is mostly a performance consideration. If it would be a nice to have on Linux too, I can research adding a customsqlitebuild for Linux with WAL enabled. I wonder why it is not included in the standardlibsqlite3-devthough?Adding the changes above will fix linking of the
GRDBcode. The build of the tests does fail however. I also fixed that build but now have a lot of failing tests. I will describe the details in a next posts in this thread.Beta Was this translation helpful? Give feedback.
All reactions