feat: add modern nativescript-sqlite - #4
Open
edusperoni wants to merge 16 commits into
Open
edusperoni wants to merge 16 commits into
edusperoni wants to merge 16 commits into
Conversation
edusperoni
force-pushed
the
feat/sqlite-plugin
branch
from
May 20, 2026 03:52
f814e56 to
99af821
Compare
…MA key * The key was interpolated into the statement unquoted, so a key containing a single quote produced invalid SQL and failed the open. * That also made SQLCipher raw keys unusable: they are recognised from the string value x'<hex>', which could not survive the broken quoting. * Add encryptionKeyFormat: 'passphrase' | 'raw' (default passphrase). A raw key skips the PBKDF2 derivation each pooled connection would otherwise run, which costs no strength when the key is full-entropy random. * Reject a raw-looking key that did not ask for raw. SQLCipher switches to key bytes on its own for anything shaped like x'<64 hex>', and which key you get must not depend on the shape of a string. * Verified against the SQLCipher CLI: the escaped form opens a raw-keyed database, while an unquoted operand is a blob literal that PRAGMA rejects outright.
@nrwl/js is no longer installed, so any change that makes these projects affected failed the build job with 'Unable to resolve @nrwl/js:tsc'.
… a single writer on both platforms (#8) * feat: add modern nativescript-sqlite * fix: transactions should wait on on another * chore: revert version number to 0.0.1 in package.json * chore: 0.0.2 * feat: add encryption support and new getArray methods to SQLite plugin * feat: add android v8 sqlite implementation * Remove headers from the Android native SQLite implementation * fix(sqlite): free Android DBInstance on close and GC * fix(sqlite): stop silently truncating multi-statement SQL * test(sqlite): add implementation-agnostic benchmark suite * test(sqlite): assert row counts in bulk read benchmarks * test(sqlite): report benchmark failures through a file * test(sqlite): verify row counts after write benchmarks * feat(sqlite): build the Android implementation against V8 14.9 Targets the V8 embedded in @nativescript/android 9.1. The public V8 headers are no longer vendored: include.gradle downloads the release pinned in native/v8-headers.properties, verifies its SHA-256 and caches the extracted headers under the Gradle user home. nscsqlite.v8IncludeDir bypasses the download and nscsqlite.v8HeadersCacheDir relocates the cache. The link stub now comes from whichever runtime flavor is present, since all of them export the public V8 API. * chore(demo): use @nativescript/android 9.1 * perf(sqlite): serve awaited statement chains without returning to the looper A statement dispatched from the continuation of the previous one is answered by the spinning worker within microseconds. Going back to the looper in between blocks the JS thread in epoll_wait and costs a thread wake-up per statement, which dominated awaited loops (about 18us per statement on @nativescript/android 9.1, against 1us of actual work). The drain now keeps polling the queues fed during the batch it just ran, bounded per completion and per looper callback. Completion queues are shared-ownership and the drain no longer touches the dispatcher after running a completion, because close() deletes the DBInstance from inside its last one. * test(sqlite): add a headless correctness-test trigger nscbench=test runs the demo's correctness suite through a wrapper that calls each test individually and writes pass/fail plus the captured console output to nscsqlite-test-<label>.json, so a release build can be checked from adb. testSQLCipher is skipped unless the runtime reports a SQLCipher build. * feat(sqlite): implement the Android binding on Node-API Mirrors the V8 backend's JS-visible API on top of Node-API: the same NSCSQLite class, argument handling, result shapes and error codes, the same per-connection FIFO dispatchers, and the same JSON fast path for async results. Completions reach the JS thread through one threadsafe function per env, which drains the whole pending queue per signal; when a continuation dispatches more work the drain spins briefly for the answer instead of paying a looper wake-up per awaited statement. * feat(sqlite): select the Android binding backend at build time nscsqlite.backend=v8|napi (or NSCSQLITE_BACKEND, since Gradle cannot pass a dotted property through the environment) picks the source set and the headers: the napi build takes Node-API headers from the runtime AAR's Prefab package and never downloads or includes V8. * feat(sqlite): resolve the native class from the napi module when needed The napi backend has no global to install into: its Node-API module is reached through the runtime's own require(), which webpack must be kept from rewriting. * feat(sqlite): report the real SQLite build in getRuntimeInfo on iOS The iOS side answered with a hard-coded stub, so the one API that tells an app which SQLite it actually linked was useless on the platform where the app chooses that SQLite itself. The compile-option enumeration is guarded: a SQLite built with SQLITE_OMIT_COMPILEOPTION_DIAGS has no sqlite3_compileoption_get to link against. * build(sqlite): let the app choose the Android SQLite, and download the presets Android had no equivalent of what iOS gets for free: on iOS the plugin declares no SQLite dependency and links whatever the Podfile provides, so an app can ship a custom engine without the plugin knowing. Android has no link line an app can influence, so the equivalent is a directory of its own: <App_Resources>/Android/nscsqlite/CMakeLists.txt defining one target, `nscsqlite_sqlite`, which the plugin add_subdirectory()s and links. `nscsqlite.sqliteProjectDir` overrides the location. `bundled` and `sqlite3mc` are now instances of that same contract rather than a second code path, so the presets double as worked examples. The `sqlcipher` preset is gone: sqlite3mc compiled with CODEC_TYPE_SQLCIPHER in legacy mode reads and writes SQLCipher 4 databases through the same `encryptionKey`, bundles its own crypto, and does not drag a ~3 MB libcrypto.so per ABI into the APK — while the old preset only built at all because its vendored amalgamation had been patched away from upstream. `custom` is gone because it never worked: the NDK toolchain re-roots find_library() under the NDK, so it could not see the user's directory. Both amalgamations are downloaded and checksum-verified at build time instead of being vendored, which removes 572k lines from the repository. The upstream zip is byte-identical to the amalgamation it replaces. `nscsqlite.sqliteFlags` is now additive on top of each preset's defaults rather than replacing them, with a required set the user cannot drop. Two link options apply to every configuration now: --exclude-libs,ALL, so libnscsqlite.so exports no sqlite3_* symbols and cannot be bound to by another SQLite in the process, and max-page-size=16384, which Play requires for apps targeting API 35+. * feat(sqlite): bind through Node-API by default on Android The V8 backend reaches into the runtime's V8 through its public C++ headers, which are not part of any stable ABI: the pointer-compression defines and the header version have to match the V8 inside @nativescript/android exactly, so every runtime upgrade needs a matching header pin and a rebuild. Node-API is versioned and stable, and the default build now downloads nothing to compile against. The V8 backend stays selectable with nscsqlite.backend=v8 as the reference implementation the Node-API one is measured against. * perf(sqlite): call the Android native object directly instead of through a Proxy Every property access on the database object went through a Proxy `get` trap that minted a fresh closure, so each of the thousands of calls a busy app makes allocated one and defeated inline caching — all to rewrap native errors as SQLiteError. The rewrapping is now explicit at each call site, which is more lines but no allocation and a monomorphic call. * feat(sqlite): bring the Android database options to parity with iOS encryptionKeyFormat, onOpen and serialized existed only on iOS, so an app using any of them could not run on Android at all. The three land together because they share one ordering constraint inside the open sequence: PRAGMA key, then onOpen, then journal_mode=WAL and query_only. onOpen has to see the decrypted database and still precede every query, and WAL has to come after both. The PRAGMA key operand is now quoted the way iOS quotes it, so a key containing a single quote can neither break the statement nor inject into it, and SQLCipher's raw-key form still reaches the codec as text. serialized runs everything on one connection. That connection is opened FULLMUTEX rather than NOMUTEX because on Android the synchronous methods run on the JavaScript thread while async work runs on a dispatcher thread, and both use the same handle. Note the resulting ordering difference from iOS, which queues sync calls behind async ones with dispatch_sync: an un-awaited async write followed by a sync read may not see the write here. Failures during open now surface as an open error carrying the SQLite code rather than succeeding and failing on the first query, which is what iOS has always done; a wrong encryption key is the case that matters. The open error also reaches JavaScript with a numeric code, so the TypeScript layer can build a real SQLiteError instead of a bare Error. Every public operation on a connection now holds a recursive mutex. SQLite's own FULLMUTEX serializes individual calls but not the "call fails, then read sqlite3_errmsg" sequence: another thread preparing a statement on the same handle frees the pending error, so a failing statement could read freed memory or report the other thread's message. The *Sync methods and getRuntimeInfo reported a closed database as a rejected promise, which no caller could catch; they throw now, and only the async methods reject. The compile-option enumeration in getRuntimeInfo is guarded: a SQLite built with SQLITE_OMIT_COMPILEOPTION_DIAGS has no sqlite3_compileoption_get to link against. * fix(sqlite): checkpoint the WAL when the database is closed close() dispatched every connection's close concurrently on its own thread, so each one could still see its siblings attached. SQLite only checkpoints and unlinks the -wal file when the closing connection can take an exclusive lock on the database, which none of them could get; the WAL was left on disk until some later close in the same process happened to win the race. Seen in half of the device runs, with a 450 KB -wal beside a 4 KB database. The readers now close first and the writer only once the last of them is done. * test(sqlite): cover the new options and encrypted WAL on a device The correctness suite decided whether to run its encryption test by looking for the compile option EXTRA_INIT=sqlcipher_extra_init, which sqlite3mc does not define — so the test silently skipped on exactly the build that can encrypt. No compile option or pragma identifies a codec across engines; whether a keyed database is unreadable without its key does, so that is the probe now. It also makes the test meaningful: it previously only checked that a keyed database round-trips, which a plaintext build passes too. The encrypted-WAL test exercises a keyed database under the writer + reader pool with WAL: concurrent readers against a running writer, prepared statements, the sync connection, durability across close and reopen, wrong and missing keys, and a fixture written by official SQLCipher on the host. It adapts to the build it finds: on a codec-less SQLite it runs the same sequence unkeyed, so WAL, the pool and durability are still covered, and on the bundled preset it requires the keyed open to be refused. * docs(sqlite): document the Android SQLite build and how to replace it The Android half of the README described three flavours that no longer exist. It now covers choosing between the two presets, what sqlite3mc does and does not give you, the cost of a passphrase, and the directory an app drops in to compile its own SQLite — with five worked CMakeLists files under docs/android-custom-sqlite, four of them reduced from builds that were actually linked against the plugin. Encryption caveats are their own section because they are not an Android topic: `PRAGMA key` against an engine with no codec succeeds and writes plaintext on both platforms, and the only proof that works on any engine is to reopen a keyed database without its key and require that to fail. * test(sqlite): satisfy the demo typecheck in the encrypted-WAL test `PRAGMA integrity_check` was read through a generic that does not satisfy the SQLiteRow constraint, and the codec mode was assigned from inside a callback, which narrowed it away from its declared type at the point it is read. * fix(sqlite): write the whole blob placeholder in iOS select results The prefix was appended with a hard-coded length of 11 against a 12-character literal, so the trailing colon never made it into the JSON. Any select returning a BLOB therefore produced `{"__blob__"0}` and the result failed to parse, taking the app down with a SyntaxError. Deriving the length from the literal keeps the two from drifting apart again. * fix(sqlite): report the primary SQLite code in SQLiteError on iOS Extended result codes are enabled on every connection, so sqlite3_errcode returns the extended form and a UNIQUE violation surfaced as 2067 rather than the SQLITE_CONSTRAINT (19) the exported constants describe. Code comparing against those constants never matched. extendedCode is unchanged and still carries the full value. * feat(sqlite): describe the single-writer connection model in the API The synchronous methods used to run on a connection of their own, so a sync write and an async write were two writers on one file arbitrated only by the busy timeout — and a sync write issued while an async transaction was open stalled the JavaScript thread for the whole timeout and then failed, because that transaction needed the JavaScript thread to finish. They now share the writer with the asynchronous methods, ordered behind whatever is already queued on it. That makes the semantics worth stating rather than discovering: a sync read sees an open transaction's uncommitted rows, a sync call after un-awaited writes sees all of them, and executeSync refuses to silently join a transaction. SyncTransaction, transactionSync(), the sync methods on the transaction objects and the joinTransaction escape hatch are the sanctioned ways to do synchronous work inside a transaction. asyncOpen and initialized() move connection opening — which with a passphrase is a PBKDF2 derivation per connection — off the JavaScript thread. * feat(sqlite): run Android sync calls on the writer, and open connections off the JS thread The dedicated sync connection is gone. Every synchronous method now runs on the writer, claimed through the writer pool's queue mutex: inline on the calling thread when the pool is idle (12-14 ns, no thread hop and no wake) and otherwise queued behind the work already there, so sync and async calls are mutually exclusive and FIFO. The claim is re-entrant for the thread holding it, which is what lets transactionSync hold the writer across a whole callback while everything dispatched during it stays queued. Because the exclusion is ours, serialized mode goes back to SQLITE_OPEN_NOMUTEX and regains the statement cache. Connections now open on the thread that owns them, so a passphrase costs one derivation on the JavaScript thread instead of poolSize + 2. asyncOpen moves the writer's open off it too, and initialized() reports the outcome; its promise is created per call, so an open that fails with nobody watching cannot become an unhandled rejection. commitTransaction and rollbackTransaction previously discarded the result of COMMIT/ROLLBACK and always resolved — a failed commit looked like success. They report it now, and a failed COMMIT is followed by a ROLLBACK, because leaving the transaction open would let the next queued write join it. Also fixes a pre-existing teardown race: the V8 dispatcher closed its eventfd before joining the worker that writes to it, so a close racing an in-flight statement could write to a reused descriptor. * feat(sqlite): run iOS sync calls on the writer, and open connections off the JS thread Mirrors the Android change. The dedicated sync connection is gone: every synchronous method resolves the connection and queue that own its work and runs through one dispatch_sync, so there is no serialized/pooled split left. transactionSync holds the writer for the whole callback by suspending its queue, so work dispatched from inside the callback stays queued until the transaction has committed or rolled back. Readers are opened by their own queues, behind a group the writer leaves, because only the writer opens with SQLITE_OPEN_CREATE and a reader that won the race against a database that did not exist yet failed with SQLITE_CANTOPEN. A reader that fails to open is no longer skipped: it keeps its slot and reads routed to it report why it failed. A synchronous open failure raised an NSException and the TypeScript layer read the SQLite code from nativeException.userInfo, which @nativescript/ios 8.9.5 does not expose at all — so every such failure reached JavaScript as SQLITE_ERROR whatever the cause. The open returns an NSError instead and both routes now produce the same message and the same code. The sync selects gained the blob side-channel the async ones already had, so a BLOB read through getSync comes back as an ArrayBuffer rather than an unhydrated placeholder, and getArraySync returns the first row like its asynchronous twin instead of every row. Transaction ids are now validated on the asynchronous in-transaction methods too; they previously ignored the id and ran on the writer whatever was passed. * test(demo): trigger the headless runs from iOS launch arguments The benchmark and test runners could only be started from an Android launch intent, so nothing below the argument parsing had ever run on iOS. Launch arguments carry the same three values there. * build(demo): link SQLite in the iOS demo The plugin deliberately declares no SQLite of its own on iOS and links whatever the app provides, which the demo never did — so an iOS build failed with undefined _sqlite3_* symbols and the demo had apparently never been built for the platform. * test(sqlite): cover the single-writer model, sync transactions and background opens Two of these caught real bugs on their first run on a simulator: readers racing the writer for SQLITE_OPEN_CREATE under asyncOpen, and a transactionSync callback returning a promise. getArraySync's existing assertion used a query matching one row, so it could not tell the method apart from selectArraySync; it now queries two. * docs(sqlite): document the single-writer connection model The README described a dedicated connection behind the *Sync methods and `poolSize + 2` connections keyed on the JavaScript thread, neither of which is true any more. It now covers what the sync methods share with the async ones and what that means for what they see, the three sanctioned ways to do synchronous work inside a transaction, opening off the JavaScript thread with asyncOpen and initialized(), and the two remaining differences between the platforms. The passphrase cost table is remeasured, and says plainly that the readers' derivations moved off the JavaScript thread rather than disappearing: they reappear as latency on the first read routed to each reader. The thenable message is aligned with the one iOS reports. * fix(sqlite): wake the pool's workers when an inline claim is released during shutdown A worker that parks while the pool is claimed waits for busy_ to clear, and the release only signalled when tasks were queued, so a shutdown begun under a claim could never be joined. * docs(sqlite): state where the SQLCipher interop was verified, and the open latency as it is * docs(sqlite): list what still differs between iOS and Android * build: use the @nx/js executor in the remaining packages @nrwl/js is no longer installed, so any change that makes these projects affected failed the build job with 'Unable to resolve @nrwl/js:tsc'. * fix(demo): import the headless runners through @demo/shared The relative imports into tools/demo tripped @nx/enforce-module-boundaries and failed the demo's lint target. * docs(sqlite): describe engine-side encryption without naming a mechanism * docs(sqlite): drop an example from the auto-extension note --------- Co-authored-by: Dylan Llewellyn <46717769+herefishyfish@users.noreply.github.com>
…iter first (#9) * feat(sqlite): let the caller order a connection's open sequence The sequence was fixed: key, then onOpen, then WAL. Some engines select their cipher with pragmas that have to precede PRAGMA key; page_size and auto_vacuum only take on a new file before the journal mode switches; and not every database wants WAL at all. None of that was reachable, and writing PRAGMA key by hand in onOpen is not a way to reach it — that bypasses the plugin's quoting and its passphrase/raw handling, and puts the key inside a statement. openSequence names the whole order, with two markers for the steps the plugin owns and an optional scope so a statement can run on the writer or the readers alone. onOpen becomes sugar for the default sequence, so there is one code path and existing behaviour is unchanged. A sequence that cannot work is refused before anything is opened, including under asyncOpen, where nothing else reports synchronously. The refusals are programming errors rather than open failures: combining onOpen with openSequence, repeating a marker, or putting wal before key with a key set. Setting encryptionKey without a key marker is refused for a different reason — a forgotten marker would otherwise write the database unencrypted. * feat(sqlite): run the caller's open sequence on Android, writer first The connection layer walks the step list it is handed instead of a fixed key/onOpen/WAL order, filtering each step by scope — the serialized connection counts as the writer — and still forcing query_only=ON onto readers after the whole sequence, where it is not the caller's to move. A failing step reports its index rather than its text: `open step <i> failed: <sqlite message>`. Steps skipped by scope keep their index, so the number matches what the caller wrote. The key step also drops the SQLite message entirely if it ever contains the key, since SQLite names the token it choked on. Readers no longer start opening until the writer's sequence has finished. A reader that touches a brand-new file first can leave it without the page_size the writer was about to set, and can hold the shared lock the writer needs to switch the journal mode. Reader opens are now chained from the writer's open completion on the runtime thread, which is also where close() runs, so a connection is either opened and then closed or never opened at all. Two consequences of that reordering are handled rather than left: a reader the pool never started is given the writer's failure, so every method reports why the database could not be opened instead of claiming it is closed; and a read issued before the writer's verdict is parked and flushed when it lands, because awaiting initialized() is documented as optional. recordOpenFailure is written from the runtime thread while a worker may be reading it, so the open-failure fields are now guarded by the mutex that already protects the last error. * feat(sqlite): run the caller's open sequence on iOS, writer first Mirrors the Android change. The connection walks the step list it is given, filtering by scope — the serialized connection counts as the writer — and query_only=ON still follows the whole sequence on readers. A failing step reports `open step <i> failed: <sqlite message>`, with steps skipped by scope keeping their index so the number matches what the caller wrote, and the key step dropping SQLite's message if it ever contains the key. Readers already waited on a group the writer leaves, but the writer left it on its failure path too, so the readers went on to open and fail on their own. The writer now publishes its verdict before leaving the group and each reader checks it: on failure no reader opens, and each is given the writer's error so every method reports why the database could not be opened rather than that it is closed. The group is still left rather than withheld, because closing drains those same queues. PRAGMA query_only=ON was run with its result assigned to a string nobody read, so a reader that could not be made read-only opened anyway. It is a step of the sequence now and aborts the open like any other. A read issued before the writer's verdict needs no special handling here: the reader queues and their open blocks are both created before the open call returns, and they are serial, so a later read necessarily queues behind a block that is itself waiting on the writer. * fix(sqlite): validate an open step's scope even when its SQL is empty The scope was only parsed on the way to building a step, which an empty statement never reaches, so a typo in `on` passed silently. * test(sqlite): cover the caller-controlled open sequence The refusals, the default sequence matching its explicit spelling, a new file keeping its rollback journal when no wal step is given, a writer-scoped page_size taking effect before WAL, scoping observed from the pool through cache_size, and a failing step naming its index without repeating the statement. Two of these exist because the reordering broke guarantees the tests did not reach: a pooled read issued before initialization has to queue rather than be turned away, and on a failed open it has to report why the database could not be opened rather than that it is closed. The existing asyncOpen test missed both by awaiting initialized() first and by using the sync path, which runs on the writer. The codec-gated case is the one that proves a step really runs before the key: a database created under a non-default cipher cannot be opened by the default sequence with the same key, and can be with the same sequence. * docs(sqlite): document how to control the open sequence The ordering constraints are SQLite's rather than the plugin's, so they are stated as such: the key before anything that reads the file, WAL never before the key, page_size and auto_vacuum only on a new file before WAL. Also why not to hand-write PRAGMA key — the plugin quotes it, decides passphrase against raw key material, and keeping it a marker keeps it out of a statement — and what leaving out the wal marker costs a reader pool.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.