fix(script): read the DOM instead of serializing and reparsing it

.changeset/read-the-dom-instead-of-reparsing-it.md

@@ -0,0 +1,7 @@
+---
+'@wdio/devtools-script': patch
+---
+
+Serialize the live DOM directly instead of writing it out as HTML and parsing that back. The collector already stands in the document, so the round trip bought nothing and cost a 148 KB HTML parser plus a view library inside a script injected into every page: 92% of the bundle, which drops from 213 KB to 10 KB.
+
+That size was not merely wasteful. The bundle is registered as a BiDi preload, and headed Chrome 154 degrades superlinearly with a preload's size (measured: 50 KB doubled the first navigation, 100 KB never completed), so a run with the service attached could hang outright. Reading the DOM also describes the page the browser actually built rather than what a second parser makes of its markup, which is what `localName` and already-adjusted attribute names give for free.

.changeset/stop-pinning-a-browser-driver.md

@@ -0,0 +1,8 @@
+---
+'@wdio/selenium-devtools': patch
+'@wdio/nightwatch-devtools': patch
+---
+
+Stop pinning `chromedriver` as a dev dependency. A pinned driver rots against whatever Chrome a developer has installed, and pnpm puts the package's `node_modules/.bin` on `PATH`, where Selenium Manager finds it, prefers it over resolving one itself, and on a version mismatch only warns before returning it anyway. The adapters' examples then failed to start a session at all, against any Chrome whose major had moved on from the pin.
+
+With no driver on `PATH`, Selenium Manager resolves one matching the installed browser. Nightwatch needs no package either: its Chrome service builder reports `requiresDriverBinary: false` and passes an unset `server_path` through to that same resolver, and it declares `chromedriver` an optional peer. This is a development-only dependency, so nothing changes for consumers of either package.

CLAUDE.md

@@ -26,7 +26,7 @@ Run from repo root unless noted.
 | `pnpm test:watch` | Run vitest in watch mode. |
 | `pnpm test:coverage` | Run vitest with v8 coverage. The thresholds in `vitest.config.ts` are aspirational, not a gate: that file states CI does not run this and the suite is currently below all four. CI runs `test`/`lint`/`test:ui`. |
 | `pnpm lint` | Lint all packages in parallel. Includes `eslint-plugin-security` for a subset of CodeQL findings; deeper taint-flow checks surface on the PR's CodeQL scan. |
-| `pnpm demo:wdio` / `pnpm demo:nightwatch` / `pnpm demo:selenium` | Run the per-framework example projects. Useful for manual verification of UI or runtime changes. |
+| `pnpm demo:wdio` / `pnpm demo:nightwatch` / `pnpm demo:selenium` | Run the per-framework example projects. Useful for manual verification of UI or runtime changes. **No package pins a browser driver**, deliberately: a pinned `chromedriver` rots against whatever Chrome the developer has, and because pnpm puts the package's `node_modules/.bin` on `PATH`, Selenium Manager finds it there, prefers it over resolving, and on a major mismatch only *warns* before returning it anyway. The session then fails to start. With nothing on `PATH` both adapters resolve a matching driver (Nightwatch's Chrome service builder sets `requiresDriverBinary: false` and passes an unset `server_path` through to the same resolver), and nightwatch declares `chromedriver` an optional peer. |
 | `pnpm demo:wdio:mobile` / `:selenium:mobile` / `:nightwatch:mobile` / `:python:mobile` | The same, against Appium. All four build the same capability bag and drive the Clock app that ships with every Android system image — starting a timer, pausing it, clearing it — so a native example needs no `.apk`. `DEVTOOLS_MOBILE_PLATFORM=ios` runs the iOS spec instead — all four adapters — which drives Settings because the simulator ships no Clock, from a separate spec per platform rather than a branch. The simulator is chosen by udid and defaults to whichever is already booted, and an unmatched `IOS_DEVICE_NAME` is refused: naming one that does not exist makes the XCUITest driver create and boot it, every run, rather than fail. That refusal and the booted-simulator preflight are **local** policy — `xcrun simctl` enumerates local simulators and nothing else — so `IOS_UDID` and a non-local `APPIUM_HOST` both bypass them, or a real device and a cloud grid would be refused a run they were correctly configured for. `DEVTOOLS_MOBILE=web` drives the device's own browser on both platforms — Chrome on Android, Safari on iOS, which the XCUITest driver serves without a chromedriver. `examples/MOBILE.md` holds the prerequisites and the `DEVTOOLS_MOBILE` / `APPIUM_APP` switches; `DEVTOOLS_MODE=trace` flips any demo to trace mode. |
 | `pnpm dev` | Run all packages in parallel dev mode. |
 | `python3 packages/selenium-devtools-py/scripts/changes.py next-version` | The version a Python-adapter release would publish, from the fragments pending in `changes/`. `check --base <ref>` is the CI gate; `apply` is what the release runs. |

package.json

@@ -39,10 +39,7 @@
       "vite": "^8.0.7",
       "@types/node": "26.2.0",
       "@codemirror/state": "6.5.4"
-    },
-    "onlyBuiltDependencies": [
-      "chromedriver"
-    ]
+    }
   },
   "devDependencies": {
     "@changesets/cli": "^2.31.0",

packages/nightwatch-devtools/package.json

@@ -66,7 +66,6 @@
     "@types/ws": "^8.18.1",
     "@wdio/devtools-core": "workspace:^",
     "@wdio/devtools-shared": "workspace:^",
-    "chromedriver": "^151.0.5",
     "nightwatch": "^3.16.0",
     "tsup": "^8.5.1",
     "typescript": "^6.0.3"

packages/script/package.json

@@ -21,9 +21,6 @@
   },
   "devDependencies": {
     "@wdio/devtools-shared": "workspace:^",
-    "htm": "^3.1.1",
-    "parse5": "^8.0.1",
-    "preact": "^10.29.2",
     "vite": "^8.0.16",
     "vite-plugin-singlefile": "^2.3.3"
   },

packages/script/src/utils.ts

@@ -1,83 +1,94 @@
-import {
-  parse,
-  parseFragment as parseFragmentImport,
-  type DefaultTreeAdapterMap
-} from 'parse5'
-import { h } from 'htm/preact'
 import type { SimplifiedVNode } from '../types.ts'
 
 import { log } from './logger.js'
 
-export type vFragment = DefaultTreeAdapterMap['documentFragment']
-export type vComment = DefaultTreeAdapterMap['commentNode']
-export type vElement = DefaultTreeAdapterMap['element']
-export type vText = DefaultTreeAdapterMap['textNode']
-export type vChildNode = DefaultTreeAdapterMap['childNode']
-
-function createVNode(elem: { type: unknown; props: unknown }) {
-  const { type, props } = elem
-  return { type, props } as SimplifiedVNode
+/** Build the wire node the replay reads.
+ *
+ *  `children` follows the shape `h()` produced when this went through preact:
+ *  absent for none, the child itself for one, an array from two. The app's
+ *  `transform` and every archive already written depend on that distinction.
+ *  `key` and `ref` are dropped for the same reason — preact lifted them off
+ *  props, so they have never reached the wire. */
+function vnode(
+  type: string | undefined,
+  props: Record<string, unknown>,
+  children: (SimplifiedVNode | string)[]
+): SimplifiedVNode {
+  const normalized: Record<string, unknown> = {}
+  for (const name in props) {
+    if (name !== 'key' && name !== 'ref') {
+      normalized[name] = props[name]
+    }
+  }
+  if (children.length === 1) {
+    normalized.children = children[0]
+  } else if (children.length > 1) {
+    normalized.children = children
+  }
+  return { type, props: normalized } as SimplifiedVNode
 }
 
-export function parseNode(
-  fragment: vFragment | vComment | vText | vChildNode
-): SimplifiedVNode | string {
-  const props: Record<string, unknown> = {}
-
-  if (fragment.nodeName === '#comment') {
+const errorNode = (className: string, err: unknown) =>
+  vnode('div', { class: className }, [(err as Error)?.stack ?? String(err)])
+
+/** Serialize a LIVE DOM node. The collector stands in the document, so the tree
+ *  is read directly rather than serialized to HTML and parsed back — that round
+ *  trip is what put a 148 KB HTML parser in a script injected into every
+ *  document, and a preload that size stalls navigation in headed Chrome (#403).
+ *
+ *  Reading the DOM also describes the page the browser actually built, rather
+ *  than what a second parser makes of its markup: `localName` keeps the case
+ *  foreign elements need (`linearGradient`), and attribute names arrive already
+ *  adjusted (`viewBox`), both of which parse5 had to special-case. */
+export function parseNode(node: Node): SimplifiedVNode | string {
+  if (node.nodeType === Node.COMMENT_NODE) {
     // Drop comment content — returning its data rendered the comment as visible
     // text on replay (e.g. an IE conditional comment's `<![endif]` showed as
     // text and added a line box that shifted the whole page layout down).
     return ''
   }
-  if (fragment.nodeName === '#text') {
-    return (fragment as vText).value
-  }
-
-  const { childNodes, attrs, tagName } = fragment as vElement
-  for (const p of attrs || []) {
-    props[p.name] = p.value
+  if (node.nodeType === Node.TEXT_NODE) {
+    return node.nodeValue ?? ''
   }
 
   try {
-    return createVNode(
-      h(tagName, props, ...(childNodes || []).map((cn) => parseNode(cn)))
+    const element = node as Element
+    const props: Record<string, unknown> = {}
+    for (const attr of Array.from(element.attributes ?? [])) {
+      props[attr.name] = attr.value
+    }
+    const children = Array.from(node.childNodes).map((child) =>
+      parseNode(child)
     )
+    return vnode(element.localName, props, children)
   } catch (err) {
-    return createVNode(h('div', { class: 'parseNode' }, (err as Error).stack))
+    return errorNode('parseNode', err)
   }
 }
 
 export function parseDocument(node: HTMLElement) {
   try {
-    const fragment = parse(node.outerHTML)
-    return parseNode(fragment.childNodes[0])
+    return parseNode(node)
   } catch (err) {
-    return createVNode(
-      h('div', { class: 'parseDocument' }, (err as Error).stack)
-    )
+    return errorNode('parseDocument', err)
   }
 }
 
 export function parseFragment(node: Element) {
-  // Only an Element has `outerHTML`: handed a Text or Comment child, parse5
-  // reads `length` off undefined and throws, and the catch below then serializes
-  // its own STACK TRACE into the page as a `<div class="parseFragmentWrapper">`.
-  // Text arrives as its data — the replay inserts a bare string as a text node —
-  // and a comment is dropped, matching `parseNode`'s policy for one it parses.
+  // A Text or Comment child has no attributes to read, and the replay inserts a
+  // bare string as a text node; a comment is dropped, matching `parseNode`.
   if (node?.nodeType === Node.TEXT_NODE) {
     return node.textContent || ''
   }
   if (node?.nodeType === Node.COMMENT_NODE) {
     return ''
   }
   try {
-    const fragment = parseFragmentImport(node.outerHTML)
-    return parseNode(fragment)
+    // The typeless wrapper is kept deliberately: it is what the fragment parser
+    // produced, `transform` unwraps it, and archives already written carry it.
+    return vnode(undefined, {}, [parseNode(node)])
   } catch (err) {
-    return createVNode(
-      h('div', { class: 'parseFragmentWrapper' }, (err as Error).stack)
-    )
+    return errorNode('parseFragmentWrapper', err)
   }
 }
 

packages/script/tests/bundle-size.test.ts

@@ -0,0 +1,44 @@
+import fs from 'node:fs'
+import path from 'node:path'
+import url from 'node:url'
+import { describe, it, expect } from 'vitest'
+
+const BUNDLE = path.resolve(
+  path.dirname(url.fileURLToPath(import.meta.url)),
+  '..',
+  'dist',
+  'script.js'
+)
+
+/** This bundle is registered as a BiDi preload script, and headed Chrome 154
+ *  degrades superlinearly with a preload's size: measured on a fresh session,
+ *  1 KB navigated in 4.1s, 50 KB in 8.8s, and 100 KB never completed at all
+ *  (#403). The ceiling is therefore a correctness guard rather than hygiene.
+ *  It sits far under the first measured slowdown so a dependency added here
+ *  fails on this assertion instead of on a user's stalled navigation. */
+const CEILING_BYTES = 40 * 1024
+
+describe('the injected collector bundle', () => {
+  const built = fs.existsSync(BUNDLE)
+
+  it.skipIf(!built)('stays small enough to preload without stalling', () => {
+    const bytes = fs.statSync(BUNDLE).size
+
+    expect(
+      bytes,
+      `dist/script.js is ${(bytes / 1024).toFixed(1)} KB, over the ${CEILING_BYTES / 1024} KB ceiling. ` +
+        'A preload this size slows navigation in headed Chrome and can hang it outright. ' +
+        'Check what was added to packages/script, not the ceiling.'
+    ).toBeLessThan(CEILING_BYTES)
+  })
+
+  it.skipIf(!built)('carries no HTML parser or view library', () => {
+    // The 148 KB parser was there to reparse `outerHTML` the collector could
+    // read straight off the DOM, and the view library built vnodes the app
+    // rebuilds itself. Both are the shapes this ceiling exists to keep out.
+    const source = fs.readFileSync(BUNDLE, 'utf8')
+
+    expect(source).not.toContain('parse5')
+    expect(source).not.toContain('htm/preact')
+  })
+})

packages/script/tests/serializer-edges.test.ts

@@ -0,0 +1,123 @@
+/**
+ * @vitest-environment happy-dom
+ *
+ * The cases where reading the live DOM could differ from parsing serialized
+ * markup. Each one is a place a difference would show up as degraded replay
+ * rather than as a failure, which is what makes this path worth pinning.
+ */
+import { describe, it, expect } from 'vitest'
+
+import { parseDocument, parseNode } from '../src/utils.js'
+
+const capture = (markup: string) => {
+  document.documentElement.innerHTML = markup
+  return JSON.parse(JSON.stringify(parseDocument(document.documentElement)))
+}
+
+/** Every element type appearing anywhere in a captured tree. */
+const types = (node: unknown, found: string[] = []): string[] => {
+  if (!node || typeof node !== 'object') {
+    return found
+  }
+  const n = node as { type?: string; props?: { children?: unknown } }
+  if (n.type) {
+    found.push(n.type)
+  }
+  const kids = n.props?.children
+  for (const child of Array.isArray(kids) ? kids : kids ? [kids] : []) {
+    types(child, found)
+  }
+  return found
+}
+
+const find = (node: unknown, type: string): any =>
+  JSON.parse(JSON.stringify(node)) &&
+  (function walk(n: any): any {
+    if (!n || typeof n !== 'object') {
+      return undefined
+    }
+    if (n.type === type) {
+      return n
+    }
+    const kids = n.props?.children
+    for (const c of Array.isArray(kids) ? kids : kids ? [kids] : []) {
+      const hit = walk(c)
+      if (hit) {
+        return hit
+      }
+    }
+    return undefined
+  })(node)
+
+describe('serializing the live DOM', () => {
+  it('keeps the case foreign elements need', () => {
+    // An HTML parser lowercases tag names; SVG's are case-sensitive, so
+    // `linearGradient` becoming `lineargradient` renders nothing.
+    const tree = capture('<body><svg><linearGradient id="g"/></svg></body>')
+
+    expect(types(tree)).toContain('linearGradient')
+  })
+
+  it('keeps the case foreign attributes need', () => {
+    const tree = capture('<body><svg viewBox="0 0 10 10"></svg></body>')
+
+    expect(find(tree, 'svg').props).toHaveProperty('viewBox')
+  })
+
+  it('decodes entities rather than carrying their source text', () => {
+    const tree = capture('<body><p title="a&amp;b">x &lt; y</p></body>')
+    const p = find(tree, 'p')
+
+    expect(p.props.title).toBe('a&b')
+    expect(p.props.children).toBe('x < y')
+  })
+
+  it('reads the tree the browser built, not the markup as written', () => {
+    // An unclosed <p> before a <div> is reparented by the HTML parser. The DOM
+    // already reflects that; capture must not re-derive it differently.
+    const tree = capture('<body><p>one<div>two</div></body>')
+
+    expect(types(tree)).toEqual(expect.arrayContaining(['p', 'div']))
+  })
+
+  it('drops comments without leaving their text behind', () => {
+    const tree = capture('<body><div><!-- build stamp -->kept</div></body>')
+
+    expect(JSON.stringify(tree)).not.toContain('build stamp')
+    expect(JSON.stringify(tree)).toContain('kept')
+  })
+
+  it('serializes a template as empty, since its content is not a child', () => {
+    // `<template>` holds its content in a separate DocumentFragment. Neither
+    // this nor the parser it replaced descends into it; pinned so a future
+    // change to template handling is a deliberate one.
+    const tree = capture('<body><template><p>hidden</p></template></body>')
+
+    expect(JSON.stringify(tree)).not.toContain('hidden')
+  })
+
+  it('carries every attribute of an element, not a curated set', () => {
+    const tree = capture(
+      '<body><input id="a" class="b" data-x="c" disabled aria-label="d"></body>'
+    )
+
+    expect(find(tree, 'input').props).toMatchObject({
+      id: 'a',
+      class: 'b',
+      'data-x': 'c',
+      disabled: '',
+      'aria-label': 'd'
+    })
+  })
+
+  it('returns a bare string for a text node and nothing for a comment', () => {
+    expect(parseNode(document.createTextNode('hello'))).toBe('hello')
+    expect(parseNode(document.createComment('gone'))).toBe('')
+  })
+
+  it('survives a node with no attributes to read', () => {
+    const fragment = document.createDocumentFragment()
+
+    expect(() => parseNode(fragment)).not.toThrow()
+  })
+})

packages/selenium-devtools/package.json

@@ -69,7 +69,6 @@
     "@wdio/devtools-shared": "workspace:^",
     "allure-js-commons": "^3.0.0",
     "allure-mocha": "^3.0.0",
-    "chromedriver": "^151.0.5",
     "jest": "^30.4.2",
     "mocha": "^11.7.6",
     "selenium-webdriver": "^4.44.0",