libjwt: JWT algorithm-confusion authentication bypass via RSA JWK without alg

# Root Cause Analysis: CVE-2026-44699 / GHSA-q843-6q5f-w55g

## Summary

`libjwt` versions 3.0.0 through 3.3.2 suffer from an algorithm confusion attack that allows JWT forgery using only a public RSA JWK. When an RSA JWK lacking an `alg` parameter is supplied as the verification key for an HMAC-signed token (`HS256`/`HS384`/`HS512`), the library incorrectly accepts the RSA key for HMAC verification. Because the internal `jwk_item_t` union shares storage between `provider_data` (an RSA `EVP_PKEY*`) and `oct.{key,len}` (HMAC bytes/length), the HMAC path reads a zero-length key and produces a successful verification for any token signed as `HMAC("", header.payload)`. An attacker who knows only the published JWKS can forge valid tokens and bypass authentication.

## Impact

- **Package**: `libjwt` (C library for JWT processing)
- **Affected versions**: 3.0.0 through 3.3.2 (inclusive)
- **Fixed version**: 3.3.3
- **Risk level**: Critical (CVSS 3.1 base 9.1)
- **Consequences**: Full authentication bypass. Any attacker with access to the public JWKS (which is public by definition) can forge HMAC-signed JWTs that the vulnerable library accepts as valid.

## Root Cause

The root cause is a failure to validate that a JWK's actual key type (`kty`) is compatible with the JWT algorithm being used for verification. The library previously relied on the optional `alg` parameter in the JWK to determine compatibility. When `alg` was absent (`JWT_ALG_NONE`), `__setkey_check()` in `jwt-common.c` allowed the key to be used with any caller-specified algorithm.

Because `jwk_item_t` is implemented as a union, setting an RSA key populates `provider_data` while leaving the HMAC `oct` fields uninitialized (effectively zero-length). When the HMAC verification backend later runs, it reads `oct.key` and `oct.len`, producing an HMAC with an empty key. Since `HMAC("", data)` is deterministic, an attacker can pre-compute the signature and forge a valid token.

### Fix commit

- Commit `49c730a` in the libjwt repository: "Fix algorithm confusion that allows JWT forgery via RSA JWK as HMAC key"
- The fix introduces `jwt_alg_required_kty()` which maps each JWA algorithm to its required JWK key type (e.g., `HS256` requires `kty=oct`).
- `__setkey_check()` now rejects any setkey/verify callback that pairs a non-none algorithm with an incompatible `kty`.
- `__verify_config_post()` adds a defensive re-check once `jwt->alg` is bound from the token header.
- `__check_hmac()` adds a final backstop refusing any non-oct key for HMAC algorithms.

## Reproduction Steps

1. Execute `repro/reproduction_steps.sh`
2. The script builds two versions of `libjwt`:
   - `v3.3.2` (vulnerable)
   - `v3.3.3` (fixed)
3. It compiles a small C harness against each version.
4. The harness:
   - parses an RSA public JWK **without** an `alg` field,
   - calls `jwt_checker_setkey(checker, JWT_ALG_HS256, rsa_jwk)`,
   - verifies a forged `HS256` token whose signature is `HMAC-SHA256("", header.payload)`.
5. On the vulnerable build, the forged token verifies successfully (harness exits `0`).
6. On the fixed build, `setkey` is rejected immediately with "Key type does not match algorithm" (harness exits `2`).

## Evidence

### Log locations
- `logs/vuln_out.txt` — output from harness linked against `v3.3.2`
- `logs/fix_out.txt` — output from harness linked against `v3.3.3`
- `logs/results.txt` — combined summary
- `repro/validation_verdict.json` — structured verdict

### Key excerpts

**Vulnerable build (v3.3.2)**
```
Exit code: 0
VERIFY SUCCEEDED (VULNERABLE)
```

**Fixed build (v3.3.3)**
```
Exit code: 2
setkey rejected: Key type does not match algorithm
```

These outputs confirm the bug and its remediation: the vulnerable library silently accepts an RSA JWK for HMAC verification and validates the forged token, while the fixed library rejects the key/algorithm mismatch at `setkey` time.

### Environment
- OS: Ubuntu Noble (24.04)
- Compiler: GCC 13.3.0
- CMake: 3.28.3
- OpenSSL: 3.0.13
- Jansson: 2.14
- libjwt built from source at `external/libjwt`

## Recommendations / Next Steps

1. **Upgrade immediately** to `libjwt >= 3.3.3`.
2. **Review JWKS handling** in applications that use `libjwt`. Ensure that verification callbacks do not blindly trust the `alg` field from an attacker-controlled JWT header without also checking the JWK's `kty`.
3. **Regression tests**: The fix commit includes a comprehensive test suite in `tests/jwt_security.c` covering RSA, EC, and OKP keys mis-targeted at HMAC algorithms, as well as malformed JWKs where `alg` disagrees with `kty`. Maintainers should ensure these tests are run in CI.
4. **Defense in depth**: Even after upgrading, applications should enforce algorithm whitelisting (e.g., only accept `RS256` if the JWKS contains RSA keys) at the application layer to reduce the impact of any future confusion bugs.

## Additional Notes

- **Idempotency**: The reproduction script was run twice consecutively with identical results, confirming the build and test process is deterministic.
- **Edge cases**: The advisory notes the bug specifically requires an RSA JWK *without* an `alg` parameter. An RSA JWK that explicitly declares `"alg": "RS256"` would also be rejected for HMAC after the fix, because the fix binds algorithm acceptance to the JWK's `kty`, not its optional `alg` hint.
- **Limitations**: The reproduction script focuses on `HS256` with a 2048-bit RSA JWK. The fix commit's tests confirm the same issue exists for `HS384` and `HS512`, as well as for EC and OKP keys misused as HMAC keys.