mirror of
https://github.com/mitch030504/Wiicompiled_VR_Frame.git
synced 2026-10-06 01:00:14 +02:00
Introduced translator_host C program to start the WiiCompiled translator on the headset, managing heap pointer tagging and dynamic library loading.
This commit is contained in:
1 parent
2f85207c39
commit
0bf2675db1
19 files changed
+1796
-60
No files matched your search
@@ -154,6 +154,28 @@ Unicode-3.0, BSL-1.0 or Unlicense licensed.
|
||||
| bzip2 (via bzip2-sys) | 1.0.8 | bzip2 license (BSD-style) | <https://sourceware.org/bzip2/> |
|
||||
| jni-rs | 0.21.1 | MIT OR Apache-2.0 | <https://github.com/jni-rs/jni-rs> |
|
||||
|
||||
For building the game on the headset, the APK also carries a toolchain
|
||||
(`android/Prepare-QuestToolchain.ps1`): the translator published with the .NET 10 runtime for
|
||||
`linux-bionic-arm64`, and Termux's Android builds of clang and lld with the libraries they load,
|
||||
redistributed unmodified from the Termux package repository (pins and license names in the script
|
||||
and in the toolchain's `llvm/licenses/`).
|
||||
|
||||
| Component (Quest APK toolchain) | Version | License | Upstream |
|
||||
| --- | --- | --- | --- |
|
||||
| .NET runtime (Mono, linux-bionic-arm64) | 10.0 | MIT | <https://github.com/dotnet/runtime> |
|
||||
| YamlDotNet | 15.1.2 | MIT | <https://github.com/aaubry/YamlDotNet> |
|
||||
| LLVM clang, lld, libLLVM, libc++ (Termux clang/lld/libllvm 21.1.8-3, libc++ 29) | 21.1.8 | Apache-2.0 WITH LLVM-exception | <https://github.com/termux/termux-packages> |
|
||||
| libffi | 3.8.0 | MIT | <https://github.com/libffi/libffi> |
|
||||
| libxml2 | 2.15.4 | MIT | <https://gitlab.gnome.org/GNOME/libxml2> |
|
||||
| GNU libiconv | 1.19 | LGPL-2.1-or-later | <https://www.gnu.org/software/libiconv/> |
|
||||
| zlib | 1.3.2 | Zlib | <https://zlib.net/> |
|
||||
| Zstandard | 1.5.7 | BSD-3-Clause | <https://github.com/facebook/zstd> |
|
||||
| OpenSSL | 3.6.3 | Apache-2.0 | <https://www.openssl.org/> |
|
||||
|
||||
libiconv is LGPL-licensed and ships as the unmodified shared library `libiconv.so`, which can be
|
||||
replaced; its complete corresponding source is available from the Termux package repository and
|
||||
from GNU, and this project will supply it on request for the version shipped.
|
||||
|
||||
### Dual-licensed components - elections made by this project
|
||||
|
||||
- **FreeType** is offered under the FreeType License (FTL) or GPL-2.0. **This project elects the
|
||||
@@ -192,6 +214,7 @@ unmodified, with their license texts, in the installer's `licenses/` folder.
|
||||
| Component | License | Upstream |
|
||||
| --- | --- | --- |
|
||||
| Android NDK r29 for Windows (clang, lld, sysroot), fetched by `--build-quest` from Google with a pinned SHA-1 | Android Software Development Kit License Agreement | <https://developer.android.com/studio/terms> |
|
||||
| Android NDK r29 aarch64 sysroot, compiler-rt builtins, libunwind and libatomic, fetched by the Quest app's Build on this Quest from Google's Linux NDK zip with pinned SHA-256s | Android Software Development Kit License Agreement | <https://developer.android.com/studio/terms> |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -94,14 +94,16 @@ if ($Flavor -eq 'base') {
|
||||
try {
|
||||
$gameLibraries = @($archive.Entries | Where-Object { $_.FullName -match '^lib/[^/]+/libmain[^/]*\.so$' })
|
||||
$hasKit = @($archive.Entries | Where-Object { $_.FullName -eq 'assets/game_kit/kit.json' }).Count -gt 0
|
||||
$hasToolchain = @($archive.Entries | Where-Object { $_.FullName -eq 'assets/quest_toolchain/files.zip' }).Count -gt 0
|
||||
} finally { $archive.Dispose() }
|
||||
if ($gameLibraries.Count -gt 0) { throw "The APK contains a translated game library: $($gameLibraries.FullName -join ', ')" }
|
||||
if (-not $hasKit) { throw 'The APK has no game kit (assets/game_kit/kit.json)' }
|
||||
if (-not $hasToolchain) { throw 'The APK has no build toolchain (assets/quest_toolchain/files.zip)' }
|
||||
}
|
||||
|
||||
if ($Install) {
|
||||
$adb = Join-Path $sdkRoot 'platform-tools\adb.exe'
|
||||
& $adb install -r $apk.FullName
|
||||
if ($LASTEXITCODE -ne 0) { throw "adb install failed ($LASTEXITCODE)" }
|
||||
Write-Host 'Installed. Build and push the game with android/Build-QuestGame.ps1 -Install.'
|
||||
Write-Host 'Installed. Build the game with Build on this Quest in the launcher, or on this PC with android/Build-QuestGame.ps1 -Install.'
|
||||
}
|
||||
@@ -0,0 +1,341 @@
|
||||
# Builds the toolchain the Quest app carries so a player can build the game on the headset
|
||||
# ("Build on this Quest"), without a PC:
|
||||
#
|
||||
# * translator/ the WiiCompiled translator published for linux-bionic-arm64 (.NET 10, which runs
|
||||
# it on Mono), started through translator_host (android/toolchain/translator_host.c);
|
||||
# * llvm/ Termux's Android build of clang and lld 21.1.8 with exactly the shared libraries
|
||||
# they load, OpenSSL for the translator's hashing (Android only has BoringSSL), and
|
||||
# clang's own builtin headers;
|
||||
# * ndk.json the Android NDK files a build also needs (the aarch64 sysroot, libunwind, libatomic,
|
||||
# compiler-rt builtins). The NDK is Google's to distribute, so the headset downloads
|
||||
# them from dl.google.com with HTTP range requests (about 8 MB of the 784 MB zip) and
|
||||
# checks each file against the SHA-256 recorded here;
|
||||
# * toolchain.json every file of translator/ and llvm/ with its SHA-256, and the fingerprint over
|
||||
# them. Those two directories travel packed in files.zip.
|
||||
#
|
||||
# powershell -ExecutionPolicy Bypass -File android/Prepare-QuestToolchain.ps1 -OutputDir <dir> -NdkLlvm <ndk>\toolchains\llvm\prebuilt\windows-x86_64
|
||||
#
|
||||
# The Gradle build runs it (prepareQuestToolchain). Downloads are cached in android/.dependencies.
|
||||
# Needs the .NET 10 SDK: the linux-bionic runtime packs only exist for .NET 10.
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory)] [string]$OutputDir,
|
||||
[Parameter(Mandatory)] [string]$NdkLlvm,
|
||||
[string]$CacheDir = '',
|
||||
[string]$Dotnet = 'dotnet'
|
||||
)
|
||||
$ErrorActionPreference = 'Stop'
|
||||
Set-StrictMode -Version Latest
|
||||
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
|
||||
Add-Type -AssemblyName System.IO.Compression, System.IO.Compression.FileSystem
|
||||
# Windows PowerShell leaves $PSScriptRoot empty while binding parameter defaults.
|
||||
if (-not $CacheDir) { $CacheDir = Join-Path $PSScriptRoot '.dependencies\quest-toolchain' }
|
||||
|
||||
$repo = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path
|
||||
$termuxRoot = 'https://packages.termux.dev/apt/termux-main/'
|
||||
$termuxPrefix = './data/data/com.termux/files/usr/'
|
||||
|
||||
# Termux packages (aarch64), pinned by the SHA-256 of their .deb. Files maps a path inside the
|
||||
# package to its name in llvm/: the libraries are stored under the SONAME their users load.
|
||||
$termuxPackages = @(
|
||||
@{ Name = 'clang'; File = 'pool/main/c/clang/clang_21.1.8-3_aarch64.deb'; Sha256 = '9fb565013b96ce10abd8f8f217aaebb48f72873f53acbf8886329bcea841a21c'
|
||||
License = 'Apache-2.0 WITH LLVM-exception'
|
||||
Files = @{ 'bin/clang-21' = 'bin/clang-21'; 'lib/libclang-cpp.so' = 'lib/libclang-cpp.so'; 'lib/clang/21/include/' = 'lib/clang/21/include/' } }
|
||||
@{ Name = 'lld'; File = 'pool/main/l/lld/lld_21.1.8-3_aarch64.deb'; Sha256 = '6299c8fdff59dcff972f6b444805bcdb211c2bd797a64d2ddb3875e699f30630'
|
||||
License = 'Apache-2.0 WITH LLVM-exception'
|
||||
Files = @{ 'bin/lld' = 'bin/ld.lld' } }
|
||||
@{ Name = 'libllvm'; File = 'pool/main/libl/libllvm/libllvm_21.1.8-3_aarch64.deb'; Sha256 = '5056a73f2645fc9c0759e6f94a1bd6edcf36305acaef58dc685211ced9cc2a02'
|
||||
License = 'Apache-2.0 WITH LLVM-exception'
|
||||
Files = @{ 'lib/libLLVM.so' = 'lib/libLLVM.so'; 'share/doc/libllvm/LICENSE.TXT' = 'licenses/llvm/LICENSE.TXT' } }
|
||||
@{ Name = 'libc++'; File = 'pool/main/libc/libc++/libc++_29_aarch64.deb'; Sha256 = 'bb9f12113c137aa0e8513bb51cc49fe77a5ce3ca39ab9e92c57d228ecdf00222'
|
||||
License = 'Apache-2.0 WITH LLVM-exception'
|
||||
Files = @{ 'lib/libc++_shared.so' = 'lib/libc++_shared.so' } }
|
||||
@{ Name = 'libffi'; File = 'pool/main/libf/libffi/libffi_3.8.0_aarch64.deb'; Sha256 = '4f255badf74cd31f6a2801c17fa1444199c84c834b517b7c843e2fe9ebe91d77'
|
||||
License = 'MIT'
|
||||
Files = @{ 'lib/libffi.so' = 'lib/libffi.so'; 'share/doc/libffi/copyright' = 'licenses/libffi/copyright' } }
|
||||
@{ Name = 'libxml2'; File = 'pool/main/libx/libxml2/libxml2_2.15.4-1_aarch64.deb'; Sha256 = 'd315861bbcf716cfb088f53cddab9fb6a96430cc66769f79774cc69c2f53bcff'
|
||||
License = 'MIT'
|
||||
Files = @{ 'lib/libxml2.so.16.1.4' = 'lib/libxml2.so.16' } }
|
||||
@{ Name = 'libiconv'; File = 'pool/main/libi/libiconv/libiconv_1.19_aarch64.deb'; Sha256 = 'fe9481b1dc101c6c3552943f25435109fd522aecc615ae49594f9fbee863bb37'
|
||||
License = 'LGPL-2.1-or-later'
|
||||
Files = @{ 'lib/libiconv.so' = 'lib/libiconv.so' } }
|
||||
@{ Name = 'zlib'; File = 'pool/main/z/zlib/zlib_1.3.2_aarch64.deb'; Sha256 = '75e7d0af17fcc3b40004309fdc00a1ddb9ae08346dce5e269902c34ac3966ac9'
|
||||
License = 'Zlib'
|
||||
Files = @{ 'lib/libz.so.1.3.2' = 'lib/libz.so.1' } }
|
||||
@{ Name = 'zstd'; File = 'pool/main/z/zstd/zstd_1.5.7-1_aarch64.deb'; Sha256 = 'e1b4a5113648da8de189620ba1fce74c48b2d0833d9043391b9a1c91fb606fd3'
|
||||
License = 'BSD-3-Clause'
|
||||
Files = @{ 'lib/libzstd.so.1.5.7' = 'lib/libzstd.so.1' } }
|
||||
@{ Name = 'openssl'; File = 'pool/main/o/openssl/openssl_1:3.6.3_aarch64.deb'; Sha256 = '86760e9ce736f463236f2c15b1eb3a3fdcfc5778d0fd7077a917448dcc90f3aa'
|
||||
License = 'Apache-2.0'
|
||||
# .NET's OpenSSL shim on Android loads "libssl.so" by that name; libssl needs libcrypto.so.3.
|
||||
Files = @{ 'lib/libssl.so.3' = 'lib/libssl.so'; 'lib/libcrypto.so.3' = 'lib/libcrypto.so.3' } }
|
||||
)
|
||||
|
||||
# The NDK files the headset downloads: the r29 Linux zip, pinned by size and by the digest of the
|
||||
# selected files (Get-ListDigest over them). The Linux zip is used because Windows file systems
|
||||
# cannot hold its sysroot headers whose names differ only in case.
|
||||
$ndkUrl = 'https://dl.google.com/android/repository/android-ndk-r29-linux.zip'
|
||||
$ndkSize = 783549481
|
||||
$ndkPrefix = 'android-ndk-r29/toolchains/llvm/prebuilt/linux-x86_64/'
|
||||
$ndkSubsetDigest = '4221ed656e864cb3434d922e8e9eb916b08833718380847d0923527394041d0f'
|
||||
|
||||
function Get-Sha256Hex([string]$Path) {
|
||||
$sha = [Security.Cryptography.SHA256]::Create()
|
||||
try {
|
||||
$stream = [IO.File]::OpenRead($Path)
|
||||
try { $bytes = $sha.ComputeHash($stream) } finally { $stream.Dispose() }
|
||||
} finally { $sha.Dispose() }
|
||||
return (($bytes | ForEach-Object { $_.ToString('x2') }) -join '')
|
||||
}
|
||||
|
||||
function Get-BytesSha256Hex([byte[]]$Bytes) {
|
||||
$sha = [Security.Cryptography.SHA256]::Create()
|
||||
try { $hash = $sha.ComputeHash($Bytes) } finally { $sha.Dispose() }
|
||||
return (($hash | ForEach-Object { $_.ToString('x2') }) -join '')
|
||||
}
|
||||
|
||||
# SHA-256 over "<path> <sha256>" lines sorted by path, joined with \n. The headset computes the
|
||||
# same over the files it downloaded.
|
||||
function Get-ListDigest([object[]]$Files) {
|
||||
[string[]]$lines = @($Files | ForEach-Object { "$($_.path) $($_.sha256)" })
|
||||
[Array]::Sort($lines, [StringComparer]::Ordinal)
|
||||
return Get-BytesSha256Hex ([Text.Encoding]::UTF8.GetBytes(($lines -join "`n")))
|
||||
}
|
||||
|
||||
function Invoke-Tar([string[]]$Arguments) {
|
||||
& (Join-Path $env:SystemRoot 'System32\tar.exe') @Arguments
|
||||
if ($LASTEXITCODE -ne 0) { throw "tar $($Arguments -join ' ') failed ($LASTEXITCODE)" }
|
||||
}
|
||||
|
||||
function Save-TermuxPackage($Package) {
|
||||
$directory = Join-Path $CacheDir 'termux'
|
||||
New-Item -ItemType Directory -Force $directory | Out-Null
|
||||
# Termux file names can contain ':', which Windows does not allow.
|
||||
$local = Join-Path $directory ($Package.File.Split('/')[-1].Replace(':', '_'))
|
||||
if (-not (Test-Path $local) -or (Get-Sha256Hex $local) -ne $Package.Sha256) {
|
||||
$url = $termuxRoot + $Package.File.Replace(':', '%3A')
|
||||
Write-Host "Downloading $url"
|
||||
Invoke-WebRequest -UseBasicParsing -Uri $url -OutFile "$local.partial"
|
||||
$actual = Get-Sha256Hex "$local.partial"
|
||||
if ($actual -ne $Package.Sha256) {
|
||||
Remove-Item "$local.partial"
|
||||
throw "$($Package.Name) package digest mismatch: $actual"
|
||||
}
|
||||
Move-Item -Force "$local.partial" $local
|
||||
}
|
||||
return $local
|
||||
}
|
||||
|
||||
function Expand-TermuxPackage($Package, [string]$Destination, [string]$Work) {
|
||||
$deb = Save-TermuxPackage $Package
|
||||
$unpack = Join-Path $Work $Package.Name
|
||||
if (Test-Path $unpack) { Remove-Item -Recurse -Force $unpack }
|
||||
New-Item -ItemType Directory -Force $unpack | Out-Null
|
||||
Invoke-Tar @('-xf', $deb, '-C', $unpack)
|
||||
$data = Get-ChildItem $unpack -Filter 'data.tar.*' | Select-Object -First 1
|
||||
if (-not $data) { throw "$($Package.Name) has no data archive" }
|
||||
# Only the listed regular files: the packages also hold symlinks, which Windows cannot create.
|
||||
$members = @($Package.Files.Keys | ForEach-Object { $termuxPrefix + $_ })
|
||||
Invoke-Tar (@('-xf', $data.FullName, '-C', $unpack) + $members)
|
||||
foreach ($source in $Package.Files.Keys) {
|
||||
$from = Join-Path $unpack ($termuxPrefix.Substring(2) + $source)
|
||||
$to = Join-Path $Destination $Package.Files[$source]
|
||||
if ($source.EndsWith('/')) {
|
||||
New-Item -ItemType Directory -Force $to | Out-Null
|
||||
Copy-Item -Recurse -Force (Join-Path $from '*') $to
|
||||
} else {
|
||||
if (-not (Test-Path $from -PathType Leaf)) { throw "$($Package.Name) has no regular file $source" }
|
||||
New-Item -ItemType Directory -Force (Split-Path -Parent $to) | Out-Null
|
||||
Copy-Item -Force $from $to
|
||||
}
|
||||
}
|
||||
Remove-Item -Recurse -Force $unpack
|
||||
}
|
||||
|
||||
function Read-RemoteRange([string]$Url, [long]$Start, [long]$End) {
|
||||
$request = [Net.HttpWebRequest]::Create($Url)
|
||||
$request.AddRange($Start, $End)
|
||||
$response = $request.GetResponse()
|
||||
try {
|
||||
if ([int]$response.StatusCode -ne 206) { throw "$Url ignored the range request" }
|
||||
$stream = $response.GetResponseStream()
|
||||
$buffer = New-Object byte[] ([int]($End - $Start + 1))
|
||||
$read = 0
|
||||
while ($read -lt $buffer.Length) {
|
||||
$count = $stream.Read($buffer, $read, $buffer.Length - $read)
|
||||
if ($count -le 0) { throw "Short read from $Url" }
|
||||
$read += $count
|
||||
}
|
||||
return , $buffer
|
||||
} finally { $response.Close() }
|
||||
}
|
||||
|
||||
# Whether a file below the NDK's prebuilt directory belongs to what the headset downloads.
|
||||
function Test-NdkSubsetFile([string]$Path) {
|
||||
if ($Path.StartsWith('sysroot/usr/include/')) {
|
||||
return $Path -notmatch '^sysroot/usr/include/(arm-linux-androideabi|i686-linux-android|x86_64-linux-android|riscv64-linux-android)/'
|
||||
}
|
||||
if ($Path.StartsWith('sysroot/usr/lib/aarch64-linux-android/29/')) { return $true }
|
||||
if ($Path -in 'sysroot/usr/lib/aarch64-linux-android/libc++_shared.so', 'sysroot/usr/lib/aarch64-linux-android/libc++.so') { return $true }
|
||||
return $Path -match '^lib/clang/[^/]+/lib/linux/(libclang_rt\.builtins-aarch64-android\.a|aarch64/(libunwind|libatomic)\.a)$'
|
||||
}
|
||||
|
||||
# Lists the NDK subset with each file's SHA-256, reading only the zip's central directory and the
|
||||
# selected entries over HTTP. Cached, since the pinned zip never changes.
|
||||
function Get-NdkSubset {
|
||||
$cached = Join-Path $CacheDir 'ndk-r29-subset.json'
|
||||
if (Test-Path $cached) { return [IO.File]::ReadAllText($cached) | ConvertFrom-Json }
|
||||
|
||||
$tail = Read-RemoteRange $ndkUrl ($ndkSize - 65557) ($ndkSize - 1)
|
||||
$eocd = -1
|
||||
for ($i = $tail.Length - 22; $i -ge 0; $i--) {
|
||||
if ([BitConverter]::ToUInt32($tail, $i) -eq 0x06054b50) { $eocd = $i; break }
|
||||
}
|
||||
if ($eocd -lt 0) { throw 'No end of central directory in the NDK zip' }
|
||||
$entryCount = [BitConverter]::ToUInt16($tail, $eocd + 10)
|
||||
$directorySize = [BitConverter]::ToUInt32($tail, $eocd + 12)
|
||||
$directoryOffset = [BitConverter]::ToUInt32($tail, $eocd + 16)
|
||||
if ($entryCount -eq 0xFFFF -or $directoryOffset -eq 0xFFFFFFFF) { throw 'The NDK zip needs ZIP64, which this reader does not support' }
|
||||
$directory = Read-RemoteRange $ndkUrl $directoryOffset ($directoryOffset + $directorySize - 1)
|
||||
|
||||
$entries = New-Object System.Collections.Generic.List[object]
|
||||
$at = 0
|
||||
for ($n = 0; $n -lt $entryCount; $n++) {
|
||||
if ([BitConverter]::ToUInt32($directory, $at) -ne 0x02014b50) { throw 'Corrupt NDK zip central directory' }
|
||||
$nameLength = [BitConverter]::ToUInt16($directory, $at + 28)
|
||||
$extraLength = [BitConverter]::ToUInt16($directory, $at + 30)
|
||||
$commentLength = [BitConverter]::ToUInt16($directory, $at + 32)
|
||||
$name = [Text.Encoding]::UTF8.GetString($directory, $at + 46, $nameLength)
|
||||
$entries.Add([pscustomobject]@{
|
||||
name = $name
|
||||
method = [BitConverter]::ToUInt16($directory, $at + 10)
|
||||
compressedSize = [long][BitConverter]::ToUInt32($directory, $at + 20)
|
||||
size = [long][BitConverter]::ToUInt32($directory, $at + 24)
|
||||
offset = [long][BitConverter]::ToUInt32($directory, $at + 42)
|
||||
})
|
||||
$at += 46 + $nameLength + $extraLength + $commentLength
|
||||
}
|
||||
$sorted = @($entries | Sort-Object offset)
|
||||
$selected = New-Object System.Collections.Generic.List[object]
|
||||
for ($i = 0; $i -lt $sorted.Count; $i++) {
|
||||
$entry = $sorted[$i]
|
||||
if (-not $entry.name.StartsWith($ndkPrefix) -or $entry.name.EndsWith('/')) { continue }
|
||||
$path = $entry.name.Substring($ndkPrefix.Length)
|
||||
if (-not (Test-NdkSubsetFile $path)) { continue }
|
||||
$end = if ($i + 1 -lt $sorted.Count) { $sorted[$i + 1].offset } else { $directoryOffset }
|
||||
$selected.Add([pscustomobject]@{ path = $path; entry = $entry; end = $end })
|
||||
}
|
||||
|
||||
# Neighbouring entries are fetched together.
|
||||
$files = New-Object System.Collections.Generic.List[object]
|
||||
$i = 0
|
||||
while ($i -lt $selected.Count) {
|
||||
$j = $i
|
||||
while ($j + 1 -lt $selected.Count -and $selected[$j + 1].entry.offset - $selected[$j].end -lt 1MB) { $j++ }
|
||||
$chunkStart = $selected[$i].entry.offset
|
||||
$chunk = Read-RemoteRange $ndkUrl $chunkStart ($selected[$j].end - 1)
|
||||
for ($k = $i; $k -le $j; $k++) {
|
||||
$item = $selected[$k]
|
||||
$local = [int]($item.entry.offset - $chunkStart)
|
||||
if ([BitConverter]::ToUInt32($chunk, $local) -ne 0x04034b50) { throw "Corrupt NDK zip entry $($item.path)" }
|
||||
$dataStart = $local + 30 + [BitConverter]::ToUInt16($chunk, $local + 26) + [BitConverter]::ToUInt16($chunk, $local + 28)
|
||||
$compressed = New-Object IO.MemoryStream($chunk, $dataStart, [int]$item.entry.compressedSize)
|
||||
$content = New-Object IO.MemoryStream
|
||||
switch ($item.entry.method) {
|
||||
0 { $compressed.CopyTo($content) }
|
||||
8 {
|
||||
$inflate = New-Object IO.Compression.DeflateStream($compressed, [IO.Compression.CompressionMode]::Decompress)
|
||||
try { $inflate.CopyTo($content) } finally { $inflate.Dispose() }
|
||||
}
|
||||
default { throw "NDK zip entry $($item.path) uses compression method $($item.entry.method)" }
|
||||
}
|
||||
$bytes = $content.ToArray()
|
||||
if ($bytes.Length -ne $item.entry.size) { throw "NDK zip entry $($item.path) inflated to the wrong size" }
|
||||
$files.Add([pscustomobject]@{ path = $item.path; size = $item.entry.size; sha256 = Get-BytesSha256Hex $bytes })
|
||||
}
|
||||
$i = $j + 1
|
||||
}
|
||||
$subset = [pscustomobject]@{ url = $ndkUrl; size = $ndkSize; prefix = $ndkPrefix; digest = Get-ListDigest $files.ToArray(); files = $files.ToArray() }
|
||||
New-Item -ItemType Directory -Force $CacheDir | Out-Null
|
||||
[IO.File]::WriteAllText($cached, ($subset | ConvertTo-Json -Depth 4 -Compress), (New-Object Text.UTF8Encoding $false))
|
||||
return $subset
|
||||
}
|
||||
|
||||
function Publish-Translator([string]$Destination, [string]$Work) {
|
||||
$sdks = (& $Dotnet --list-sdks) -join "`n"
|
||||
if ($sdks -notmatch '(?m)^10\.') { throw 'Publishing the translator for Android needs the .NET 10 SDK.' }
|
||||
# A staged copy retargeted to net10.0, so the checkout's own net8.0 build outputs stay untouched.
|
||||
$stage = Join-Path $Work 'translator-stage'
|
||||
if (Test-Path $stage) { Remove-Item -Recurse -Force $stage }
|
||||
foreach ($tree in 'translator\src', 'Launcher\WiiCompiled.Setup.Common') {
|
||||
& (Join-Path $env:SystemRoot 'System32\robocopy.exe') (Join-Path $repo $tree) (Join-Path $stage $tree) /E /XD bin obj /NFL /NDL /NJH /NJS /NP | Out-Null
|
||||
if ($LASTEXITCODE -ge 8) { throw "Staging $tree failed (robocopy $LASTEXITCODE)" }
|
||||
}
|
||||
foreach ($project in Get-ChildItem -Recurse -Filter *.csproj $stage) {
|
||||
$text = [IO.File]::ReadAllText($project.FullName)
|
||||
$retargeted = $text.Replace('<TargetFramework>net8.0</TargetFramework>', '<TargetFramework>net10.0</TargetFramework>')
|
||||
if ($retargeted -eq $text) { throw "$($project.Name) does not target net8.0" }
|
||||
[IO.File]::WriteAllText($project.FullName, $retargeted)
|
||||
}
|
||||
if (Test-Path $Destination) { Remove-Item -Recurse -Force $Destination }
|
||||
& $Dotnet publish (Join-Path $stage 'translator\src\Translator.Cli\Translator.Cli.csproj') -c Release `
|
||||
-r linux-bionic-arm64 --self-contained true -p:PublishSingleFile=false -p:InvariantGlobalization=true `
|
||||
-p:DebugType=none -o $Destination | Out-Host
|
||||
if ($LASTEXITCODE -ne 0) { throw "Publishing the translator failed ($LASTEXITCODE)" }
|
||||
Get-ChildItem -Recurse -Filter *.pdb $Destination | Remove-Item
|
||||
if (-not (Test-Path (Join-Path $Destination 'libhostfxr.so'))) { throw 'The translator was not published self-contained' }
|
||||
}
|
||||
|
||||
$work = Join-Path $CacheDir 'work'
|
||||
New-Item -ItemType Directory -Force $work | Out-Null
|
||||
# The tree is staged in the cache and packed into one deflated files.zip: Android's asset packaging
|
||||
# never compresses .so files, and libLLVM alone is 134 MB.
|
||||
$stage = Join-Path $work 'toolchain'
|
||||
if (Test-Path $stage) { Remove-Item -Recurse -Force $stage }
|
||||
New-Item -ItemType Directory -Force $stage | Out-Null
|
||||
if (Test-Path $OutputDir) { Remove-Item -Recurse -Force $OutputDir }
|
||||
New-Item -ItemType Directory -Force $OutputDir | Out-Null
|
||||
|
||||
$llvm = Join-Path $stage 'llvm'
|
||||
foreach ($package in $termuxPackages) { Expand-TermuxPackage $package $llvm $work }
|
||||
$licenses = @($termuxPackages | ForEach-Object { "$($_.Name) $($_.File.Split('/')[-1].Replace('.deb', '')) - $($_.License) - https://github.com/termux/termux-packages" })
|
||||
New-Item -ItemType Directory -Force (Join-Path $llvm 'licenses') | Out-Null
|
||||
[IO.File]::WriteAllText((Join-Path $llvm 'licenses\PACKAGES.txt'), (($licenses -join "`n") + "`n"))
|
||||
|
||||
$translator = Join-Path $stage 'translator'
|
||||
Publish-Translator $translator $work
|
||||
# Static archives for embedding the runtime in a native app; the translator runs on the shared one.
|
||||
Get-ChildItem -Filter *.a $translator | Remove-Item
|
||||
& (Join-Path $NdkLlvm 'bin\clang.exe') --target=aarch64-linux-android29 -O2 -Wall -Wextra -Werror -pie -fPIE `
|
||||
-o (Join-Path $translator 'translator_host') (Join-Path $PSScriptRoot 'toolchain\translator_host.c')
|
||||
if ($LASTEXITCODE -ne 0) { throw "Building translator_host failed ($LASTEXITCODE)" }
|
||||
|
||||
$ndk = Get-NdkSubset
|
||||
if ($ndkSubsetDigest -and $ndk.digest -ne $ndkSubsetDigest) {
|
||||
throw "The NDK subset digest $($ndk.digest) does not match the pinned $ndkSubsetDigest"
|
||||
}
|
||||
[IO.File]::WriteAllText((Join-Path $OutputDir 'ndk.json'), ($ndk | ConvertTo-Json -Depth 4 -Compress), (New-Object Text.UTF8Encoding $false))
|
||||
|
||||
# .NET enumeration keeps the root as given (Get-ChildItem can expand an 8.3 TEMP name).
|
||||
$root = [IO.Path]::GetFullPath($stage).TrimEnd('\')
|
||||
$files = @([IO.Directory]::EnumerateFiles($root, '*', [IO.SearchOption]::AllDirectories) | ForEach-Object {
|
||||
[pscustomobject]@{ path = $_.Substring($root.Length + 1).Replace('\', '/'); size = (New-Object IO.FileInfo $_).Length; sha256 = Get-Sha256Hex $_ }
|
||||
} | Sort-Object path)
|
||||
$zip = [IO.Compression.ZipFile]::Open((Join-Path $OutputDir 'files.zip'), 'Create')
|
||||
try {
|
||||
foreach ($file in $files) {
|
||||
[void][IO.Compression.ZipFileExtensions]::CreateEntryFromFile($zip, (Join-Path $root $file.path), $file.path, 'Optimal')
|
||||
}
|
||||
} finally { $zip.Dispose() }
|
||||
|
||||
$toolchain = [ordered]@{
|
||||
schema = 1
|
||||
fingerprint = Get-ListDigest $files
|
||||
archive = 'files.zip'
|
||||
ndk = 'ndk.json'
|
||||
files = $files
|
||||
}
|
||||
[IO.File]::WriteAllText((Join-Path $OutputDir 'toolchain.json'), ($toolchain | ConvertTo-Json -Depth 4), (New-Object Text.UTF8Encoding $false))
|
||||
$unpacked = [math]::Round((($files | Measure-Object size -Sum).Sum) / 1MB)
|
||||
$packed = [math]::Round((New-Object IO.FileInfo (Join-Path $OutputDir 'files.zip')).Length / 1MB)
|
||||
Write-Host "Quest toolchain $($toolchain.fingerprint) ($unpacked MB, $packed MB packed; NDK subset $($ndk.digest))"
|
||||
+113
-11
@@ -81,13 +81,22 @@ function Get-StringSha256Hex([string]$Text) {
|
||||
# Identity of a runtime/include tree. Translated code compiles against these headers, so a game is
|
||||
# only built from a translation whose runtime headers are the kit's own.
|
||||
function Get-RuntimeIncludeFingerprint([string]$Directory) {
|
||||
$root = (Resolve-Path $Directory).Path.TrimEnd('\', '/')
|
||||
$lines = Get-ChildItem -Recurse -File $root | Sort-Object FullName | ForEach-Object {
|
||||
$_.FullName.Substring($root.Length + 1).Replace('\', '/') + ' ' + (Get-Sha256Hex $_.FullName)
|
||||
$lines = Get-RelativeFiles $Directory | Sort-Object FullName | ForEach-Object {
|
||||
$_.Relative + ' ' + (Get-Sha256Hex $_.FullName)
|
||||
}
|
||||
return Get-StringSha256Hex ($lines -join "`n")
|
||||
}
|
||||
|
||||
# The files below a directory with their paths relative to it. .NET enumeration keeps the root
|
||||
# exactly as given; Resolve-Path and Get-ChildItem can disagree about an 8.3 short name (TEMP),
|
||||
# which would shift every relative path.
|
||||
function Get-RelativeFiles([string]$Directory) {
|
||||
$root = [IO.Path]::GetFullPath($Directory).TrimEnd('\', '/')
|
||||
foreach ($path in [IO.Directory]::EnumerateFiles($root, '*', [IO.SearchOption]::AllDirectories)) {
|
||||
[pscustomobject]@{ FullName = $path; Relative = $path.Substring($root.Length + 1).Replace('\', '/') }
|
||||
}
|
||||
}
|
||||
|
||||
# compile_commands.json as {file, command} objects. Windows PowerShell's ConvertFrom-Json refuses
|
||||
# documents over 2 MB, so 5.1 uses the serializer underneath it with the limit lifted.
|
||||
function Read-CompileCommands([string]$Path) {
|
||||
@@ -133,6 +142,91 @@ function ConvertTo-KitCompileFlags {
|
||||
return ,$flags.ToArray()
|
||||
}
|
||||
|
||||
# The arguments of a `clang -###` command line: double-quoted, with backslash escapes.
|
||||
function Split-DriverCommand([string]$Line) {
|
||||
$arguments = New-Object System.Collections.Generic.List[string]
|
||||
$current = New-Object System.Text.StringBuilder
|
||||
$inQuotes = $false
|
||||
for ($i = 0; $i -lt $Line.Length; $i++) {
|
||||
$c = $Line[$i]
|
||||
if ($inQuotes) {
|
||||
if ($c -eq '\' -and $i + 1 -lt $Line.Length) { [void]$current.Append($Line[++$i]) }
|
||||
elseif ($c -eq '"') { $arguments.Add($current.ToString()); [void]$current.Clear(); $inQuotes = $false }
|
||||
else { [void]$current.Append($c) }
|
||||
} elseif ($c -eq '"') { $inQuotes = $true }
|
||||
}
|
||||
return , $arguments.ToArray()
|
||||
}
|
||||
|
||||
# The kit's link as a raw ld.lld command, for the headset. Android lets the app start its tools
|
||||
# only through the system linker, so clang there cannot start lld itself and the builder runs lld
|
||||
# directly. The NDK's own driver expands the link here, with the kit and marker objects standing in
|
||||
# as real files. Placeholders: {kit}, {ndk} (the NDK's prebuilt toolchain directory, which holds
|
||||
# sysroot/ and lib/clang/), {output}, and {game:runtime}, {game:product}, {game:translated}, each
|
||||
# replaced by its object files (the translated ones already sit inside --start-lib/--end-lib).
|
||||
function Get-KitLldArguments {
|
||||
param([string]$ClangCxx, [string]$KitDir, [string[]]$Flags, [string[]]$Inputs)
|
||||
$kit = ConvertTo-ForwardPath $KitDir
|
||||
$ndk = ConvertTo-ForwardPath (Join-Path (Split-Path -Parent $ClangCxx) '..')
|
||||
# Forward slashes throughout: clang reads response files with GNU quoting, where '\' escapes.
|
||||
$probe = ConvertTo-ForwardPath (Join-Path ([IO.Path]::GetTempPath()) ('mkw-kit-link-' + [Guid]::NewGuid().ToString('N')))
|
||||
New-Item -ItemType Directory $probe | Out-Null
|
||||
try {
|
||||
$markers = [ordered]@{}
|
||||
foreach ($slot in 'runtime', 'product', 'translated') {
|
||||
$markers[$slot] = ConvertTo-ForwardPath (Join-Path $probe "game_$slot.o")
|
||||
[IO.File]::WriteAllBytes($markers[$slot], [byte[]]@())
|
||||
}
|
||||
$real = { param([string]$s) $s.Replace('{kit}', $kit).Replace('{sysroot}', "$ndk/sysroot") }
|
||||
$arguments = New-Object System.Collections.Generic.List[string]
|
||||
foreach ($flag in $Flags) { $arguments.Add((& $real $flag)) }
|
||||
$arguments.Add('-o'); $arguments.Add("$probe/libmain.so")
|
||||
foreach ($linkInput in $Inputs) {
|
||||
switch ($linkInput) {
|
||||
'{game:runtime}' { $arguments.Add($markers.runtime) }
|
||||
'{game:product}' { $arguments.Add($markers.product) }
|
||||
'{game:translated}' { $arguments.Add('-Wl,--start-lib'); $arguments.Add($markers.translated); $arguments.Add('-Wl,--end-lib') }
|
||||
default { $arguments.Add((& $real $linkInput)) }
|
||||
}
|
||||
}
|
||||
$rsp = Join-Path $probe 'link.rsp'
|
||||
[IO.File]::WriteAllText($rsp, (($arguments | ForEach-Object { ConvertTo-QuotedArgument $_ }) -join "`n"), (New-Object Text.UTF8Encoding $false))
|
||||
|
||||
# -### prints the command on stderr; Windows PowerShell turns redirected native stderr into errors.
|
||||
$start = New-Object Diagnostics.ProcessStartInfo $ClangCxx, "-### `"@$rsp`""
|
||||
$start.UseShellExecute = $false
|
||||
$start.RedirectStandardError = $true
|
||||
$start.RedirectStandardOutput = $true
|
||||
$process = [Diagnostics.Process]::Start($start)
|
||||
$stdout = $process.StandardOutput.ReadToEndAsync()
|
||||
$stderr = $process.StandardError.ReadToEnd()
|
||||
$process.WaitForExit()
|
||||
[void]$stdout.Result
|
||||
if ($process.ExitCode -ne 0) { throw "clang++ -### failed for the kit link:`n$stderr" }
|
||||
$line = @($stderr -split "`r?`n" | Where-Object { $_ -match '^\s*"[^"]*ld(\.lld)?(\.exe)?"' }) | Select-Object -Last 1
|
||||
if (-not $line) { throw "clang++ -### printed no linker command:`n$stderr" }
|
||||
|
||||
$lld = New-Object System.Collections.Generic.List[string]
|
||||
$driverArguments = Split-DriverCommand $line
|
||||
for ($i = 1; $i -lt $driverArguments.Length; $i++) {
|
||||
$argument = $driverArguments[$i].Replace('\', '/')
|
||||
if ($argument -eq "$probe/libmain.so") { $lld.Add('{output}'); continue }
|
||||
$slot = @($markers.Keys | Where-Object { $markers[$_] -eq $argument })
|
||||
if ($slot.Count -eq 1) { $lld.Add("{game:$($slot[0])}"); continue }
|
||||
$argument = [regex]::Replace($argument, [regex]::Escape($kit), '{kit}', 'IgnoreCase')
|
||||
$argument = [regex]::Replace($argument, [regex]::Escape($ndk), '{ndk}', 'IgnoreCase')
|
||||
if ($argument -match '[A-Za-z]:/') { throw "The kit link names a path outside the kit and the NDK: $argument" }
|
||||
$lld.Add($argument)
|
||||
}
|
||||
foreach ($required in '{output}', '{game:runtime}', '{game:product}', '{game:translated}') {
|
||||
if (-not $lld.Contains($required)) { throw "The expanded kit link lacks $required" }
|
||||
}
|
||||
return , $lld.ToArray()
|
||||
} finally {
|
||||
Remove-Item -Recurse -Force $probe
|
||||
}
|
||||
}
|
||||
|
||||
function Export-QuestGameKit {
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
@@ -239,6 +333,16 @@ function Export-QuestGameKit {
|
||||
}
|
||||
}
|
||||
$linkFlags.Add('-Wl,-soname,libmain.so')
|
||||
$lld = Get-KitLldArguments -ClangCxx (Join-Path (Split-Path -Parent $LlvmStrip) 'clang++.exe') -KitDir $OutputDir `
|
||||
-Flags $linkFlags.ToArray() -Inputs $inputs.ToArray()
|
||||
|
||||
# What the headset's translator needs besides the disc: the game manifest and symbol map, and
|
||||
# the runtime sources it indexes for native registrations, which must be the ones compiled into
|
||||
# this kit.
|
||||
$translation = Join-Path $OutputDir 'translation'
|
||||
New-Item -ItemType Directory -Force (Join-Path $translation 'projects/mkwii'), (Join-Path $translation 'runtime') | Out-Null
|
||||
Copy-Item (Join-Path $RepoRoot 'projects/mkwii/recomp.yml'), (Join-Path $RepoRoot 'projects/mkwii/MAP.txt') (Join-Path $translation 'projects/mkwii')
|
||||
Copy-Item -Recurse (Join-Path $RepoRoot 'runtime/src') (Join-Path $translation 'runtime/src')
|
||||
|
||||
$recipe = [ordered]@{
|
||||
schema = $script:KitSchema
|
||||
@@ -247,12 +351,11 @@ function Export-QuestGameKit {
|
||||
target = $target
|
||||
runtimeIncludeFingerprint = Get-RuntimeIncludeFingerprint (Join-Path $OutputDir 'include')
|
||||
compile = $compile
|
||||
link = [ordered]@{ flags = $linkFlags.ToArray(); inputs = $inputs.ToArray() }
|
||||
link = [ordered]@{ flags = $linkFlags.ToArray(); inputs = $inputs.ToArray(); lld = $lld }
|
||||
}
|
||||
$recipeJson = $recipe | ConvertTo-Json -Depth 8 -Compress
|
||||
$files = Get-ChildItem -Recurse -File $OutputDir | Sort-Object FullName | ForEach-Object {
|
||||
$relative = $_.FullName.Substring((Resolve-Path $OutputDir).Path.Length).Replace('\', '/')
|
||||
"$relative $(Get-Sha256Hex $_.FullName)"
|
||||
$files = Get-RelativeFiles $OutputDir | Sort-Object FullName | ForEach-Object {
|
||||
"/$($_.Relative) $(Get-Sha256Hex $_.FullName)"
|
||||
}
|
||||
$recipe.fingerprint = Get-StringSha256Hex ($recipeJson + "`n" + ($files -join "`n"))
|
||||
[IO.File]::WriteAllText((Join-Path $OutputDir 'kit.json'), ($recipe | ConvertTo-Json -Depth 8), (New-Object Text.UTF8Encoding $false))
|
||||
@@ -404,6 +507,7 @@ function New-QuestGamePackage {
|
||||
}
|
||||
Add-Type -AssemblyName System.IO.Compression, System.IO.Compression.FileSystem
|
||||
if (Test-Path $OutputPath) { Remove-Item -Force $OutputPath }
|
||||
New-Item -ItemType Directory -Force (Split-Path -Parent ([IO.Path]::GetFullPath($OutputPath))) | Out-Null
|
||||
$zip = [IO.Compression.ZipFile]::Open($OutputPath, 'Create')
|
||||
try {
|
||||
$entry = $zip.CreateEntry('game.json')
|
||||
@@ -411,10 +515,8 @@ function New-QuestGamePackage {
|
||||
try { $writer.Write(($game | ConvertTo-Json)) } finally { $writer.Dispose() }
|
||||
[void][IO.Compression.ZipFileExtensions]::CreateEntryFromFile($zip, $Library, $recipe.output, 'Optimal')
|
||||
if ($DataDir) {
|
||||
$dataRoot = (Resolve-Path $DataDir).Path.TrimEnd('\', '/')
|
||||
foreach ($file in Get-ChildItem -Recurse -File $dataRoot) {
|
||||
$name = 'DATA/' + $file.FullName.Substring($dataRoot.Length + 1).Replace('\', '/')
|
||||
[void][IO.Compression.ZipFileExtensions]::CreateEntryFromFile($zip, $file.FullName, $name, 'NoCompression')
|
||||
foreach ($file in Get-RelativeFiles $DataDir) {
|
||||
[void][IO.Compression.ZipFileExtensions]::CreateEntryFromFile($zip, $file.FullName, "DATA/$($file.Relative)", 'NoCompression')
|
||||
}
|
||||
}
|
||||
} finally { $zip.Dispose() }
|
||||
|
||||
+4
-2
@@ -4,7 +4,8 @@ Standalone Android/OpenXR build of the Mario Kart Wii recompilation for Quest 2,
|
||||
Quest 3, Quest 3S and Quest Pro. The full design, build walkthrough and current
|
||||
status live in [docs/quest-port.md](../docs/quest-port.md); this directory only
|
||||
holds the Gradle project, its helper scripts, the game kit tooling
|
||||
(`QuestGameKit.psm1`, `Build-QuestGame.ps1`) and `nod-jni`.
|
||||
(`QuestGameKit.psm1`, `Build-QuestGame.ps1`), the on-headset build toolchain
|
||||
(`Prepare-QuestToolchain.ps1`, `toolchain/`) and `nod-jni`.
|
||||
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File android/Prepare-QuestDependencies.ps1 # stages the SDL3 3.4.4 AAR once
|
||||
@@ -16,7 +17,8 @@ The translated game (the translator's `generated/` tree for **your own**
|
||||
PAL `RMCP01` disc) is passed with `-Generated <dir>`; it defaults to the
|
||||
installer workspace next to this checkout. No game data is ever part of the
|
||||
APK, and neither is any translated game code: the base APK carries a game kit,
|
||||
and your `libmain.so` is built from your own disc and imported
|
||||
and your `libmain.so` is built from your own disc, either on the headset
|
||||
(**Build on this Quest**, about half an hour) or on a PC and imported
|
||||
(`Build-QuestGame.ps1 -Install`, or **Import from computer** with its `.wcgame`).
|
||||
Copy your disc image to the headset (for example `adb push MarioKart.iso
|
||||
/sdcard/Download/`) and press **Select disc image** in the launcher, which checks
|
||||
|
||||
@@ -229,8 +229,52 @@ abstract class ExportQuestGameKit : DefaultTask() {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The toolchain for building the game on the headset (android/Prepare-QuestToolchain.ps1): the
|
||||
* translator for Android, clang/lld, and the pin of the NDK files the headset downloads. Packaged
|
||||
* into the base variants' assets as quest_toolchain/.
|
||||
*/
|
||||
abstract class PrepareQuestToolchain : DefaultTask() {
|
||||
@get:OutputDirectory
|
||||
abstract val outputDir: DirectoryProperty
|
||||
|
||||
@get:InputFiles
|
||||
@get:PathSensitive(PathSensitivity.RELATIVE)
|
||||
abstract val sources: ConfigurableFileCollection
|
||||
|
||||
@get:Internal
|
||||
abstract val script: RegularFileProperty
|
||||
|
||||
@get:Internal
|
||||
abstract val ndkLlvm: DirectoryProperty
|
||||
|
||||
@get:Inject
|
||||
abstract val execOperations: ExecOperations
|
||||
|
||||
@TaskAction
|
||||
fun prepare() {
|
||||
execOperations.exec {
|
||||
commandLine(
|
||||
"powershell", "-NoProfile", "-ExecutionPolicy", "Bypass", "-File", script.get().asFile.path,
|
||||
"-OutputDir", File(outputDir.get().asFile, "quest_toolchain").path, "-NdkLlvm", ndkLlvm.get().asFile.path,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
androidComponents {
|
||||
onVariants(selector().withFlavor("profile" to "base")) { variant ->
|
||||
val prepareToolchain = tasks.register<PrepareQuestToolchain>(
|
||||
"prepare${variant.name.replaceFirstChar { it.uppercase() }}QuestToolchain"
|
||||
) {
|
||||
script.set(rootProject.layout.projectDirectory.file("Prepare-QuestToolchain.ps1"))
|
||||
sources.from(script, rootProject.layout.projectDirectory.dir("toolchain"))
|
||||
sources.from(fileTree(File(mkwRepoRoot, "translator/src")) { exclude("**/bin/**", "**/obj/**") })
|
||||
sources.from(fileTree(File(mkwRepoRoot, "Launcher/WiiCompiled.Setup.Common")) { exclude("**/bin/**", "**/obj/**") })
|
||||
val host = if (System.getProperty("os.name").startsWith("Windows")) "windows-x86_64" else "linux-x86_64"
|
||||
ndkLlvm.set(sdkComponents.ndkDirectory.map { it.dir("toolchains/llvm/prebuilt/$host") })
|
||||
}
|
||||
variant.sources.assets?.addGeneratedSourceDirectory(prepareToolchain, PrepareQuestToolchain::outputDir)
|
||||
val capitalized = variant.name.replaceFirstChar { it.uppercase() }
|
||||
// AGP packages every library in the CMake output directory, so a game library left over
|
||||
// from a build that still linked the game would otherwise ship in the APK.
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
package org.wiicompiled.quest.launcher
|
||||
|
||||
import java.security.MessageDigest
|
||||
|
||||
/**
|
||||
* The text-in, text-out parts of building the game on the headset ([GameBuild]): reading the
|
||||
* translator's outputs and turning the game kit's recipe (kit.json) into compiler and linker
|
||||
* command lines. They mirror android/QuestGameKit.psm1, which does the same on a PC.
|
||||
*/
|
||||
object BuildRecipe {
|
||||
|
||||
/** The translated source lists emit-build-shards writes into shards.cmake, in link order. */
|
||||
val TRANSLATED_LISTS = listOf("MKW_BASE_COMMON_SHARDS", "MKW_BASE_PORTABLE_SENSITIVE_SHARDS")
|
||||
const val REGISTRATION_LIST = "MKW_BASE_REGISTRATION_SOURCES"
|
||||
|
||||
/**
|
||||
* One argument of a response file for clang or lld on Android, which read them with GNU
|
||||
* quoting: a backslash escapes the next character.
|
||||
*/
|
||||
fun responseFileArgument(argument: String): String = buildString {
|
||||
for (c in argument) {
|
||||
if (c == '\\' || c == '"' || c == '\'' || c.isWhitespace()) append('\\')
|
||||
append(c)
|
||||
}
|
||||
}
|
||||
|
||||
fun responseFile(arguments: List<String>): String =
|
||||
arguments.joinToString(separator = "\n", postfix = "\n") { responseFileArgument(it) }
|
||||
|
||||
/** Replaces the kit recipe's {name} placeholders; an unknown placeholder is an error. */
|
||||
fun expand(argument: String, values: Map<String, String>): String {
|
||||
var result = argument
|
||||
for ((name, value) in values) result = result.replace("{$name}", value)
|
||||
PLACEHOLDER.find(result)?.let { throw IllegalArgumentException("Unexpanded ${it.value} in the game kit recipe") }
|
||||
return result
|
||||
}
|
||||
|
||||
/**
|
||||
* The link.lld arguments with every {game:slot} replaced by that slot's object files
|
||||
* (runtime, product, translated) and the other placeholders expanded.
|
||||
*/
|
||||
fun linkArguments(template: List<String>, values: Map<String, String>, objects: Map<String, List<String>>): List<String> {
|
||||
val arguments = ArrayList<String>(template.size + objects.values.sumOf { it.size })
|
||||
for (argument in template) {
|
||||
val slot = GAME_SLOT.matchEntire(argument)?.groupValues?.get(1)
|
||||
if (slot != null) {
|
||||
arguments += objects[slot] ?: throw IllegalArgumentException("The game kit recipe names an unknown slot {game:$slot}")
|
||||
} else {
|
||||
arguments += expand(argument, values)
|
||||
}
|
||||
}
|
||||
return arguments
|
||||
}
|
||||
|
||||
/** The quoted entries of `set(NAME ...)` in a shards.cmake, or an empty list without it. */
|
||||
fun sourceList(shardsCmake: String, name: String): List<String> {
|
||||
val body = Regex("""set\(${Regex.escape(name)}\s(.*?)\)""", RegexOption.DOT_MATCHES_ALL).find(shardsCmake)
|
||||
?.groupValues?.get(1) ?: return emptyList()
|
||||
return Regex("\"([^\"]+)\"").findAll(body).map { it.groupValues[1].replace('\\', '/') }.toList()
|
||||
}
|
||||
|
||||
/** The first translation entry point in recomp.yml, as the translator expects it (0x...). */
|
||||
fun entryPoint(manifest: String): String =
|
||||
Regex("""\n\s*entry_points:\s*\n\s*-\s*(0x[0-9A-Fa-f]{8})""").find(manifest)?.groupValues?.get(1)
|
||||
?: throw IllegalArgumentException("recomp.yml names no translation entry point")
|
||||
|
||||
/**
|
||||
* The blob assembly for ELF, as runtime/cmake/PublicProducts.cmake rewrites a Windows-generated
|
||||
* one. The headset's translator already writes ELF sections, so this normally changes nothing.
|
||||
*/
|
||||
fun elfBlobAssembly(text: String): String {
|
||||
val elf = text.replace(".section .rdata,\"dr\"", ".section .rodata,\"a\",@progbits")
|
||||
return if (elf == text) text else elf + "\n.section .note.GNU-stack,\"\",@progbits\n"
|
||||
}
|
||||
|
||||
/** An object file name for a generated source: its stem, which the translator keeps unique. */
|
||||
fun objectName(source: String): String = source.substringAfterLast('/').substringBeforeLast('.') + ".o"
|
||||
|
||||
/** SHA-256 over "path sha256" lines sorted by path (ordinal), joined with \n. */
|
||||
fun listDigest(files: Map<String, String>): String {
|
||||
val lines = files.entries.map { "${it.key} ${it.value}" }.sorted()
|
||||
return hex(MessageDigest.getInstance("SHA-256").digest(lines.joinToString("\n").toByteArray(Charsets.UTF_8)))
|
||||
}
|
||||
|
||||
fun hex(bytes: ByteArray): String = bytes.joinToString("") { "%02x".format(it) }
|
||||
|
||||
private val PLACEHOLDER = Regex("""\{[a-z:]+\}""")
|
||||
private val GAME_SLOT = Regex("""\{game:(\w+)\}""")
|
||||
}
|
||||
@@ -0,0 +1,493 @@
|
||||
package org.wiicompiled.quest.launcher
|
||||
|
||||
import android.app.ActivityManager
|
||||
import android.content.Context
|
||||
import android.os.StatFs
|
||||
import android.util.Log
|
||||
import java.io.File
|
||||
import java.io.IOException
|
||||
import java.io.InterruptedIOException
|
||||
import java.net.HttpURLConnection
|
||||
import java.net.URL
|
||||
import java.security.MessageDigest
|
||||
import java.text.SimpleDateFormat
|
||||
import java.util.Date
|
||||
import java.util.Locale
|
||||
import java.util.TimeZone
|
||||
import java.util.concurrent.Executors
|
||||
import java.util.concurrent.Future
|
||||
import java.util.concurrent.TimeUnit
|
||||
import java.util.concurrent.atomic.AtomicInteger
|
||||
import java.util.concurrent.atomic.AtomicReference
|
||||
import java.util.zip.ZipInputStream
|
||||
import org.json.JSONArray
|
||||
import org.json.JSONObject
|
||||
import org.wiicompiled.quest.BuildConfig
|
||||
import org.wiicompiled.quest.GameLibrary
|
||||
import org.wiicompiled.quest.GameStorage
|
||||
|
||||
/**
|
||||
* Builds the game on the headset from the player's extracted disc, the way the PC launcher's
|
||||
* Build for Quest does (android/Build-QuestGame.ps1), for players without a PC:
|
||||
*
|
||||
* 1. unpack the toolchain (assets/quest_toolchain, android/Prepare-QuestToolchain.ps1) and the
|
||||
* game kit (assets/game_kit) into private storage;
|
||||
* 2. download the Android NDK files the build needs from Google, checked against the pins the
|
||||
* toolchain carries (ndk.json);
|
||||
* 3. translate main.dol and StaticR.rel: translate-recursive, generate-data-init, emit-build-shards;
|
||||
* 4. compile the generated sources with the kit's flags, several at a time;
|
||||
* 5. link them with the kit's objects and archives (kit.json link.lld);
|
||||
* 6. install libmain.so and its game.json like an imported game.
|
||||
*
|
||||
* About half an hour on a Quest 3. Work survives a cancelled or failed run: a translation whose
|
||||
* inputs have not changed is reused and finished objects are not compiled again. A successful
|
||||
* build removes everything it unpacked and generated.
|
||||
*/
|
||||
object GameBuild {
|
||||
|
||||
enum class Step { Prepare, Download, Translate, Compile, Link, Install }
|
||||
|
||||
/** [permille] of the whole build; [done] of [total] items in [step]. False stops the build. */
|
||||
fun interface Reporter {
|
||||
fun update(permille: Int, step: Step, done: Int, total: Int): Boolean
|
||||
}
|
||||
|
||||
private const val TAG = "WiiCompiledLauncher"
|
||||
private const val TOOLCHAIN_ASSETS = "quest_toolchain"
|
||||
private const val KIT_ASSETS = "game_kit"
|
||||
private const val MARKER = ".complete"
|
||||
private const val REQUIRED_SPACE = 2_500_000_000L
|
||||
// Translation peaks near 3.1 GB of memory with the runtime's default heap; this limit keeps it
|
||||
// near 2.1 GB for the same output, at some cost in time.
|
||||
private const val MONO_GC_PARAMS = "soft-heap-limit=1200m"
|
||||
private const val TRANSLATOR_THREADS = 4
|
||||
private const val EXPECTED_TRANSLATION_SECONDS = 600.0
|
||||
private const val COMPILE_MEMORY_BYTES = 700L * 1024 * 1024
|
||||
|
||||
/** Null on success, otherwise the message to show. */
|
||||
fun run(context: Context, reporter: Reporter, cancelled: () -> Boolean, finishing: () -> Unit): String? {
|
||||
val disc = GameStorage.discDirectory(context)
|
||||
if (GameStorage.discStatus(context) != GameStorage.DiscStatus.Ready) {
|
||||
return "The game files (DATA) are needed to build the game. Select your disc image first."
|
||||
}
|
||||
GameFiles.validateData(disc)?.let { return it }
|
||||
val root = File(context.filesDir, "build")
|
||||
root.mkdirs()
|
||||
val available = StatFs(root.absolutePath).availableBytes
|
||||
if (available < REQUIRED_SPACE) {
|
||||
return "Not enough free space: building needs ${GameFiles.gigabytes(REQUIRED_SPACE)}, and ${GameFiles.gigabytes(available)} is free."
|
||||
}
|
||||
|
||||
val stamp = SimpleDateFormat("yyyyMMdd_HHmmss", Locale.US).format(Date())
|
||||
val log = BuildLog(File(GameStorage.logsDirectory(context), "build_$stamp.log"))
|
||||
try {
|
||||
return Build(context, root, disc, reporter, cancelled, finishing, log).run()
|
||||
} catch (e: InterruptedIOException) {
|
||||
throw e
|
||||
} catch (e: Exception) {
|
||||
Log.e(TAG, "Game build failed", e)
|
||||
log.line("Build failed: $e")
|
||||
return "${e.message ?: e.toString()}\n\nThe build log is ${log.file.absolutePath}"
|
||||
} finally {
|
||||
log.close()
|
||||
}
|
||||
}
|
||||
|
||||
private class Build(
|
||||
val context: Context,
|
||||
val root: File,
|
||||
val disc: File,
|
||||
val reporter: Reporter,
|
||||
val cancelled: () -> Boolean,
|
||||
val finishing: () -> Unit,
|
||||
val log: BuildLog,
|
||||
) {
|
||||
val toolchain = File(root, "toolchain")
|
||||
val ndk = File(root, "ndk")
|
||||
val kit = File(root, "kit")
|
||||
val work = File(root, GameLibrary.directory(context).name)
|
||||
val workspace = File(work, "ws")
|
||||
val objects = File(work, "obj")
|
||||
val temporary = File(root, "tmp")
|
||||
val failure = AtomicReference<String?>(null)
|
||||
|
||||
fun run(): String? {
|
||||
log.line("WiiCompiled Quest ${BuildConfig.VERSION_NAME} building ${BuildConfig.PROFILE} on this headset")
|
||||
report(0, Step.Prepare)
|
||||
val toolchainManifest = JSONObject(context.assets.open("$TOOLCHAIN_ASSETS/toolchain.json").reader().use { it.readText() })
|
||||
unpackToolchain(toolchainManifest)
|
||||
report(30, Step.Prepare)
|
||||
val kitFingerprint = GameLibrary.kitFingerprint(context) ?: return "This app carries no game kit."
|
||||
unpackKit(kitFingerprint)
|
||||
val recipe = JSONObject(File(kit, "kit.json").readText())
|
||||
report(40, Step.Prepare)
|
||||
downloadNdk(JSONObject(context.assets.open("$TOOLCHAIN_ASSETS/${toolchainManifest.getString("ndk")}").reader().use { it.readText() }))
|
||||
|
||||
temporary.mkdirs()
|
||||
val tools = ToolProcess(
|
||||
mapOf(
|
||||
"LD_LIBRARY_PATH" to File(toolchain, "llvm/lib").absolutePath,
|
||||
"HOME" to temporary.absolutePath,
|
||||
"TMPDIR" to temporary.absolutePath,
|
||||
"MONO_GC_PARAMS" to MONO_GC_PARAMS,
|
||||
),
|
||||
log,
|
||||
cancelled = { cancelled() || failure.get() != null },
|
||||
)
|
||||
translate(tools, "$kitFingerprint ${toolchainManifest.getString("fingerprint")}")?.let { return it }
|
||||
val objectsBySlot = compile(tools, recipe, toolchainManifest.getString("fingerprint"))
|
||||
failure.get()?.let { return it }
|
||||
val library = link(tools, recipe, objectsBySlot) ?: return failure.get() ?: "Linking the game failed."
|
||||
|
||||
finishing()
|
||||
report(990, Step.Install)
|
||||
install(library, kitFingerprint)?.let { return it }
|
||||
log.line("Installed the game built on this headset")
|
||||
// Everything above is only needed again after an app update, which changes the kit.
|
||||
if (!root.deleteRecursively()) Log.w(TAG, "Could not remove all of ${root.absolutePath}")
|
||||
return null
|
||||
}
|
||||
|
||||
fun report(permille: Int, step: Step, done: Int = 0, total: Int = 0) {
|
||||
if (!reporter.update(permille, step, done, total) || cancelled()) throw InterruptedIOException("Build cancelled")
|
||||
}
|
||||
|
||||
/** Unpacks files.zip, which holds exactly the files toolchain.json lists. */
|
||||
fun unpackToolchain(manifest: JSONObject) {
|
||||
val fingerprint = manifest.getString("fingerprint")
|
||||
if (File(toolchain, MARKER).takeIf { it.isFile }?.readText() == fingerprint) return
|
||||
toolchain.deleteRecursively()
|
||||
val listed = manifest.getJSONArray("files")
|
||||
val sizes = HashMap<String, Long>()
|
||||
for (i in 0 until listed.length()) {
|
||||
val file = listed.getJSONObject(i)
|
||||
sizes[file.getString("path")] = file.getLong("size")
|
||||
}
|
||||
var unpacked = 0
|
||||
ZipInputStream(context.assets.open("$TOOLCHAIN_ASSETS/${manifest.getString("archive")}").buffered(1 shl 20)).use { zip ->
|
||||
while (true) {
|
||||
val entry = zip.nextEntry ?: break
|
||||
if (entry.isDirectory) continue
|
||||
val expected = sizes[entry.name] ?: throw IOException("The app's toolchain holds an unlisted file ${entry.name}.")
|
||||
val target = File(toolchain, entry.name)
|
||||
target.parentFile?.mkdirs()
|
||||
target.outputStream().use { zip.copyTo(it, 1 shl 20) }
|
||||
if (target.length() != expected) throw IOException("The app's toolchain file ${entry.name} is damaged.")
|
||||
unpacked++
|
||||
report(30 * unpacked / sizes.size, Step.Prepare)
|
||||
}
|
||||
}
|
||||
if (unpacked != sizes.size) throw IOException("The app's toolchain is incomplete.")
|
||||
File(toolchain, MARKER).writeText(fingerprint)
|
||||
}
|
||||
|
||||
fun unpackKit(fingerprint: String) {
|
||||
if (File(kit, MARKER).takeIf { it.isFile }?.readText() == fingerprint) return
|
||||
kit.deleteRecursively()
|
||||
copyAssetTree(KIT_ASSETS, kit)
|
||||
File(kit, MARKER).writeText(fingerprint)
|
||||
}
|
||||
|
||||
fun copyAssetTree(asset: String, target: File) {
|
||||
val children = context.assets.list(asset).orEmpty()
|
||||
if (children.isEmpty()) {
|
||||
copyAsset(asset, target)
|
||||
return
|
||||
}
|
||||
for (child in children) copyAssetTree("$asset/$child", File(target, child))
|
||||
}
|
||||
|
||||
fun copyAsset(asset: String, target: File) {
|
||||
target.parentFile?.mkdirs()
|
||||
context.assets.open(asset).use { input -> target.outputStream().use { input.copyTo(it, 1 shl 20) } }
|
||||
}
|
||||
|
||||
/** The NDK subset ndk.json lists, from Google's zip, each file checked against its SHA-256. */
|
||||
fun downloadNdk(subset: JSONObject) {
|
||||
val digest = subset.getString("digest")
|
||||
if (File(ndk, MARKER).takeIf { it.isFile }?.readText() == digest) return
|
||||
report(40, Step.Download)
|
||||
ndk.deleteRecursively()
|
||||
val url = subset.getString("url")
|
||||
val prefix = subset.getString("prefix")
|
||||
val expected = HashMap<String, String>()
|
||||
val listed: JSONArray = subset.getJSONArray("files")
|
||||
for (i in 0 until listed.length()) {
|
||||
val file = listed.getJSONObject(i)
|
||||
expected[prefix + file.getString("path")] = file.getString("sha256")
|
||||
}
|
||||
log.line("Downloading ${expected.size} Android NDK files from $url")
|
||||
val zip = RemoteZip(subset.getLong("size")) { start, end -> fetchRange(url, start, end) }
|
||||
val wanted = zip.entries().filter { it.name in expected }
|
||||
if (wanted.size != expected.size) throw IOException("Google's Android NDK download no longer has the expected files.")
|
||||
val totalBytes = wanted.sumOf { it.end - it.offset }
|
||||
var doneBytes = 0L
|
||||
val actual = HashMap<String, String>()
|
||||
zip.read(wanted) { entry, bytes ->
|
||||
val sha256 = BuildRecipe.hex(MessageDigest.getInstance("SHA-256").digest(bytes))
|
||||
if (sha256 != expected[entry.name]) throw IOException("The downloaded Android NDK file ${entry.name} is damaged. Try again.")
|
||||
val path = entry.name.removePrefix(prefix)
|
||||
File(ndk, path).apply { parentFile?.mkdirs() }.writeBytes(bytes)
|
||||
actual[path] = sha256
|
||||
doneBytes += entry.end - entry.offset
|
||||
report(40 + (10 * doneBytes / totalBytes).toInt(), Step.Download, (doneBytes shr 20).toInt(), (totalBytes shr 20).toInt())
|
||||
}
|
||||
if (BuildRecipe.listDigest(actual) != digest) throw IOException("The downloaded Android NDK files do not match this app's pins.")
|
||||
File(ndk, MARKER).writeText(digest)
|
||||
}
|
||||
|
||||
fun fetchRange(url: String, start: Long, end: Long): ByteArray {
|
||||
if (cancelled()) throw InterruptedIOException("Build cancelled")
|
||||
val connection = URL(url).openConnection() as HttpURLConnection
|
||||
try {
|
||||
connection.connectTimeout = 30_000
|
||||
connection.readTimeout = 60_000
|
||||
connection.setRequestProperty("Range", "bytes=$start-$end")
|
||||
// Android asks for gzip by default, and Google's server then serves a gzip-encoded
|
||||
// zip whose ranges are not the file's (HTTP 416 near the end).
|
||||
connection.setRequestProperty("Accept-Encoding", "identity")
|
||||
val code = try {
|
||||
connection.responseCode
|
||||
} catch (e: IOException) {
|
||||
throw IOException("The Android NDK files could not be downloaded from Google. Check the headset's internet connection.", e)
|
||||
}
|
||||
if (code != HttpURLConnection.HTTP_PARTIAL) throw IOException("Google's server answered $code to the Android NDK download.")
|
||||
val bytes = connection.inputStream.use { it.readBytes() }
|
||||
if (bytes.size.toLong() != end - start + 1) throw IOException("The Android NDK download was cut short. Try again.")
|
||||
return bytes
|
||||
} finally {
|
||||
connection.disconnect()
|
||||
}
|
||||
}
|
||||
|
||||
/** Translates the disc unless this workspace already holds a translation of the same inputs. */
|
||||
fun translate(tools: ToolProcess, identity: String): String? {
|
||||
val generated = File(workspace, "generated")
|
||||
val provenance = File(generated, "translation-provenance.txt")
|
||||
val expected = "$identity ${BuildConfig.DISC_DOL_SHA256} ${BuildConfig.DISC_REL_SHA256}"
|
||||
val shards = File(generated, "build_shards/shards.cmake")
|
||||
if (provenance.isFile && provenance.readText() == expected && shards.isFile) {
|
||||
log.line("Reusing the translation of an earlier build")
|
||||
report(400, Step.Translate)
|
||||
return null
|
||||
}
|
||||
provenance.delete()
|
||||
val project = File(workspace, "projects/mkwii")
|
||||
File(kit, "translation/projects/mkwii").copyRecursively(project, overwrite = true)
|
||||
File(workspace, "runtime").deleteRecursively()
|
||||
File(kit, "translation/runtime").copyRecursively(File(workspace, "runtime"), overwrite = true)
|
||||
File(disc, DiscChecks.DOL_PATH).copyTo(File(workspace, "Assets/main.dol"), overwrite = true)
|
||||
File(disc, DiscChecks.REL_PATH).copyTo(File(workspace, "Assets/StaticR.rel"), overwrite = true)
|
||||
val manifest = "projects/mkwii/recomp.yml"
|
||||
val entryPoint = BuildRecipe.entryPoint(File(workspace, manifest).readText())
|
||||
|
||||
report(50, Step.Translate, 1, 3)
|
||||
val timing = Regex("""\(t=([0-9.]+)s\)""")
|
||||
translator(
|
||||
tools, "translating the game",
|
||||
"translate-recursive", entryPoint, "--project", manifest, "--outdir", "generated/functions",
|
||||
"--output-metadata", "generated/base_translation_output.json",
|
||||
"--production-source-bundle", "generated/base_translation_sources.bin",
|
||||
"--no-function-files", "--prune-stale", "--threads", TRANSLATOR_THREADS.toString(),
|
||||
) { line ->
|
||||
timing.find(line)?.groupValues?.get(1)?.toDoubleOrNull()?.let { seconds ->
|
||||
val fraction = minOf(seconds / EXPECTED_TRANSLATION_SECONDS, 0.97)
|
||||
reporter.update(50 + (320 * fraction).toInt(), Step.Translate, 1, 3)
|
||||
}
|
||||
}?.let { return it }
|
||||
report(370, Step.Translate, 2, 3)
|
||||
translator(tools, "generating the game data", "generate-data-init", "--project", manifest, "--target-os", "android")?.let { return it }
|
||||
report(385, Step.Translate, 3, 3)
|
||||
translator(
|
||||
tools, "preparing the build",
|
||||
"emit-build-shards", "--project", manifest, "--base-metadata", "generated/base_translation_output.json",
|
||||
"--base-functions-dir", "generated/functions", "--native-source-dir", "runtime/src", "--out", "generated/build_shards",
|
||||
)?.let { return it }
|
||||
provenance.writeText(expected)
|
||||
report(400, Step.Translate, 3, 3)
|
||||
return null
|
||||
}
|
||||
|
||||
fun translator(tools: ToolProcess, what: String, vararg arguments: String, onLine: (String) -> Unit = {}): String? {
|
||||
log.line("== translator ${arguments.first()}")
|
||||
val result = tools.run(
|
||||
workspace,
|
||||
File(toolchain, "translator/translator_host"),
|
||||
listOf(File(toolchain, "translator").absolutePath, "Translator.Cli") + arguments,
|
||||
onLine,
|
||||
)
|
||||
if (result.exitCode == 0) return null
|
||||
return if (result.killed) {
|
||||
"The headset ran out of memory while $what. Close other apps, then build again."
|
||||
} else {
|
||||
"The translator failed while $what (exit ${result.exitCode}).\n${result.tail.takeLast(6).joinToString("\n")}"
|
||||
}
|
||||
}
|
||||
|
||||
/** Compiles every generated source; returns their objects by link slot (runtime, product, translated). */
|
||||
fun compile(tools: ToolProcess, recipe: JSONObject, toolchainFingerprint: String): Map<String, List<String>> {
|
||||
val generated = File(workspace, "generated")
|
||||
val shards = File(generated, "build_shards/shards.cmake").readText()
|
||||
val stamp = File(objects, ".stamp")
|
||||
val identity = "${recipe.getString("fingerprint")} $toolchainFingerprint"
|
||||
if (stamp.takeIf { it.isFile }?.readText() != identity) {
|
||||
objects.deleteRecursively()
|
||||
objects.mkdirs()
|
||||
stamp.writeText(identity)
|
||||
}
|
||||
|
||||
val blob = File(generated, "data_sections_init_blobs.S")
|
||||
val elfBlob = File(work, "data_sections_init_blobs_android.S")
|
||||
val blobText = blob.readText()
|
||||
val elfText = BuildRecipe.elfBlobAssembly(blobText)
|
||||
if (!elfBlob.isFile || elfBlob.readText() != elfText) elfBlob.writeText(elfText)
|
||||
|
||||
data class Job(val kind: String, val source: File, val slot: String)
|
||||
val jobs = ArrayList<Job>()
|
||||
jobs += Job("runtime", File(generated, "data_sections_init.cpp"), "runtime")
|
||||
jobs += Job("runtime", File(generated, "guest_symbol_table.cpp"), "runtime")
|
||||
jobs += Job("asm", elfBlob, "runtime")
|
||||
BuildRecipe.sourceList(shards, BuildRecipe.REGISTRATION_LIST).forEach { jobs += Job("product", File(it), "product") }
|
||||
val translated = BuildRecipe.TRANSLATED_LISTS.flatMap { BuildRecipe.sourceList(shards, it) }
|
||||
if (translated.isEmpty()) throw IOException("The translation produced no game code.")
|
||||
translated.forEach { jobs += Job("translated", File(it), "translated") }
|
||||
if (jobs.map { BuildRecipe.objectName(it.source.path) }.toSet().size != jobs.size) {
|
||||
throw IOException("Two generated sources share an object name.")
|
||||
}
|
||||
|
||||
val values = mapOf(
|
||||
"kit" to kit.absolutePath,
|
||||
"sysroot" to File(ndk, "sysroot").absolutePath,
|
||||
"workspace" to workspace.absolutePath,
|
||||
)
|
||||
val compileFlags = recipe.getJSONObject("compile")
|
||||
for (kind in listOf("runtime", "asm", "product", "translated")) {
|
||||
val flags = compileFlags.getJSONArray(kind).let { array -> (0 until array.length()).map { BuildRecipe.expand(array.getString(it), values) } }
|
||||
File(work, "$kind.rsp").writeText(BuildRecipe.responseFile(flags))
|
||||
}
|
||||
|
||||
val clang = File(toolchain, "llvm/bin/clang-21")
|
||||
val resourceDir = File(toolchain, "llvm/lib/clang/21").absolutePath
|
||||
val pending = jobs.filter { job ->
|
||||
val obj = File(objects, BuildRecipe.objectName(job.source.path))
|
||||
!(obj.isFile && obj.length() > 0 && obj.lastModified() >= job.source.lastModified())
|
||||
}
|
||||
val finished = AtomicInteger(jobs.size - pending.size)
|
||||
log.line("== compiling ${pending.size} of ${jobs.size} sources")
|
||||
val pool = Executors.newFixedThreadPool(compileJobs())
|
||||
try {
|
||||
val futures: List<Future<*>> = pending.map { job ->
|
||||
pool.submit {
|
||||
if (failure.get() != null || cancelled()) return@submit
|
||||
val obj = File(objects, BuildRecipe.objectName(job.source.path))
|
||||
val partial = File(objects, obj.name + ".partial")
|
||||
// The blob assembly needs no preprocessing, and preprocessing would make clang
|
||||
// start itself again, which Android does not allow here.
|
||||
val language = if (job.kind == "asm") listOf("--driver-mode=gcc", "-x", "assembler") else listOf("--driver-mode=g++")
|
||||
val result = tools.run(
|
||||
work, clang,
|
||||
language + listOf(
|
||||
"-resource-dir", resourceDir, "@" + File(work, "${job.kind}.rsp").absolutePath,
|
||||
"-c", job.source.absolutePath, "-o", partial.absolutePath,
|
||||
),
|
||||
)
|
||||
if (result.exitCode != 0) {
|
||||
partial.delete()
|
||||
failure.compareAndSet(
|
||||
null,
|
||||
if (result.killed) "The headset ran out of memory while compiling the game. Close other apps, then build again."
|
||||
else "Compiling ${job.source.name} failed.\n${result.tail.takeLast(6).joinToString("\n")}",
|
||||
)
|
||||
return@submit
|
||||
}
|
||||
if (!partial.renameTo(obj)) failure.compareAndSet(null, "Could not store ${obj.name}.")
|
||||
finished.incrementAndGet()
|
||||
}
|
||||
}
|
||||
while (futures.any { !it.isDone }) {
|
||||
report(400 + 570 * finished.get() / jobs.size, Step.Compile, finished.get(), jobs.size)
|
||||
TimeUnit.MILLISECONDS.sleep(500)
|
||||
}
|
||||
futures.forEach { future ->
|
||||
try {
|
||||
future.get()
|
||||
} catch (e: java.util.concurrent.ExecutionException) {
|
||||
val cause = e.cause
|
||||
// A failed compile stops the others, which is not the player cancelling.
|
||||
if (cause is InterruptedIOException && failure.get() == null) throw cause
|
||||
failure.compareAndSet(null, cause?.message ?: cause.toString())
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
pool.shutdownNow()
|
||||
}
|
||||
if (failure.get() == null) report(970, Step.Compile, jobs.size, jobs.size)
|
||||
return jobs.groupBy({ it.slot }, { File(objects, BuildRecipe.objectName(it.source.path)).absolutePath })
|
||||
}
|
||||
|
||||
fun compileJobs(): Int {
|
||||
val memory = ActivityManager.MemoryInfo()
|
||||
context.getSystemService(ActivityManager::class.java)?.getMemoryInfo(memory)
|
||||
val byMemory = (memory.availMem / COMPILE_MEMORY_BYTES).toInt()
|
||||
return minOf(4, Runtime.getRuntime().availableProcessors(), byMemory).coerceAtLeast(1)
|
||||
}
|
||||
|
||||
fun link(tools: ToolProcess, recipe: JSONObject, objectsBySlot: Map<String, List<String>>): File? {
|
||||
report(970, Step.Link)
|
||||
val library = File(work, recipe.getString("output"))
|
||||
library.delete()
|
||||
val template = recipe.getJSONObject("link").getJSONArray("lld").let { array -> (0 until array.length()).map { array.getString(it) } }
|
||||
val arguments = BuildRecipe.linkArguments(
|
||||
template,
|
||||
mapOf("kit" to kit.absolutePath, "ndk" to ndk.absolutePath, "output" to library.absolutePath),
|
||||
objectsBySlot,
|
||||
)
|
||||
val rsp = File(work, "link.rsp").apply { writeText(BuildRecipe.responseFile(arguments)) }
|
||||
log.line("== linking")
|
||||
val result = tools.run(work, File(toolchain, "llvm/bin/ld.lld"), listOf("@" + rsp.absolutePath))
|
||||
if (result.exitCode != 0 || !library.isFile) {
|
||||
failure.compareAndSet(null, "Linking the game failed.\n${result.tail.takeLast(6).joinToString("\n")}")
|
||||
return null
|
||||
}
|
||||
return library
|
||||
}
|
||||
|
||||
fun install(library: File, kitFingerprint: String): String? {
|
||||
val destination = GameLibrary.directory(context)
|
||||
val staging = File(destination.parentFile, "${destination.name}.building")
|
||||
staging.deleteRecursively()
|
||||
staging.mkdirs()
|
||||
try {
|
||||
val installed = File(staging, GameLibrary.LIBRARY_NAME)
|
||||
library.copyTo(installed)
|
||||
val digest = MessageDigest.getInstance("SHA-256")
|
||||
installed.inputStream().use { input ->
|
||||
val buffer = ByteArray(1 shl 20)
|
||||
while (true) {
|
||||
val read = input.read(buffer)
|
||||
if (read < 0) break
|
||||
digest.update(buffer, 0, read)
|
||||
}
|
||||
}
|
||||
val builtAt = SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss'Z'", Locale.US).apply { timeZone = TimeZone.getTimeZone("UTC") }.format(Date())
|
||||
val manifest = JSONObject()
|
||||
.put("schema", 1)
|
||||
.put("profile", BuildConfig.PROFILE)
|
||||
.put("gameId", BuildConfig.DISC_GAME_ID)
|
||||
.put("dolSha256", BuildConfig.DISC_DOL_SHA256)
|
||||
.put("relSha256", BuildConfig.DISC_REL_SHA256)
|
||||
.put("kitFingerprint", kitFingerprint)
|
||||
.put("library", GameLibrary.LIBRARY_NAME)
|
||||
.put("librarySha256", BuildRecipe.hex(digest.digest()))
|
||||
.put("includesData", false)
|
||||
.put("builtBy", "this headset (WiiCompiled Quest ${BuildConfig.VERSION_NAME})")
|
||||
.put("builtAt", builtAt)
|
||||
File(staging, GameLibrary.MANIFEST_NAME).writeText(manifest.toString(2))
|
||||
destination.parentFile?.mkdirs()
|
||||
return GameFiles.replace(destination, staging)
|
||||
} finally {
|
||||
staging.deleteRecursively()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -15,18 +15,27 @@ import java.io.InterruptedIOException
|
||||
* - [Task.ExtractDisc]: the player's disc image becomes DATA ([DiscExtraction]).
|
||||
* - [Task.ImportPackage]: a .wcgame built on a PC becomes the game library, and DATA too when
|
||||
* the package carries it ([GamePackageImport]).
|
||||
* - [Task.BuildGame]: the game library is built on the headset from DATA ([GameBuild]).
|
||||
*
|
||||
* Every task stages its output next to the destination and swaps it in only after it has been
|
||||
* checked, so a failed, cancelled or killed run never costs working files.
|
||||
*/
|
||||
object GameSetup {
|
||||
|
||||
enum class Task { ExtractDisc, ImportPackage }
|
||||
enum class Task { ExtractDisc, ImportPackage, BuildGame }
|
||||
|
||||
sealed interface State {
|
||||
data object Idle : State
|
||||
data class Checking(val task: Task) : State
|
||||
data class Working(val task: Task, val done: Long, val total: Long) : State
|
||||
/** A build also reports its [step] and how many of that step's items are [stepDone]. */
|
||||
data class Working(
|
||||
val task: Task,
|
||||
val done: Long,
|
||||
val total: Long,
|
||||
val step: GameBuild.Step? = null,
|
||||
val stepDone: Int = 0,
|
||||
val stepTotal: Int = 0,
|
||||
) : State
|
||||
data class Finishing(val task: Task) : State
|
||||
data class Done(val task: Task) : State
|
||||
data class Failed(val task: Task, val message: String) : State
|
||||
@@ -77,25 +86,33 @@ object GameSetup {
|
||||
}
|
||||
}
|
||||
|
||||
/** The whole task, on a worker thread. Always ends in Done, Failed or Cancelled. */
|
||||
fun run(context: Context, task: Task, uri: Uri, deleteSource: Boolean) {
|
||||
/** The whole task, on a worker thread. Always ends in Done, Failed or Cancelled. [uri] is null only for a build. */
|
||||
fun run(context: Context, task: Task, uri: Uri?, deleteSource: Boolean) {
|
||||
val progress = Progress { done, total ->
|
||||
publish(State.Working(task, done, total))
|
||||
!cancelRequested
|
||||
}
|
||||
val finishing = { publish(State.Finishing(task)) }
|
||||
try {
|
||||
val failure = context.contentResolver.openFileDescriptor(uri, "r")?.use { descriptor ->
|
||||
when (task) {
|
||||
Task.ExtractDisc -> DiscExtraction.run(context, descriptor.fd, displayName(context, uri), progress, finishing)
|
||||
Task.ImportPackage -> GamePackageImport.run(context, descriptor, progress, finishing)
|
||||
val failure = if (task == Task.BuildGame) {
|
||||
val reporter = GameBuild.Reporter { permille, step, done, total ->
|
||||
publish(State.Working(task, permille.toLong(), 1000, step, done, total))
|
||||
!cancelRequested
|
||||
}
|
||||
} ?: "The selected file could not be opened."
|
||||
GameBuild.run(context, reporter, cancelled = { cancelRequested }, finishing)
|
||||
} else {
|
||||
uri?.let { context.contentResolver.openFileDescriptor(it, "r") }?.use { descriptor ->
|
||||
when (task) {
|
||||
Task.ExtractDisc -> DiscExtraction.run(context, descriptor.fd, displayName(context, uri), progress, finishing)
|
||||
else -> GamePackageImport.run(context, descriptor, progress, finishing)
|
||||
}
|
||||
} ?: "The selected file could not be opened."
|
||||
}
|
||||
if (failure != null) {
|
||||
publish(State.Failed(task, failure))
|
||||
return
|
||||
}
|
||||
if (deleteSource && uri.scheme == "file") {
|
||||
if (deleteSource && uri?.scheme == "file") {
|
||||
uri.path?.let { File(it).delete() }
|
||||
}
|
||||
publish(State.Done(task))
|
||||
|
||||
@@ -17,8 +17,8 @@ import org.wiicompiled.quest.R
|
||||
|
||||
/**
|
||||
* Runs a [GameSetup] task as a foreground service. Extracting a disc or importing a game takes
|
||||
* minutes, and a panel app's process is otherwise fair game once the panel is closed; taking the
|
||||
* headset off also sleeps the CPU, hence the partial wake lock.
|
||||
* minutes and building one half an hour, and a panel app's process is otherwise fair game once the
|
||||
* panel is closed; taking the headset off also sleeps the CPU, hence the partial wake lock.
|
||||
*/
|
||||
class GameSetupService : Service() {
|
||||
|
||||
@@ -41,7 +41,7 @@ class GameSetupService : Service() {
|
||||
?: GameSetup.Task.ExtractDisc
|
||||
startForeground(NOTIFICATION_ID, notification(GameSetup.State.Checking(task)), ServiceInfo.FOREGROUND_SERVICE_TYPE_DATA_SYNC)
|
||||
val uri = intent?.data
|
||||
if (uri == null || !GameSetup.begin(task)) {
|
||||
if ((uri == null && task != GameSetup.Task.BuildGame) || !GameSetup.begin(task)) {
|
||||
// A task already going keeps the service, and ends it itself.
|
||||
if (!GameSetup.isRunning) {
|
||||
stopForeground(STOP_FOREGROUND_REMOVE)
|
||||
@@ -50,16 +50,16 @@ class GameSetupService : Service() {
|
||||
return START_NOT_STICKY
|
||||
}
|
||||
GameSetup.addListener(listener)
|
||||
val deleteSource = intent.getBooleanExtra(EXTRA_DELETE_SOURCE, false)
|
||||
val deleteSource = intent?.getBooleanExtra(EXTRA_DELETE_SOURCE, false) ?: false
|
||||
|
||||
val wakeLock = getSystemService(PowerManager::class.java)
|
||||
.newWakeLock(PowerManager.PARTIAL_WAKE_LOCK, "WiiCompiled:GameSetup")
|
||||
wakeLock.acquire(WAKE_LOCK_TIMEOUT_MS)
|
||||
wakeLock.acquire(if (task == GameSetup.Task.BuildGame) BUILD_WAKE_LOCK_TIMEOUT_MS else WAKE_LOCK_TIMEOUT_MS)
|
||||
thread(name = "GameSetup") {
|
||||
try {
|
||||
GameSetup.run(applicationContext, task, uri, deleteSource)
|
||||
} finally {
|
||||
releaseReadPermission(uri)
|
||||
uri?.let(::releaseReadPermission)
|
||||
if (wakeLock.isHeld) wakeLock.release()
|
||||
mainExecutor.execute {
|
||||
GameSetup.removeListener(listener)
|
||||
@@ -88,17 +88,23 @@ class GameSetupService : Service() {
|
||||
}
|
||||
|
||||
private fun notification(state: GameSetup.State): Notification {
|
||||
val building = state is GameSetup.State.Checking && state.task == GameSetup.Task.BuildGame ||
|
||||
state is GameSetup.State.Working && state.task == GameSetup.Task.BuildGame
|
||||
val builder = Notification.Builder(this, CHANNEL_ID)
|
||||
.setSmallIcon(R.drawable.ic_wheel)
|
||||
.setContentTitle(getString(R.string.disc_setup_notification_title))
|
||||
.setContentTitle(getString(if (building) R.string.build_notification_title else R.string.disc_setup_notification_title))
|
||||
.setOngoing(true)
|
||||
.setOnlyAlertOnce(true)
|
||||
if (state is GameSetup.State.Working) {
|
||||
val percent = if (state.total > 0) (state.done * 100 / state.total).toInt() else 0
|
||||
val text = if (state.task == GameSetup.Task.ImportPackage) R.string.disc_setup_importing else R.string.disc_setup_extracting
|
||||
val text = when (state.task) {
|
||||
GameSetup.Task.ImportPackage -> R.string.disc_setup_importing
|
||||
GameSetup.Task.BuildGame -> R.string.disc_setup_building
|
||||
GameSetup.Task.ExtractDisc -> R.string.disc_setup_extracting
|
||||
}
|
||||
builder.setContentText(getString(text, percent)).setProgress(100, percent, false)
|
||||
} else {
|
||||
builder.setContentText(getString(R.string.disc_setup_checking)).setProgress(0, 0, true)
|
||||
builder.setContentText(getString(if (building) R.string.build_preparing else R.string.disc_setup_checking)).setProgress(0, 0, true)
|
||||
}
|
||||
return builder.build()
|
||||
}
|
||||
@@ -112,11 +118,18 @@ class GameSetupService : Service() {
|
||||
private const val EXTRA_DELETE_SOURCE = "deleteSource"
|
||||
// Longer than any real task; only a hung run would reach it.
|
||||
private const val WAKE_LOCK_TIMEOUT_MS = 60L * 60 * 1000
|
||||
private const val BUILD_WAKE_LOCK_TIMEOUT_MS = 3L * 60 * 60 * 1000
|
||||
|
||||
/**
|
||||
* Starts [task] on [uri]: a document the player picked, or a file in the Import folder,
|
||||
* which [deleteSource] removes once it has been imported.
|
||||
*/
|
||||
/** Builds the game from DATA on this headset ([GameBuild]). */
|
||||
fun startBuild(context: Context) {
|
||||
val intent = Intent(context, GameSetupService::class.java).putExtra(EXTRA_TASK, GameSetup.Task.BuildGame.name)
|
||||
context.startForegroundService(intent)
|
||||
}
|
||||
|
||||
fun start(context: Context, task: GameSetup.Task, uri: Uri, deleteSource: Boolean = false) {
|
||||
if (uri.scheme == "content") {
|
||||
try {
|
||||
|
||||
@@ -27,8 +27,8 @@ import org.wiicompiled.quest.R
|
||||
*
|
||||
* The APK carries no game code. Playing needs two things the player owns: the game files (DATA,
|
||||
* extracted from their disc image here or on a PC) and the game itself (libmain.so, built from
|
||||
* their disc on a PC and brought over with Import from computer). Home's main button is always
|
||||
* the next of those steps, and Play once both are there.
|
||||
* their disc on this headset, or on a PC and brought over with Import from computer). Home's main
|
||||
* button is always the next of those steps, and Play once both are there.
|
||||
*
|
||||
* The game runs as [QuestActivity], an immersive activity in its own `:game` process. SDL and the
|
||||
* runtime cannot start twice in one process, so that process ends with every game session, and
|
||||
@@ -39,7 +39,7 @@ class LauncherActivity : Activity() {
|
||||
private enum class Page { Home, Settings }
|
||||
|
||||
/** What Home's main and secondary buttons do. */
|
||||
private enum class Action { Play, Resume, SelectDisc, ImportGame }
|
||||
private enum class Action { Play, Resume, SelectDisc, ImportGame, BuildGame }
|
||||
|
||||
private lateinit var navHome: View
|
||||
private lateinit var navSettings: View
|
||||
@@ -111,6 +111,7 @@ class LauncherActivity : Activity() {
|
||||
gameRunning = ::isGameRunning,
|
||||
selectDiscImage = ::selectDiscImage,
|
||||
importGame = ::importGame,
|
||||
buildGame = ::buildGame,
|
||||
)
|
||||
savedInstanceState?.getString(KEY_TAB)?.let { name ->
|
||||
SettingsPage.Tab.entries.firstOrNull { it.name == name }?.let(settings::select)
|
||||
@@ -124,6 +125,14 @@ class LauncherActivity : Activity() {
|
||||
|
||||
val restored = savedInstanceState?.getString(KEY_PAGE)?.let { name -> Page.entries.firstOrNull { it.name == name } }
|
||||
showPage(restored ?: Page.Home)
|
||||
|
||||
// Unattended headset tests (docs/quest-port.md): debug builds start a build from adb.
|
||||
if (BuildConfig.DEBUG && savedInstanceState == null && intent.getBooleanExtra(EXTRA_DEBUG_BUILD_GAME, false) &&
|
||||
!GameSetup.isRunning && GameStorage.discStatus(this) == GameStorage.DiscStatus.Ready
|
||||
) {
|
||||
Log.i(TAG, "Starting a game build requested over adb")
|
||||
GameSetupService.startBuild(this)
|
||||
}
|
||||
}
|
||||
|
||||
override fun onResume() {
|
||||
@@ -193,27 +202,42 @@ class LauncherActivity : Activity() {
|
||||
val running = isGameRunning()
|
||||
val setup = GameSetup.state
|
||||
val settingUp = GameSetup.isRunning
|
||||
val importing = setupTask(setup) == GameSetup.Task.ImportPackage
|
||||
val task = setupTask(setup)
|
||||
val importing = task == GameSetup.Task.ImportPackage
|
||||
val building = task == GameSetup.Task.BuildGame
|
||||
|
||||
// Without anything yet, a player with only a headset starts from their disc image, and
|
||||
// builds the game once its files are there; a PC-built game can always be imported instead.
|
||||
mainAction = when {
|
||||
running -> Action.Resume
|
||||
gameStatus != GameLibrary.Status.Ready -> Action.ImportGame
|
||||
discStatus != GameStorage.DiscStatus.Ready -> Action.SelectDisc
|
||||
gameStatus != GameLibrary.Status.Ready -> Action.BuildGame
|
||||
else -> Action.Play
|
||||
}
|
||||
// Without anything yet, a player with only a headset starts from their disc image.
|
||||
secondaryAction = when {
|
||||
settingUp || running -> null
|
||||
mainAction == Action.ImportGame && discStatus != GameStorage.DiscStatus.Ready -> Action.SelectDisc
|
||||
mainAction == Action.SelectDisc -> Action.ImportGame
|
||||
gameStatus != GameLibrary.Status.Ready || mainAction == Action.SelectDisc -> Action.ImportGame
|
||||
else -> null
|
||||
}
|
||||
|
||||
playButton.isEnabled = !settingUp
|
||||
playIcon.setImageResource(if (!settingUp && (mainAction == Action.Play || mainAction == Action.Resume)) R.drawable.ic_play else R.drawable.ic_disc)
|
||||
playText.text = when {
|
||||
setup is GameSetup.State.Checking -> getString(if (importing) R.string.home_checking_package else R.string.home_checking)
|
||||
setup is GameSetup.State.Working -> getString(if (importing) R.string.home_importing else R.string.home_extracting, percent(setup))
|
||||
setup is GameSetup.State.Checking -> getString(
|
||||
when {
|
||||
building -> R.string.home_build_preparing
|
||||
importing -> R.string.home_checking_package
|
||||
else -> R.string.home_checking
|
||||
},
|
||||
)
|
||||
setup is GameSetup.State.Working -> getString(
|
||||
when {
|
||||
building -> R.string.home_building
|
||||
importing -> R.string.home_importing
|
||||
else -> R.string.home_extracting
|
||||
},
|
||||
percent(setup),
|
||||
)
|
||||
setup is GameSetup.State.Finishing -> getString(R.string.home_finishing)
|
||||
else -> getString(label(mainAction))
|
||||
}
|
||||
@@ -228,22 +252,31 @@ class LauncherActivity : Activity() {
|
||||
cancel.visibility = if (settingUp && setup !is GameSetup.State.Finishing) View.VISIBLE else View.GONE
|
||||
|
||||
homeStatus.text = when {
|
||||
setup is GameSetup.State.Working && building -> buildStatus(setup)
|
||||
setup is GameSetup.State.Working -> getString(
|
||||
R.string.home_extract_progress,
|
||||
Formatter.formatShortFileSize(this, setup.done),
|
||||
Formatter.formatShortFileSize(this, setup.total),
|
||||
)
|
||||
setup is GameSetup.State.Checking -> getString(if (importing) R.string.home_checking_package_status else R.string.home_checking_status)
|
||||
setup is GameSetup.State.Checking -> getString(
|
||||
when {
|
||||
building -> R.string.build_step_prepare
|
||||
importing -> R.string.home_checking_package_status
|
||||
else -> R.string.home_checking_status
|
||||
},
|
||||
)
|
||||
settingUp || setup is GameSetup.State.Failed -> ""
|
||||
setup is GameSetup.State.Cancelled -> getString(R.string.home_setup_cancelled)
|
||||
setup is GameSetup.State.Cancelled -> getString(if (building) R.string.home_build_cancelled else R.string.home_setup_cancelled)
|
||||
running -> getString(R.string.home_running)
|
||||
mainAction == Action.Play && setup is GameSetup.State.Done -> getString(R.string.home_setup_done)
|
||||
mainAction == Action.Play && setup is GameSetup.State.Done -> getString(if (building) R.string.home_build_done else R.string.home_setup_done)
|
||||
mainAction == Action.Play -> getString(R.string.home_put_on_headset)
|
||||
else -> ""
|
||||
}
|
||||
|
||||
when {
|
||||
settingUp && building -> showBanner(getString(R.string.home_build_running), warning = false)
|
||||
settingUp -> showBanner(null)
|
||||
setup is GameSetup.State.Failed && building -> showBanner(getString(R.string.home_build_failed, setup.message), warning = true)
|
||||
setup is GameSetup.State.Failed -> showBanner(getString(R.string.home_setup_failed, setup.message), warning = true)
|
||||
discStatus == GameStorage.DiscStatus.Incomplete -> showBanner(getString(R.string.home_data_incomplete, disc), warning = true)
|
||||
gameStatus == GameLibrary.Status.Stale -> showBanner(getString(R.string.home_game_stale), warning = true)
|
||||
@@ -265,6 +298,7 @@ class LauncherActivity : Activity() {
|
||||
Action.Resume -> R.string.home_resume
|
||||
Action.SelectDisc -> R.string.home_select_disc
|
||||
Action.ImportGame -> R.string.home_import
|
||||
Action.BuildGame -> R.string.home_build
|
||||
}
|
||||
|
||||
private fun perform(action: Action) {
|
||||
@@ -273,6 +307,26 @@ class LauncherActivity : Activity() {
|
||||
Action.Play, Action.Resume -> play()
|
||||
Action.SelectDisc -> selectDiscImage()
|
||||
Action.ImportGame -> importGame()
|
||||
Action.BuildGame -> buildGame()
|
||||
}
|
||||
}
|
||||
|
||||
private fun buildStatus(state: GameSetup.State.Working): String = when (state.step) {
|
||||
GameBuild.Step.Download -> getString(R.string.build_step_download, state.stepDone, state.stepTotal)
|
||||
GameBuild.Step.Translate -> getString(R.string.build_step_translate, state.stepDone, state.stepTotal)
|
||||
GameBuild.Step.Compile -> getString(R.string.build_step_compile, state.stepDone, state.stepTotal)
|
||||
GameBuild.Step.Link -> getString(R.string.build_step_link)
|
||||
GameBuild.Step.Install -> getString(R.string.build_step_install)
|
||||
GameBuild.Step.Prepare, null -> getString(R.string.build_step_prepare)
|
||||
}
|
||||
|
||||
/** Builds the game on this headset from DATA, after saying what that takes. */
|
||||
private fun buildGame() {
|
||||
if (GameSetup.isRunning || GameStorage.discStatus(this) != GameStorage.DiscStatus.Ready) return
|
||||
val message = if (GameLibrary.status(this) == GameLibrary.Status.Ready) R.string.home_build_replace_message else R.string.home_build_message
|
||||
confirm(R.string.home_build_title, message, R.string.home_build) {
|
||||
GameSetupService.startBuild(this)
|
||||
showPage(Page.Home)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -403,5 +457,6 @@ class LauncherActivity : Activity() {
|
||||
const val REQUEST_GAME_PACKAGE = 2
|
||||
const val PREFERENCES = "launcher"
|
||||
const val KEY_LAST_DROPPED_IMPORT = "lastDroppedImport"
|
||||
const val EXTRA_DEBUG_BUILD_GAME = "org.wiicompiled.quest.debug.BUILD_GAME"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,126 @@
|
||||
package org.wiicompiled.quest.launcher
|
||||
|
||||
import java.io.ByteArrayOutputStream
|
||||
import java.io.IOException
|
||||
import java.nio.ByteBuffer
|
||||
import java.nio.ByteOrder
|
||||
import java.util.zip.Inflater
|
||||
|
||||
/**
|
||||
* Reads a few entries of a large zip on a web server without downloading the rest: the central
|
||||
* directory from the end of the file, then only the wanted entries, neighbouring ones in one
|
||||
* request. [fetch] returns the bytes from start to endInclusive (an HTTP range request).
|
||||
*
|
||||
* Only what the Android NDK zip needs: no ZIP64, stored or deflated entries.
|
||||
*/
|
||||
class RemoteZip(private val size: Long, private val fetch: (start: Long, endInclusive: Long) -> ByteArray) {
|
||||
|
||||
/** [end] is where the next entry (or the central directory) starts, which bounds this one's bytes. */
|
||||
data class Entry(val name: String, val method: Int, val compressedSize: Long, val size: Long, val offset: Long, val end: Long)
|
||||
|
||||
fun entries(): List<Entry> {
|
||||
val tailStart = maxOf(0L, size - EOCD_SEARCH)
|
||||
val tail = fetch(tailStart, size - 1)
|
||||
var eocd = -1
|
||||
for (i in tail.size - EOCD_SIZE downTo 0) {
|
||||
if (u32(tail, i) == EOCD_SIGNATURE) { eocd = i; break }
|
||||
}
|
||||
if (eocd < 0) throw IOException("No end of central directory record")
|
||||
val count = u16(tail, eocd + 10)
|
||||
val directorySize = u32(tail, eocd + 12)
|
||||
val directoryOffset = u32(tail, eocd + 16)
|
||||
if (count == 0xFFFF || directoryOffset == 0xFFFFFFFFL) throw IOException("ZIP64 archives are not supported")
|
||||
val directory = fetch(directoryOffset, directoryOffset + directorySize - 1)
|
||||
|
||||
data class Raw(val name: String, val method: Int, val compressedSize: Long, val size: Long, val offset: Long)
|
||||
val raw = ArrayList<Raw>(count)
|
||||
var at = 0
|
||||
repeat(count) {
|
||||
if (u32(directory, at) != CENTRAL_SIGNATURE) throw IOException("Corrupt central directory")
|
||||
val nameLength = u16(directory, at + 28)
|
||||
val extraLength = u16(directory, at + 30)
|
||||
val commentLength = u16(directory, at + 32)
|
||||
raw += Raw(
|
||||
name = String(directory, at + 46, nameLength, Charsets.UTF_8),
|
||||
method = u16(directory, at + 10),
|
||||
compressedSize = u32(directory, at + 20),
|
||||
size = u32(directory, at + 24),
|
||||
offset = u32(directory, at + 42),
|
||||
)
|
||||
at += 46 + nameLength + extraLength + commentLength
|
||||
}
|
||||
val sorted = raw.sortedBy { it.offset }
|
||||
return sorted.mapIndexed { i, entry ->
|
||||
val end = if (i + 1 < sorted.size) sorted[i + 1].offset else directoryOffset
|
||||
Entry(entry.name, entry.method, entry.compressedSize, entry.size, entry.offset, end)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Calls [onEntry] with each of [wanted]'s contents, in offset order. Entries less than
|
||||
* [maxGap] apart are fetched in one request of at most [maxRequest] bytes.
|
||||
*/
|
||||
fun read(wanted: List<Entry>, maxGap: Long = 1L shl 20, maxRequest: Long = 16L shl 20, onEntry: (Entry, ByteArray) -> Unit) {
|
||||
val entries = wanted.sortedBy { it.offset }
|
||||
var first = 0
|
||||
while (first < entries.size) {
|
||||
var last = first
|
||||
while (last + 1 < entries.size &&
|
||||
entries[last + 1].offset - entries[last].end < maxGap &&
|
||||
entries[last + 1].end - entries[first].offset <= maxRequest
|
||||
) {
|
||||
last++
|
||||
}
|
||||
val start = entries[first].offset
|
||||
val chunk = fetch(start, entries[last].end - 1)
|
||||
for (i in first..last) {
|
||||
val entry = entries[i]
|
||||
val local = (entry.offset - start).toInt()
|
||||
if (u32(chunk, local) != LOCAL_SIGNATURE) throw IOException("Corrupt entry ${entry.name}")
|
||||
val dataStart = local + 30 + u16(chunk, local + 26) + u16(chunk, local + 28)
|
||||
onEntry(entry, content(entry, chunk, dataStart))
|
||||
}
|
||||
first = last + 1
|
||||
}
|
||||
}
|
||||
|
||||
private fun content(entry: Entry, chunk: ByteArray, dataStart: Int): ByteArray {
|
||||
if (dataStart + entry.compressedSize > chunk.size) throw IOException("Truncated entry ${entry.name}")
|
||||
val bytes = when (entry.method) {
|
||||
0 -> chunk.copyOfRange(dataStart, dataStart + entry.compressedSize.toInt())
|
||||
8 -> {
|
||||
val inflater = Inflater(true)
|
||||
try {
|
||||
inflater.setInput(chunk, dataStart, entry.compressedSize.toInt())
|
||||
val out = ByteArrayOutputStream(entry.size.toInt())
|
||||
val buffer = ByteArray(64 * 1024)
|
||||
while (!inflater.finished()) {
|
||||
val n = inflater.inflate(buffer)
|
||||
if (n == 0 && (inflater.needsInput() || inflater.needsDictionary())) break
|
||||
out.write(buffer, 0, n)
|
||||
}
|
||||
out.toByteArray()
|
||||
} finally {
|
||||
inflater.end()
|
||||
}
|
||||
}
|
||||
else -> throw IOException("Entry ${entry.name} uses compression method ${entry.method}")
|
||||
}
|
||||
if (bytes.size.toLong() != entry.size) throw IOException("Entry ${entry.name} has the wrong size")
|
||||
return bytes
|
||||
}
|
||||
|
||||
private fun u16(bytes: ByteArray, at: Int): Int =
|
||||
ByteBuffer.wrap(bytes, at, 2).order(ByteOrder.LITTLE_ENDIAN).short.toInt() and 0xFFFF
|
||||
|
||||
private fun u32(bytes: ByteArray, at: Int): Long =
|
||||
ByteBuffer.wrap(bytes, at, 4).order(ByteOrder.LITTLE_ENDIAN).int.toLong() and 0xFFFFFFFFL
|
||||
|
||||
private companion object {
|
||||
const val EOCD_SIGNATURE = 0x06054b50L
|
||||
const val CENTRAL_SIGNATURE = 0x02014b50L
|
||||
const val LOCAL_SIGNATURE = 0x04034b50L
|
||||
const val EOCD_SIZE = 22
|
||||
const val EOCD_SEARCH = EOCD_SIZE + 0xFFFFL
|
||||
}
|
||||
}
|
||||
@@ -38,6 +38,7 @@ class SettingsPage(
|
||||
private val gameRunning: () -> Boolean,
|
||||
private val selectDiscImage: () -> Unit,
|
||||
private val importGame: () -> Unit,
|
||||
private val buildGame: () -> Unit,
|
||||
) {
|
||||
|
||||
enum class Tab(val label: Int) {
|
||||
@@ -281,6 +282,12 @@ class SettingsPage(
|
||||
GameLibrary.Status.Missing -> activity.getString(R.string.about_game_missing)
|
||||
}
|
||||
info(R.string.about_game, gameText, stacked = true)
|
||||
action(
|
||||
R.string.home_build, R.string.about_build_helper, R.string.home_build,
|
||||
enabled = !GameSetup.isRunning && GameStorage.discStatus(activity) == GameStorage.DiscStatus.Ready,
|
||||
) {
|
||||
buildGame()
|
||||
}
|
||||
action(R.string.home_import, R.string.about_import_helper, R.string.home_import, enabled = !GameSetup.isRunning) {
|
||||
importGame()
|
||||
}
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
package org.wiicompiled.quest.launcher
|
||||
|
||||
import java.io.File
|
||||
import java.io.InterruptedIOException
|
||||
import java.util.concurrent.TimeUnit
|
||||
import kotlin.concurrent.thread
|
||||
|
||||
/**
|
||||
* Runs a build tool from the app's private storage. Android refuses to exec files there, but lets
|
||||
* the system linker load them (the same permission that lets the game load libmain.so), so every
|
||||
* tool starts as `/system/bin/linker64 <absolute path> <arguments>`.
|
||||
*
|
||||
* Output goes to [log] and to the caller line by line. [cancelled] is polled while the tool runs;
|
||||
* when it turns true the tool is stopped and [run] throws [InterruptedIOException].
|
||||
*/
|
||||
class ToolProcess(
|
||||
private val environment: Map<String, String>,
|
||||
private val log: BuildLog,
|
||||
private val cancelled: () -> Boolean,
|
||||
) {
|
||||
|
||||
class Result(val exitCode: Int, val tail: List<String>) {
|
||||
/** Android's low memory killer ends a process with SIGKILL, which reads as 128 + 9. */
|
||||
val killed: Boolean get() = exitCode == 137
|
||||
}
|
||||
|
||||
fun run(workingDirectory: File, executable: File, arguments: List<String>, onLine: (String) -> Unit = {}): Result {
|
||||
val command = listOf(LINKER, executable.absolutePath) + arguments
|
||||
val process = ProcessBuilder(command)
|
||||
.directory(workingDirectory)
|
||||
.redirectErrorStream(true)
|
||||
.apply { environment().putAll(this@ToolProcess.environment) }
|
||||
.start()
|
||||
val tail = ArrayDeque<String>()
|
||||
val reader = thread(name = "BuildOutput") {
|
||||
process.inputStream.bufferedReader().useLines { lines ->
|
||||
for (line in lines) {
|
||||
log.line(line)
|
||||
synchronized(tail) {
|
||||
tail.addLast(line)
|
||||
if (tail.size > TAIL_LINES) tail.removeFirst()
|
||||
}
|
||||
onLine(line)
|
||||
}
|
||||
}
|
||||
}
|
||||
try {
|
||||
while (!process.waitFor(POLL_MS, TimeUnit.MILLISECONDS)) {
|
||||
if (cancelled()) {
|
||||
stop(process)
|
||||
throw InterruptedIOException("Build cancelled")
|
||||
}
|
||||
}
|
||||
} catch (e: InterruptedException) {
|
||||
stop(process)
|
||||
throw InterruptedIOException("Build interrupted")
|
||||
}
|
||||
reader.join(READER_JOIN_MS)
|
||||
return Result(process.exitValue(), synchronized(tail) { tail.toList() })
|
||||
}
|
||||
|
||||
private fun stop(process: Process) {
|
||||
process.destroy()
|
||||
if (!process.waitFor(STOP_GRACE_MS, TimeUnit.MILLISECONDS)) process.destroyForcibly()
|
||||
}
|
||||
|
||||
private companion object {
|
||||
const val LINKER = "/system/bin/linker64"
|
||||
const val POLL_MS = 250L
|
||||
const val STOP_GRACE_MS = 2000L
|
||||
const val READER_JOIN_MS = 5000L
|
||||
const val TAIL_LINES = 40
|
||||
}
|
||||
}
|
||||
|
||||
/** The build's log file, which several tools write to at once. */
|
||||
class BuildLog(val file: File) {
|
||||
init {
|
||||
file.parentFile?.mkdirs()
|
||||
}
|
||||
|
||||
private val writer = file.bufferedWriter()
|
||||
|
||||
@Synchronized
|
||||
fun line(text: String) {
|
||||
writer.write(text)
|
||||
writer.newLine()
|
||||
writer.flush()
|
||||
}
|
||||
|
||||
@Synchronized
|
||||
fun close() = writer.close()
|
||||
}
|
||||
@@ -21,9 +21,25 @@
|
||||
<string name="home_put_on_headset">The game opens in the headset.</string>
|
||||
<string name="home_data_missing">The game is ready, but its game files (DATA) are missing. Select your own PAL (RMCP01) disc image to extract them, or import a game file from your computer that includes them.\n\nThe game files go to:\n%1$s</string>
|
||||
<string name="home_import">Import from computer</string>
|
||||
<string name="home_setup_intro">Nothing from Mario Kart Wii comes with this app: the game is built from your own PAL (RMCP01) disc.\n\nWith a computer: in the WiiCompiled VR launcher, use Build for Quest with your disc image, copy the .wcgame file it makes to this headset, then press Import from computer.\n\nOn this headset: copy your disc image (ISO, WBFS, RVZ…) here, for example into Download, and press Select disc image.</string>
|
||||
<string name="home_game_missing">The game files are ready. The game itself is built from your disc: in the WiiCompiled VR launcher on your computer, use Build for Quest, copy the .wcgame file to this headset, then press Import from computer.</string>
|
||||
<string name="home_game_stale">This app was updated, so the game must be built again for it: use Build for Quest in the WiiCompiled VR launcher again, then Import from computer.</string>
|
||||
<string name="home_setup_intro">Nothing from Mario Kart Wii comes with this app: the game is built from your own PAL (RMCP01) disc.\n\nOn this headset: copy your disc image (ISO, WBFS, RVZ…) here, for example into Download, press Select disc image, then Build on this Quest.\n\nWith a computer: in the WiiCompiled VR launcher, use Build for Quest with your disc image, copy the .wcgame file it makes to this headset, then press Import from computer.</string>
|
||||
<string name="home_game_missing">The game files are ready. Now build the game from them: Build on this Quest takes about 30–40 minutes on a Quest 3. With a computer, use Build for Quest in the WiiCompiled VR launcher instead, copy the .wcgame file to this headset and press Import from computer.</string>
|
||||
<string name="home_game_stale">This app was updated, so the game must be built again for it: press Build on this Quest, or use Build for Quest in the WiiCompiled VR launcher again and Import from computer.</string>
|
||||
<string name="home_build">Build on this Quest</string>
|
||||
<string name="home_build_preparing">Preparing…</string>
|
||||
<string name="home_building">Building %1$d%%</string>
|
||||
<string name="home_build_title">Build the game on this Quest?</string>
|
||||
<string name="home_build_message">The game is built on this headset from your own disc files. It takes about 30–40 minutes on a Quest 3 and uses most of its memory, so close other apps and keep the headset on its charger.\n\nThe first build downloads about 10 MB of Android build files from Google.</string>
|
||||
<string name="home_build_replace_message">The installed game stays until the new build has finished.\n\nBuilding takes about 30–40 minutes on a Quest 3 and uses most of its memory, so close other apps and keep the headset on its charger.</string>
|
||||
<string name="home_build_running">Building the game from your disc files: about 30–40 minutes on a Quest 3. Keep the headset on its charger. The build carries on if you close this panel.</string>
|
||||
<string name="home_build_done">The game is built.</string>
|
||||
<string name="home_build_cancelled">Build cancelled. Building again picks up where it stopped.</string>
|
||||
<string name="home_build_failed">The game could not be built.\n%1$s</string>
|
||||
<string name="build_step_prepare">Unpacking the build tools</string>
|
||||
<string name="build_step_download">Downloading Android build files (%1$d of %2$d MB)</string>
|
||||
<string name="build_step_translate">Translating the game (step %1$d of %2$d)</string>
|
||||
<string name="build_step_compile">Compiling %1$d of %2$d</string>
|
||||
<string name="build_step_link">Linking the game</string>
|
||||
<string name="build_step_install">Installing the game</string>
|
||||
<string name="home_checking_package">Checking game file…</string>
|
||||
<string name="home_checking_package_status">Checking the game file was built from your disc for this version of the app.</string>
|
||||
<string name="home_importing">Importing %1$d%%</string>
|
||||
@@ -159,6 +175,7 @@
|
||||
<string name="about_game_ready">Built by %1$s on %2$s</string>
|
||||
<string name="about_game_missing">Not installed yet</string>
|
||||
<string name="about_game_stale">Built for another version of this app; build it again</string>
|
||||
<string name="about_build_helper">Builds the game from DATA on this headset: about 30–40 minutes on a Quest 3.</string>
|
||||
<string name="about_import_helper">Installs a .wcgame made by Build for Quest in the WiiCompiled VR launcher on a computer.</string>
|
||||
<string name="section_about_diagnostics">Diagnostics</string>
|
||||
<string name="about_openxr_logging">OpenXR diagnostic logging</string>
|
||||
@@ -169,6 +186,9 @@
|
||||
<string name="disc_setup_notification_title">Setting up the game files</string>
|
||||
<string name="disc_setup_checking">Checking the disc image…</string>
|
||||
<string name="disc_setup_extracting">Extracting: %1$d%%</string>
|
||||
<string name="disc_setup_building">Building: %1$d%%</string>
|
||||
<string name="build_notification_title">Building the game</string>
|
||||
<string name="build_preparing">Preparing the build…</string>
|
||||
<string name="disc_setup_importing">Importing: %1$d%%</string>
|
||||
<string name="game_not_installed">The game is not installed. Open WiiCompiled VR and set it up from Home.</string>
|
||||
</resources>
|
||||
@@ -0,0 +1,97 @@
|
||||
package org.wiicompiled.quest.launcher
|
||||
|
||||
import org.junit.Assert.assertEquals
|
||||
import org.junit.Assert.assertThrows
|
||||
import org.junit.Test
|
||||
|
||||
/** The headset must turn the game kit's recipe into the same commands android/QuestGameKit.psm1 runs. */
|
||||
class BuildRecipeTest {
|
||||
|
||||
@Test
|
||||
fun quotesResponseFileArgumentsForGnuTokenizers() {
|
||||
assertEquals("-DIMGUI_USER_CONFIG=\\\"aurora/imgui_config.h\\\"", BuildRecipe.responseFileArgument("-DIMGUI_USER_CONFIG=\"aurora/imgui_config.h\""))
|
||||
assertEquals("/data/with\\ space/a\\\\b", BuildRecipe.responseFileArgument("/data/with space/a\\b"))
|
||||
assertEquals("-O2\n-c\n", BuildRecipe.responseFile(listOf("-O2", "-c")))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun expandsPlaceholdersAndRejectsUnknownOnes() {
|
||||
val values = mapOf("kit" to "/k", "sysroot" to "/n/sysroot")
|
||||
assertEquals("-I/k/include", BuildRecipe.expand("-I{kit}/include", values))
|
||||
assertEquals("--sysroot=/n/sysroot", BuildRecipe.expand("--sysroot={sysroot}", values))
|
||||
assertThrows(IllegalArgumentException::class.java) { BuildRecipe.expand("-I{workspace}", values) }
|
||||
}
|
||||
|
||||
@Test
|
||||
fun expandsTheLinkSlotsInPlace() {
|
||||
val template = listOf("-o", "{output}", "{kit}/objects/001_a.o", "{game:runtime}", "--start-lib", "{game:translated}", "--end-lib", "-lm")
|
||||
val arguments = BuildRecipe.linkArguments(
|
||||
template,
|
||||
mapOf("kit" to "/k", "output" to "/w/libmain.so"),
|
||||
mapOf("runtime" to listOf("/o/data.o", "/o/blob.o"), "translated" to listOf("/o/s1.o", "/o/s2.o")),
|
||||
)
|
||||
assertEquals(
|
||||
listOf("-o", "/w/libmain.so", "/k/objects/001_a.o", "/o/data.o", "/o/blob.o", "--start-lib", "/o/s1.o", "/o/s2.o", "--end-lib", "-lm"),
|
||||
arguments,
|
||||
)
|
||||
assertThrows(IllegalArgumentException::class.java) {
|
||||
BuildRecipe.linkArguments(listOf("{game:product}"), emptyMap(), emptyMap())
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun readsSourceListsFromShardsCmake() {
|
||||
val shards = """
|
||||
set(MKW_TRANSLATED_SHARD_ROOT "/w/generated/build_shards")
|
||||
set(MKW_BASE_COMMON_SHARDS
|
||||
"/w/generated/build_shards/base_common/shard_1.cpp"
|
||||
"/w/generated/build_shards/base_common/shard_2.cpp"
|
||||
)
|
||||
set(MKW_BASE_PORTABLE_SENSITIVE_SHARDS
|
||||
)
|
||||
set(MKW_BASE_REGISTRATION_SOURCES
|
||||
"C:\w\generated\build_shards\base_registration\registration_00.cpp"
|
||||
)
|
||||
""".trimIndent()
|
||||
assertEquals(
|
||||
listOf("/w/generated/build_shards/base_common/shard_1.cpp", "/w/generated/build_shards/base_common/shard_2.cpp"),
|
||||
BuildRecipe.sourceList(shards, "MKW_BASE_COMMON_SHARDS"),
|
||||
)
|
||||
assertEquals(emptyList<String>(), BuildRecipe.sourceList(shards, "MKW_BASE_PORTABLE_SENSITIVE_SHARDS"))
|
||||
assertEquals(emptyList<String>(), BuildRecipe.sourceList(shards, "MKW_RETRO_MOD_SHARDS"))
|
||||
assertEquals(listOf("C:/w/generated/build_shards/base_registration/registration_00.cpp"), BuildRecipe.sourceList(shards, BuildRecipe.REGISTRATION_LIST))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun readsTheEntryPointFromTheGameManifest() {
|
||||
val manifest = "translation:\n entry_points:\n - 0x800060A4\n function_map:\n path: MAP.txt\n"
|
||||
assertEquals("0x800060A4", BuildRecipe.entryPoint(manifest))
|
||||
assertThrows(IllegalArgumentException::class.java) { BuildRecipe.entryPoint("translation:\n") }
|
||||
}
|
||||
|
||||
@Test
|
||||
fun rewritesWindowsBlobSectionsForElfOnly() {
|
||||
val windows = "// blobs\n.section .rdata,\"dr\"\n.incbin \"a.bin\"\n"
|
||||
assertEquals(
|
||||
"// blobs\n.section .rodata,\"a\",@progbits\n.incbin \"a.bin\"\n\n.section .note.GNU-stack,\"\",@progbits\n",
|
||||
BuildRecipe.elfBlobAssembly(windows),
|
||||
)
|
||||
val elf = "// blobs\n.section .rodata,\"a\",@progbits\n"
|
||||
assertEquals(elf, BuildRecipe.elfBlobAssembly(elf))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun namesObjectsAfterTheSourceStem() {
|
||||
assertEquals("shard_8b98.o", BuildRecipe.objectName("/w/generated/build_shards/base_common/shard_8b98.cpp"))
|
||||
assertEquals("data_sections_init_blobs_android.o", BuildRecipe.objectName("/w/data_sections_init_blobs_android.S"))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun digestsTheFileListLikePrepareQuestToolchain() {
|
||||
// SHA-256 of "a/x.h 11\nb.so 22", the lines sorted by path and joined with \n.
|
||||
assertEquals(
|
||||
java.security.MessageDigest.getInstance("SHA-256").digest("a/x.h 11\nb.so 22".toByteArray()).joinToString("") { "%02x".format(it) },
|
||||
BuildRecipe.listDigest(mapOf("b.so" to "22", "a/x.h" to "11")),
|
||||
)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
package org.wiicompiled.quest.launcher
|
||||
|
||||
import java.io.ByteArrayOutputStream
|
||||
import java.util.zip.CRC32
|
||||
import java.util.zip.ZipEntry
|
||||
import java.util.zip.ZipOutputStream
|
||||
import org.junit.Assert.assertArrayEquals
|
||||
import org.junit.Assert.assertEquals
|
||||
import org.junit.Assert.assertTrue
|
||||
import org.junit.Test
|
||||
|
||||
/** The NDK download reads only the wanted entries of a zip, through range requests. */
|
||||
class RemoteZipTest {
|
||||
|
||||
private val contents = linkedMapOf(
|
||||
"ndk/sysroot/usr/include/stdio.h" to "int printf(const char *, ...);\n".repeat(200).toByteArray(),
|
||||
"ndk/sysroot/usr/lib/aarch64-linux-android/29/libc.so" to ByteArray(70_000) { (it * 31).toByte() },
|
||||
"ndk/prebuilt/bin/clang" to ByteArray(300_000) { (it * 7).toByte() },
|
||||
"ndk/lib/clang/21/lib/linux/libclang_rt.builtins-aarch64-android.a" to "!<arch>\n".toByteArray(),
|
||||
)
|
||||
|
||||
private val zip: ByteArray = ByteArrayOutputStream().also { bytes ->
|
||||
ZipOutputStream(bytes).use { out ->
|
||||
for ((name, data) in contents) {
|
||||
val entry = ZipEntry(name)
|
||||
if (name.endsWith(".a")) {
|
||||
// A stored entry: sizes and CRC must be known up front.
|
||||
entry.method = ZipEntry.STORED
|
||||
entry.size = data.size.toLong()
|
||||
entry.crc = CRC32().apply { update(data) }.value
|
||||
}
|
||||
out.putNextEntry(entry)
|
||||
out.write(data)
|
||||
out.closeEntry()
|
||||
}
|
||||
out.putNextEntry(ZipEntry("ndk/empty-dir/"))
|
||||
out.closeEntry()
|
||||
}
|
||||
}.toByteArray()
|
||||
|
||||
@Test
|
||||
fun readsWantedEntriesWithoutTheRest() {
|
||||
val requests = mutableListOf<LongRange>()
|
||||
val remote = RemoteZip(zip.size.toLong()) { start, end ->
|
||||
requests += start..end
|
||||
zip.copyOfRange(start.toInt(), end.toInt() + 1)
|
||||
}
|
||||
val entries = remote.entries()
|
||||
assertEquals(contents.keys + "ndk/empty-dir/", entries.map { it.name }.toSet())
|
||||
|
||||
val wanted = entries.filter { it.name.contains("sysroot") || it.name.endsWith(".a") }
|
||||
val read = LinkedHashMap<String, ByteArray>()
|
||||
requests.clear()
|
||||
remote.read(wanted, maxGap = 64) { entry, bytes -> read[entry.name] = bytes }
|
||||
|
||||
assertEquals(wanted.map { it.name }.toSet(), read.keys)
|
||||
for ((name, bytes) in read) assertArrayEquals(name, contents.getValue(name), bytes)
|
||||
// The two sysroot entries sit next to each other and share a request; clang, 300 KB, is skipped.
|
||||
assertEquals(2, requests.size)
|
||||
assertTrue(requests.sumOf { it.last - it.first + 1 } < 150_000)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun keepsRequestsWithinTheSizeLimit() {
|
||||
val requests = mutableListOf<LongRange>()
|
||||
val remote = RemoteZip(zip.size.toLong()) { start, end ->
|
||||
requests += start..end
|
||||
zip.copyOfRange(start.toInt(), end.toInt() + 1)
|
||||
}
|
||||
val files = remote.entries().filter { !it.name.endsWith("/") }
|
||||
requests.clear()
|
||||
var count = 0
|
||||
remote.read(files, maxGap = 1L shl 20, maxRequest = 1) { _, _ -> count++ }
|
||||
assertEquals(files.size, count)
|
||||
assertEquals(files.size, requests.size)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
/*
|
||||
* Starts the WiiCompiled translator, a self-contained .NET app, on the headset.
|
||||
*
|
||||
* Android refuses to exec files in an app's private storage, so the on-device builder starts its
|
||||
* tools through /system/bin/linker64. Under the linker, the .NET apphost reads its own location from
|
||||
* /proc/self/exe, gets the linker's, and cannot find the app. This host is told the app's directory
|
||||
* and name instead and hands them to hostfxr exactly as the apphost would:
|
||||
*
|
||||
* linker64 translator_host <app directory> <app name> [app arguments...]
|
||||
*
|
||||
* It also switches off bionic's heap pointer tagging first. The .NET runtime for Android (Mono)
|
||||
* crashes at startup on tagged heap pointers.
|
||||
*/
|
||||
#include <dlfcn.h>
|
||||
#include <limits.h>
|
||||
#include <malloc.h>
|
||||
#include <stdio.h>
|
||||
|
||||
#ifndef M_BIONIC_SET_HEAP_TAGGING_LEVEL
|
||||
#define M_BIONIC_SET_HEAP_TAGGING_LEVEL (-204)
|
||||
#endif
|
||||
#ifndef M_HEAP_TAGGING_LEVEL_NONE
|
||||
#define M_HEAP_TAGGING_LEVEL_NONE 0
|
||||
#endif
|
||||
|
||||
typedef int (*hostfxr_main_startupinfo_fn)(int argc, const char **argv, const char *host_path,
|
||||
const char *dotnet_root, const char *app_path);
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
if (argc < 3) {
|
||||
fprintf(stderr, "usage: translator_host <app directory> <app name> [arguments...]\n");
|
||||
return 2;
|
||||
}
|
||||
/* Only lowering the level is allowed, and bionic accepts pointers tagged before this call. */
|
||||
mallopt(M_BIONIC_SET_HEAP_TAGGING_LEVEL, M_HEAP_TAGGING_LEVEL_NONE);
|
||||
|
||||
const char *directory = argv[1];
|
||||
const char *name = argv[2];
|
||||
char hostfxr[PATH_MAX], apphost[PATH_MAX], assembly[PATH_MAX];
|
||||
if (snprintf(hostfxr, sizeof hostfxr, "%s/libhostfxr.so", directory) >= (int)sizeof hostfxr ||
|
||||
snprintf(apphost, sizeof apphost, "%s/%s", directory, name) >= (int)sizeof apphost ||
|
||||
snprintf(assembly, sizeof assembly, "%s/%s.dll", directory, name) >= (int)sizeof assembly) {
|
||||
fprintf(stderr, "translator_host: path too long\n");
|
||||
return 2;
|
||||
}
|
||||
|
||||
void *library = dlopen(hostfxr, RTLD_NOW | RTLD_LOCAL);
|
||||
if (library == NULL) {
|
||||
fprintf(stderr, "translator_host: %s\n", dlerror());
|
||||
return 3;
|
||||
}
|
||||
hostfxr_main_startupinfo_fn start = (hostfxr_main_startupinfo_fn)dlsym(library, "hostfxr_main_startupinfo");
|
||||
if (start == NULL) {
|
||||
fprintf(stderr, "translator_host: %s\n", dlerror());
|
||||
return 3;
|
||||
}
|
||||
|
||||
/* The app sees the apphost path as argv[0], then its own arguments. */
|
||||
argv[2] = apphost;
|
||||
return start(argc - 2, (const char **)argv + 2, apphost, directory, assembly);
|
||||
}
|
||||
+85
-11
@@ -281,13 +281,70 @@ if the game files pass the same checks as an extraction. A package built for ano
|
||||
is refused, and an installed game whose kit fingerprint no longer matches the APK shows as stale
|
||||
on Home.
|
||||
|
||||
Home's main button is always the next step: **Import from computer** until a game is installed
|
||||
(with **Select disc image** beside it while there are no game files), **Select disc image** when
|
||||
only the game files are missing, and **Play** once both are present. Building the game on the
|
||||
headset itself is the next step of this work. The translator (a self-contained
|
||||
`linux-bionic-arm64` .NET build, which needs heap pointer tagging disabled) and Termux's clang
|
||||
21.1.8 both ran on a Quest 3. A base translation took 6.7 minutes with a 3.1 GB memory peak, and
|
||||
the heaviest shard compiled in 65 seconds with a 480 MB peak.
|
||||
Home's main button is always the next step: **Select disc image** while there are no game files,
|
||||
**Build on this Quest** once they are there and no game is installed (or the installed one is
|
||||
stale), and **Play** once both are present. **Import from computer** sits beside the first two.
|
||||
|
||||
### Building the game on the headset
|
||||
|
||||
**Build on this Quest** (`GameBuild`, a `GameSetup` task in `GameSetupService`) does on the
|
||||
headset what `Build-QuestGame.ps1` does on a PC, from `DATA`, in about 28 minutes on a Quest 3:
|
||||
|
||||
1. It unpacks `assets/quest_toolchain` and `assets/game_kit` into `files/build/`.
|
||||
2. It downloads the NDK files a build needs (below) into `files/build/ndk`.
|
||||
3. It translates the disc: `translate-recursive`, `generate-data-init --target-os android` and
|
||||
`emit-build-shards`, in a workspace made of the kit's `translation/` copy of `recomp.yml`,
|
||||
`MAP.txt` and `runtime/src`, plus `main.dol` and `StaticR.rel` from `DATA`. A translation of the
|
||||
same kit, toolchain and disc is reused.
|
||||
4. It compiles the generated sources with the kit's flags, up to four at a time (fewer when the
|
||||
available memory allows less than 700 MB each). The blob assembly is compiled as plain
|
||||
`-x assembler`: preprocessing a `.S` makes clang start itself, which the linker trick below
|
||||
cannot do. Finished objects survive a cancelled or failed run.
|
||||
5. It links with `kit.json`'s `link.lld` and installs `libmain.so` and `game.json` through the same
|
||||
staging swap as an import. `builtBy` says the headset built it. A successful build deletes
|
||||
`files/build`.
|
||||
|
||||
The log goes to `Logs/build_<stamp>.log` next to `DATA`.
|
||||
|
||||
The toolchain (`android/Prepare-QuestToolchain.ps1`, run by the `prepareBase*QuestToolchain`
|
||||
Gradle tasks) is 249 MB unpacked. It travels as one deflated 80 MB `files.zip`, because asset
|
||||
packaging never compresses `.so` files, next to `toolchain.json`, which lists every file. It holds:
|
||||
|
||||
- the translator published for `linux-bionic-arm64` from a copy of the sources retargeted to
|
||||
net10.0, which is the first .NET with those runtime packs. That runtime is Mono.
|
||||
- `translator_host` (`android/toolchain/translator_host.c`).
|
||||
- Termux's clang/lld 21.1.8 and the ten shared libraries they and the translator load, pinned by
|
||||
package SHA-256 and stored under the names their users load. OpenSSL is stored as `libssl.so`,
|
||||
the name .NET's shim opens on Android; Android's own BoringSSL lacks symbols it needs.
|
||||
- `ndk.json`: the pin of the NDK files.
|
||||
|
||||
Android facts this design rests on, all measured on a Quest 3:
|
||||
|
||||
- An app cannot `exec` files in its private storage, but `/system/bin/linker64 <absolute path>`
|
||||
runs them (`PrivateCodeExecutionTest`). Every tool starts that way (`ToolProcess`), with
|
||||
`LD_LIBRARY_PATH` at the toolchain's `llvm/lib`.
|
||||
- Under the linker, the .NET apphost reads `/proc/self/exe`, gets the linker, and cannot find the
|
||||
app. `translator_host` hands hostfxr the app directory instead. It also turns off bionic's heap
|
||||
pointer tagging, which crashes the runtime at startup.
|
||||
- Termux's clang driver compiles with the same `cc1` arguments as the NDK's. Its link line is
|
||||
patched for Termux (`-rpath`, `-L/system/lib64`), and clang could not start lld anyway. So the kit
|
||||
export expands the link with the NDK's own driver (`clang++ -###`) into `link.lld`, a raw lld
|
||||
command with `{kit}`, `{ndk}`, `{output}` and `{game:*}` placeholders, and the headset runs
|
||||
`ld.lld` directly. The compile uses Termux's clang resource headers, which match that compiler.
|
||||
- The NDK files come from Google's `android-ndk-r29-linux.zip`, not from the APK. `RemoteZip`
|
||||
reads the zip's central directory and then only the 3,499 wanted entries, with HTTP range
|
||||
requests: the aarch64 sysroot, `libc++_shared.so`, the API 29 stubs and CRT objects,
|
||||
compiler-rt builtins, `libunwind.a` and `libatomic.a`. That is about 8 MB compressed of 784 MB.
|
||||
Each file must match the SHA-256 in `ndk.json`, and the list must match the digest pinned in the
|
||||
script. The requests say `Accept-Encoding: identity`: Android's HTTP stack asks for gzip by
|
||||
default, and Google's server then serves a gzip-encoded zip whose byte ranges are not the file's
|
||||
(HTTP 416). The Linux zip is used because its sysroot holds headers whose names differ only in case
|
||||
(`xt_TCPMSS.h`, `xt_tcpmss.h`), which a Windows copy of the NDK loses.
|
||||
- Translation peaks at 3.1 GB with Mono's default heap and 2.1 GB with
|
||||
`MONO_GC_PARAMS=soft-heap-limit=1200m` and four threads, for byte-identical output. The builder
|
||||
uses the latter.
|
||||
|
||||
`BuildRecipeTest` and `RemoteZipTest` cover the command lines and the zip reader.
|
||||
|
||||
### Build system
|
||||
|
||||
@@ -318,13 +375,14 @@ the heaviest shard compiled in 65 seconds with a 480 MB peak.
|
||||
|
||||
Prerequisites on the Windows host (all already present on the machine this
|
||||
was developed on): JDK 17, Android SDK with platform 34+, NDK `29.0.14206865`,
|
||||
SDK CMake `3.22.1`, `adb`, Rust 1.85+ with `rustup target add aarch64-linux-android`;
|
||||
a translated graph for your own disc (the installer's `BuildWorkspace/generated`,
|
||||
produced by the normal Windows pipeline).
|
||||
SDK CMake `3.22.1`, `adb`, Rust 1.85+ with `rustup target add aarch64-linux-android`,
|
||||
the .NET 10 SDK (for the headset's translator; the base APK build downloads ~70 MB of Termux
|
||||
packages into `android/.dependencies` the first time); a translated graph for your own disc (the
|
||||
installer's `BuildWorkspace/generated`, produced by the normal Windows pipeline).
|
||||
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File android/Prepare-QuestDependencies.ps1 # SDL3 3.4.4 AAR into android/app/libs
|
||||
powershell -ExecutionPolicy Bypass -File android/Build-Quest.ps1 -Install # the app and its game kit, debug-signed
|
||||
powershell -ExecutionPolicy Bypass -File android/Build-Quest.ps1 -Install # the app, its game kit and toolchain, debug-signed
|
||||
powershell -ExecutionPolicy Bypass -File android/Build-QuestGame.ps1 -Install # your game, against that kit, into Import (or WheelWizard VR's Build for Quest)
|
||||
powershell -ExecutionPolicy Bypass -File android/Build-Quest.ps1 -Flavor retroRewind # Retro Rewind (needs translate-mod output)
|
||||
adb push MarioKart.iso /sdcard/Download/ # then Select disc image in the launcher
|
||||
@@ -379,6 +437,15 @@ What has been verified on the development machine (September 2026):
|
||||
all 189 perspective draws replayed per eye, the first projection layer is
|
||||
submitted, and the menus return to the virtual screen. No WebGPU or OpenXR
|
||||
errors over a 90 second session.
|
||||
- **Device, 2026-09-17: the headset built its own game.** Build on this Quest ran inside the app
|
||||
in 27.8 minutes on a Quest 3: unpacking the toolchain, 43 MB of NDK files downloaded from Google
|
||||
and checked in about 6 seconds, translation in 646 s (2.1 GB peak), 92 sources compiled four at a
|
||||
time in 16 minutes (about 250 MB each, 3 GB still available), `ld.lld` in under a second, then
|
||||
the install and the cleanup of everything it unpacked. The game ran from the result: session
|
||||
`FOCUSED`, past 1,000 frames, the intro movie on the virtual screen. That `libmain.so` is
|
||||
byte-identical to one the same toolchain built from a shell, and against the PC-built library it
|
||||
has the same soname, `NEEDED` list, 785 undefined symbols and 29,995 translated functions
|
||||
(61,974 defined against 61,975: the PC's older clang keeps one inline helper out of line).
|
||||
|
||||
Bring-up fixes that only a device could reveal:
|
||||
|
||||
@@ -438,6 +505,13 @@ screen is up, reach Grand Prix character select. A value left over from an
|
||||
earlier run is ignored on the first read. Presses only land while the XR
|
||||
session is `FOCUSED`.
|
||||
|
||||
A debug APK also builds the game on the headset without a press, once `DATA` is there; the
|
||||
build log is the newest `Logs/build_*.log`:
|
||||
|
||||
```powershell
|
||||
adb shell am start -n org.wiicompiled.quest/.launcher.LauncherActivity --ez org.wiicompiled.quest.debug.BUILD_GAME true
|
||||
```
|
||||
|
||||
Performance, measured 2026-09-16 on a 50cc Luigi Circuit start with the player
|
||||
idle, over 40 s, with an optimized build (`-O3`, translated code `-O2`,
|
||||
`-mcpu=cortex-a77`):
|
||||
|
||||
Reference in new issue
Block a user