Skip to content

Commit bb6ef51

Browse files
lionel-lionel-
authored
Add support for github refs when downloading Ark (#7645)
Adds support for specifying Ark versions as github refs in `package.json`: ```ts "positron": { "binaryDependencies": { "ark": "posit-dev/ark@main" // Use the main branch "ark": "posit-dev/ark@experimental-feature" // Use a feature branch "ark": "posit-dev/ark@a1b2c3d" // Use a specific commit "ark": "posit-dev/ark@v0.1.183" // Use a specific tag } } ``` The github revision is downloaded, built, and installed. This allows CI-testing a branch in the Positron repo against a branch in the Ark repo without having to release Ark first. Releasing just for the purpose of testing is really not ideal as the feature might not be working well yet and other developers might need to do further Ark releases to make progress with their own work. To prevent committing a dev ref that might have been forgotten in the `package.json` file, a github action watches changes to this file and checks that a release version is used. <!-- Thank you for submitting a pull request. If this is your first pull request you can find information about contributing here: * https://github.com/posit-dev/positron/blob/main/CONTRIBUTING.md We recommend synchronizing your branch with the latest changes in the main branch by either pulling or rebasing. --> <!-- Describe briefly what problem this pull request resolves, or what new feature it introduces. Include screenshots of any new or altered UI. Link to any GitHub issues but avoid "magic" keywords that will automatically close the issue. If there are any details about your approach that are unintuitive or you want to draw attention to, please describe them here. --> ### Release Notes <!-- Optionally, replace `N/A` with text to be included in the next release notes. The `N/A` bullets are ignored. If you refer to one or more Positron issues, these issues are used to collect information about the feature or bugfix, such as the relevant language pack as determined by Github labels of type `lang: `. The note will automatically be tagged with the language. These notes are typically filled by the Positron team. If you are an external contributor, you may ignore this section. --> #### New Features - N/A #### Bug Fixes - N/A ### QA Notes <!-- Add additional information for QA on how to validate the change, paying special attention to the level of risk, adjacent areas that could be affected by the change, and any important contextual information not present in the linked issues. -->
1 parent 86dc431 commit bb6ef51

File tree

3 files changed

+242
-4
lines changed

3 files changed

+242
-4
lines changed

‎.github/workflows/prevent-repo-references.yml

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
name: Prevent GitHub Repo References in Package.json
2+
3+
on:
4+
pull_request:
5+
branches: [main, release/*, prerelease/*]
6+
paths:
7+
- "extensions/positron-r/package.json"
8+
push:
9+
branches: [main, release/*, prerelease/*]
10+
paths:
11+
- "extensions/positron-r/package.json"
12+
13+
jobs:
14+
check-repo-references:
15+
name: Check for GitHub Repo References in Ark Version
16+
runs-on: ubuntu-latest
17+
steps:
18+
- name: Checkout code
19+
uses: actions/checkout@v3
20+
21+
- name: Check for repo references in package.json
22+
run: |
23+
echo "Checking for GitHub repo references in package.json"
24+
25+
# Extract the Ark version from package.json using jq
26+
ARK_VERSION=$(jq -r '.positron.binaryDependencies.ark // empty' extensions/positron-r/package.json)
27+
28+
# Check if the extracted version follows the GitHub reference pattern
29+
if [[ "$ARK_VERSION" =~ ^[a-zA-Z0-9_-]+/[a-zA-Z0-9_-]+@[a-zA-Z0-9._\/-]+ ]] then
30+
echo "::error::GitHub repo reference found in extensions/positron-r/package.json: $ARK_VERSION"
31+
echo "GitHub repo references (org/repo@revision format) are only for development and should not be used in main or release branches."
32+
exit 1
33+
else
34+
echo "No GitHub repo references found in extensions/positron-r/package.json"
35+
fi

‎extensions/positron-r/scripts/README.md

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# Positron R Extension Scripts
2+
3+
## `install-kernel.ts`
4+
5+
This script handles downloading and installing the Ark R kernel, which is used by the Positron R extension to execute R code.
6+
7+
8+
### Installation Methods
9+
10+
#### Release Mode (Production Use)
11+
12+
- Downloads pre-built binaries from GitHub releases
13+
- Uses a semantic version number like `"0.1.182"`
14+
- Example in package.json:
15+
```json
16+
"positron": {
17+
"binaryDependencies": {
18+
"ark": "0.1.182"
19+
}
20+
}
21+
```
22+
23+
24+
#### Local development mode
25+
26+
For kernel developers working directly on the Ark kernel, the script will check for locally built versions in a sibling `ark` directory before attempting to download or build from source.
27+
28+
Note that this has precedence over downloading Ark based on the version specified in `package.json` (both release and github references).
29+
30+
31+
#### CI development Mode
32+
33+
- Clones and builds the Ark kernel from source using a GitHub repositoryreference
34+
- Uses the format `"org/repo@branch_or_revision"`
35+
- Examples in package.json:
36+
```json
37+
"positron": {
38+
"binaryDependencies": {
39+
"ark": "posit-dev/ark@main" // Use the main branch
40+
"ark": "posit-dev/ark@experimental-feature" // Use a feature branch
41+
"ark": "posit-dev/ark@a1b2c3d" // Use a specific commit
42+
"ark": "posit-dev/ark@v0.1.183" // Use a specific tag
43+
}
44+
}
45+
```
46+
47+
The repository reference format (`org/repo@branch_or_revision`) should only be used during development and never be merged into main or release branches. A GitHub Action (`prevent-repo-references.yml`) enforces this restriction by checking pull requests to main and release branches for this pattern.
48+
49+
50+
### Authentication
51+
52+
When accessing GitHub repositories or releases, the script attempts to find a GitHub Personal Access Token (PAT) in the following order:
53+
54+
1. The `GITHUB_PAT` environment variable
55+
2. The `POSITRON_GITHUB_PAT` environment variable
56+
3. The git config setting `credential.https://api.github.com.token`
57+
58+
Providing a PAT is recommended to avoid rate limiting and to access private repositories.
59+
60+
61+
## `compile-syntax.ts`
62+
63+
This script compiles TextMate grammar files for syntax highlighting.
64+
65+
66+
## `post-install.ts`
67+
68+
This script performs additional setup steps after the extension is installed.

‎extensions/positron-r/scripts/install-kernel.ts

Lines changed: 139 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ import * as decompress from 'decompress';
77
import * as fs from 'fs';
88
import { IncomingMessage } from 'http';
99
import * as https from 'https';
10+
import * as os from 'os';
1011
import { platform, arch } from 'os';
1112
import * as path from 'path';
1213
import { promisify } from 'util';
@@ -16,6 +17,7 @@ import { promisify } from 'util';
1617
const readFileAsync = promisify(fs.readFile);
1718
const writeFileAsync = promisify(fs.writeFile);
1819
const existsAsync = promisify(fs.exists);
20+
const mkdtempAsync = promisify(fs.mkdtemp);
1921

2022
// Create a promisified version of https.get. We can't use the built-in promisify
2123
// because the callback doesn't follow the promise convention of (error, result).
@@ -64,19 +66,29 @@ async function getLocalArkVersion(): Promise<string | null> {
6466
*
6567
* @param command The command to execute.
6668
* @param stdin Optional stdin to pass to the command.
69+
* @param cwd Optional working directory for the command
6770
* @returns A promise that resolves with the stdout and stderr of the command.
6871
*/
69-
async function executeCommand(command: string, stdin?: string):
70-
Promise<{ stdout: string; stderr: string }> {
72+
async function executeCommand(
73+
command: string,
74+
stdin?: string,
75+
cwd?: string
76+
): Promise<{ stdout: string; stderr: string }> {
7177
const { exec } = require('child_process');
7278
return new Promise((resolve, reject) => {
73-
const process = exec(command, (error: any, stdout: string, stderr: string) => {
79+
const options: { cwd?: string } = {};
80+
if (cwd) {
81+
options.cwd = cwd;
82+
}
83+
84+
const process = exec(command, options, (error: any, stdout: string, stderr: string) => {
7485
if (error) {
7586
reject(error);
7687
} else {
7788
resolve({ stdout, stderr });
7889
}
7990
});
91+
8092
if (stdin) {
8193
process.stdin.write(stdin);
8294
process.stdin.end();
@@ -214,6 +226,100 @@ async function downloadAndReplaceArk(version: string,
214226
}
215227
}
216228

229+
/**
230+
* Downloads and builds Ark from a GitHub repository at a specific branch or revision.
231+
*
232+
* This function supports development workflows by allowing developers to:
233+
* - Test changes from non-released branches
234+
* - Use experimental features not yet in a release
235+
* - Develop against the latest code in a repository
236+
*
237+
* IMPORTANT: This feature is for DEVELOPMENT ONLY and should not be used in
238+
* production environments or merged to main branches. A GitHub Action enforces
239+
* this restriction by blocking PRs with repo references in package.json.
240+
*
241+
* @param ref The GitHub repo reference in the format 'org/repo@branch_or_revision'
242+
* @param githubPat An optional Github Personal Access Token
243+
*/
244+
async function downloadFromGitHubRepository(
245+
ref: string,
246+
githubPat: string | undefined
247+
): Promise<void> {
248+
const { org, repo, revision } = parseGitHubRepoReference(ref);
249+
250+
console.log(`Downloading and building Ark from GitHub repo: ${org}/${repo} at revision: ${revision}`);
251+
252+
// Create a temporary directory for cloning the repo
253+
const tempDir = await mkdtempAsync(path.join(os.tmpdir(), 'ark-build-'));
254+
255+
try {
256+
console.log(`Created temporary build directory: ${tempDir}`);
257+
258+
// Set up git command with credentials if available
259+
let gitCloneCommand = `git clone https://github.com/${org}/${repo}.git ${tempDir}`;
260+
if (githubPat) {
261+
gitCloneCommand = `git clone https://x-access-token:${githubPat}@github.com/${org}/${repo}.git ${tempDir}`;
262+
}
263+
264+
// Clone the repository
265+
console.log('Cloning repository...');
266+
await executeCommand(gitCloneCommand);
267+
268+
// Checkout the specific revision
269+
console.log(`Checking out revision: ${revision}`);
270+
await executeCommand(`git checkout ${revision}`, undefined, tempDir);
271+
272+
// Verify that we have a valid Ark repository structure
273+
const cargoTomlPath = path.join(tempDir, 'Cargo.toml');
274+
if (!await existsAsync(cargoTomlPath)) {
275+
throw new Error(`Invalid Ark repository: Cargo.toml not found at the repository root`);
276+
}
277+
278+
console.log('Building Ark from source...');
279+
280+
const buildOutput = await executeCommand('cargo build --release', undefined, tempDir);
281+
console.log('Ark build stdout:', buildOutput.stdout);
282+
console.log('Ark build stderr:', buildOutput.stderr);
283+
284+
// Determine the location of the built binary
285+
const kernelName = platform() === 'win32' ? 'ark.exe' : 'ark';
286+
const binaryPath = path.join(tempDir, 'target', 'release', kernelName);
287+
288+
// Ensure the binary was built successfully
289+
if (!fs.existsSync(binaryPath)) {
290+
throw new Error(`Failed to build Ark binary at ${binaryPath}`);
291+
}
292+
293+
// Run the binary and check output. An error will be thrown if this fails.
294+
const { stdout: versionStdout, stderr: versionStderr } = await executeCommand(`${binaryPath}`);
295+
console.log('Ark stdout:', versionStdout);
296+
console.log('Ark stderr:', versionStderr);
297+
298+
// Create the resources/ark directory if it doesn't exist
299+
const arkDir = path.join('resources', 'ark');
300+
if (!await existsAsync(arkDir)) {
301+
await fs.promises.mkdir(arkDir, { recursive: true });
302+
}
303+
304+
// Copy the binary to the resources directory
305+
await fs.promises.copyFile(binaryPath, path.join(arkDir, kernelName));
306+
console.log(`Successfully built and installed Ark from ${org}/${repo}@${revision}`);
307+
308+
// Write the version information to VERSION file
309+
await writeFileAsync(path.join(arkDir, 'VERSION'), ref);
310+
311+
} catch (err) {
312+
throw new Error(`Error building Ark from GitHub repository: ${err}`);
313+
} finally {
314+
// Clean up the temporary directory
315+
try {
316+
await fs.promises.rm(tempDir, { recursive: true, force: true });
317+
} catch (err) {
318+
console.warn(`Warning: Failed to clean up temporary directory ${tempDir}: ${err}`);
319+
}
320+
}
321+
}
322+
217323
async function main() {
218324
const kernelName = platform() === 'win32' ? 'ark.exe' : 'ark';
219325

@@ -252,6 +358,7 @@ async function main() {
252358
console.log(`package.json version: ${packageJsonVersion} `);
253359
console.log(`Downloaded ark version: ${localArkVersion ? localArkVersion : 'Not found'} `);
254360

361+
// Skip installation if versions match
255362
if (packageJsonVersion === localArkVersion) {
256363
console.log('Versions match. No action required.');
257364
return;
@@ -293,7 +400,35 @@ async function main() {
293400
}
294401
}
295402

296-
await downloadAndReplaceArk(packageJsonVersion, githubPat);
403+
// Check if the version is a GitHub repo reference
404+
if (isGitHubRepoReference(packageJsonVersion)) {
405+
await downloadFromGitHubRepository(packageJsonVersion, githubPat);
406+
} else {
407+
await downloadAndReplaceArk(packageJsonVersion, githubPat);
408+
}
409+
}
410+
411+
/**
412+
* Check if the version string follows the format 'org/repo@branch_or_revision'.
413+
*/
414+
function isGitHubRepoReference(version: string): boolean {
415+
return /^[a-zA-Z0-9_-]+\/[a-zA-Z0-9_-]+@[a-zA-Z0-9._\/-]+$/.test(version);
416+
}
417+
418+
/**
419+
* Parse a GitHub repo reference in the format 'org/repo@branch_or_revision'.
420+
*/
421+
function parseGitHubRepoReference(reference: string): { org: string; repo: string; revision: string } {
422+
const orgRepoMatch = reference.match(/^([a-zA-Z0-9_-]+)\/([a-zA-Z0-9_-]+)@([a-zA-Z0-9._\/-]+)$/);
423+
if (!orgRepoMatch) {
424+
throw new Error(`Invalid GitHub repo reference: ${reference}`);
425+
}
426+
427+
return {
428+
org: orgRepoMatch[1],
429+
repo: orgRepoMatch[2],
430+
revision: orgRepoMatch[3]
431+
};
297432
}
298433

299434
main().catch((error) => {

0 commit comments

Comments
 (0)