From 6a25b3bfa072edf079096d59d8ec29005d938264 Mon Sep 17 00:00:00 2001 From: Anmol Sethi Date: Sun, 17 May 2020 16:53:08 -0400 Subject: [PATCH] Document structure better Closes #1648 --- README.md | 20 ++-- ci/README.md | 169 +++++++++++++++++---------------- ci/dev/diff-vscode.sh | 1 + ci/steps/build-docker-image.sh | 2 +- doc/CONTRIBUTING.md | 73 ++++++++++++-- doc/FAQ.md | 3 + doc/assets/droplet.svg | 24 ----- src/node/app/README.md | 63 ------------ 8 files changed, 164 insertions(+), 191 deletions(-) delete mode 100644 doc/assets/droplet.svg delete mode 100644 src/node/app/README.md diff --git a/README.md b/README.md index 9af21d04..62f95592 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@ # code-server -Run [VS Code](https://github.com/Microsoft/vscode) on any machine anywhere and access it through the browser. +Run [VS Code](https://github.com/Microsoft/vscode) on any machine anywhere and access it in the browser. -- **Code anywhere:** Code on your Chromebook, tablet, and laptop with a +- **Code everywhere:** Code on your Chromebook, tablet, and laptop with a consistent dev environment. Develop on a Linux machine and pick up from any device with a web browser. - **Server-powered:** Take advantage of large cloud servers to speed up tests, compilations, downloads, and more. @@ -13,7 +13,7 @@ Run [VS Code](https://github.com/Microsoft/vscode) on any machine anywhere and a ## Getting started -For a proper setup and walkthrough, please see [./doc/guide.md](./doc/guide.md). +For a full setup and walkthrough, please see [./doc/guide.md](./doc/guide.md). ### Debian, Ubuntu @@ -38,10 +38,10 @@ systemctl --user enable --now code-server We recommend installing from `npm` if we don't have a precompiled release for your machine's platform or architecture. -**note:** Installing via `npm` requires certain dependencies for the native module builds. +**note:** Installing via `npm` builds native modules on install and so requires C dependencies. See [./doc/npm.md](./doc/npm.md) for installing these dependencies. -You also need at least node v12 installed. See [#1633](https://github.com/cdr/code-server/issues/1633). +You will need at least node v12 installed. See [#1633](https://github.com/cdr/code-server/issues/1633). ```bash npm install -g code-server @@ -75,13 +75,13 @@ docker run -it -p 127.0.0.1:8080:8080 \ We publish self contained `.tar.gz` archives for every release on [github](https://github.com/cdr/code-server/releases). They bundle the node binary and compiled native modules. -1. Download the latest release archive for your system from [github](https://github.com/cdr/code-server/releases) -2. Unpack the release -3. You can run code-server by executing `bin/code-server` +1. Download the latest release archive for your system from [github](https://github.com/cdr/code-server/releases). +2. Unpack the release. +3. You can run code-server by executing `./bin/code-server`. -Add the code-server `bin` directory to your `$PATH` to easily execute it without the full path every time. +Add the code-server `bin` directory to your `$PATH` to easily execute `code-server` without the full path every time. -Here is an example script for installing and using a self-contained code-server release on Linux: +Here is an example script for installing and using a self-contained `code-server` release on Linux: ```bash curl -sSL https://github.com/cdr/code-server/releases/download/3.3.0/code-server-3.3.0-linux-amd64.tar.gz | sudo tar -C /usr/local -xz diff --git a/ci/README.md b/ci/README.md index c3f4f86a..aa0dd5a3 100644 --- a/ci/README.md +++ b/ci/README.md @@ -2,105 +2,107 @@ This directory contains scripts used for code-server's continuous integration infrastructure. -Many of these scripts contain more detailed documentation and options in comments at the top. +Some of these scripts contain more detailed documentation and options +in header comments. -Any file and directory added into this tree should be documented here. +Any file or directory in this subdirectory should be documented here. + +- [./ci/lib.sh](./lib.sh) + - Contains code duplicated across these scripts. ## Publishing a release Make sure you have `$GITHUB_TOKEN` set and [hub](https://github.com/github/hub) installed. -1. Update the version of code-server in `package.json` and README.md/guide.md install examples and push a commit -2. GitHub actions will generate the `npm-package` and `release-packages` artifacts +1. Update the version of code-server in `package.json` and README.md/guide.md install examples and push a commit. +2. GitHub actions will generate the `npm-package`, `release-packages` and `release-images` artifacts. 3. Run `yarn release:github-draft` to create a GitHub draft release from the template with the updated version. 1. Summarize the major changes in the release notes and link to the relevant issues. -4. Wait for the artifacts in step 2 to build -5. Run `yarn release:github-assets` to download the artifacts and then upload them to the draft release -6. Run some basic sanity tests on one of the released packages -7. Publish the release - 1. CI will automatically grab the artifacts and then - 1. Publish the NPM package - 2. Publish the Docker Hub image -8. Update the homebrew and AUR packages +4. Wait for the artifacts in step 2 to build. +5. Run `yarn release:github-assets` to download the `release-packages` artifact and then upload them to the draft release. +6. Run some basic sanity tests on one of the released packages. +7. Publish the release. + 1. CI will automatically grab the artifacts and then: + 1. Publish the NPM package from `npm-package`. + 2. Publish the Docker Hub image from `release-images`. +8. Update the homebrew and AUR packages. ## dev This directory contains scripts used for the development of code-server. -- [./dev/container](./dev/container) - - See [CONTRIBUTING.md](../doc/CONTRIBUTING.md) for docs on the development container -- [./dev/ci.sh](./dev/ci.sh) (`yarn ci`) - - Runs `yarn fmt`, `yarn lint` and `yarn test` -- [./dev/fmt.sh](./dev/fmt.sh) (`yarn fmt`) - - Runs formatters -- [./dev/lint.sh](./dev/lint.sh) (`yarn lint`) - - Runs linters -- [./dev/test.sh](./dev/test.sh) (`yarn test`) - - Runs tests -- [./dev/vscode.sh](./dev/vscode.sh) (`yarn vscode`) - - Ensures [../lib/vscode](../lib/vscode) is cloned, patched and dependencies are installed -- [./dev/patch-vscode.sh](./dev/patch-vscode.sh) (`yarn vscode:patch`) - - Applies [./dev/vscode.patch](./dev/vscode.patch) to [../lib/vscode](../lib/vscode) -- [./dev/diff-vscode.sh](./dev/diff-vscode.sh) (`yarn vscode:diff`) - - Diffs [../lib/vscode](../lib/vscode) into [./dev/vscode.patch](./dev/vscode.patch) -- [./dev/vscode.patch](./dev/vscode.patch) - - Our patch of VS Code to enable remote browser access - - Generate it with `yarn vscode:diff` and apply with `yarn vscode:patch` -- [./dev/watch.ts](./dev/watch.ts) (`yarn watch`) - - Starts a process to build and launch code-server and restart on any code changes - - Example usage in [CONTRIBUTING.md](../doc/CONTRIBUTING.md) +- [./ci/dev/container](./dev/container) + - See [./doc/CONTRIBUTING.md](../doc/CONTRIBUTING.md) for docs on the development container. +- [./ci/dev/fmt.sh](./dev/fmt.sh) (`yarn fmt`) + - Runs formatters. +- [./ci/dev/lint.sh](./dev/lint.sh) (`yarn lint`) + - Runs linters. +- [./ci/dev/test.sh](./dev/test.sh) (`yarn test`) + - Runs tests. +- [./ci/dev/ci.sh](./dev/ci.sh) (`yarn ci`) + - Runs `yarn fmt`, `yarn lint` and `yarn test`. +- [./ci/dev/vscode.sh](./dev/vscode.sh) (`yarn vscode`) + - Ensures [./lib/vscode](../lib/vscode) is cloned, patched and dependencies are installed. +- [./ci/dev/patch-vscode.sh](./dev/patch-vscode.sh) (`yarn vscode:patch`) + - Applies [./ci/dev/vscode.patch](./dev/vscode.patch) to [./lib/vscode](../lib/vscode). +- [./ci/dev/diff-vscode.sh](./dev/diff-vscode.sh) (`yarn vscode:diff`) + - Diffs [./lib/vscode](../lib/vscode) into [./ci/dev/vscode.patch](./dev/vscode.patch). +- [./ci/dev/vscode.patch](./dev/vscode.patch) + - Our patch of VS Code, see [./doc/CONTRIBUTING.md](../doc/CONTRIBUTING.md#vs-code-patch). + - Generate it with `yarn vscode:diff` and apply with `yarn vscode:patch`. +- [./ci/dev/watch.ts](./dev/watch.ts) (`yarn watch`) + - Starts a process to build and launch code-server and restart on any code changes. + - Example usage in [./doc/CONTRIBUTING.md](../doc/CONTRIBUTING.md). ## build This directory contains the scripts used to build and release code-server. You can disable minification by setting `MINIFY=`. -- [./lib.sh](./lib.sh) - - Contains code duplicated across these scripts. -- [./build/build-code-server.sh](./build/build-code-server.sh) (`yarn build`) - - Builds code-server into ./out and bundles the frontend into ./dist. -- [./build/build-vscode.sh](./build/build-vscode.sh) (`yarn build:vscode`) - - Builds vscode into ./lib/vscode/out-vscode. -- [./build/build-release.sh](./build/build-release.sh) (`yarn release`) +- [./ci/build/build-code-server.sh](./build/build-code-server.sh) (`yarn build`) + - Builds code-server into `./out` and bundles the frontend into `./dist`. +- [./ci/build/build-vscode.sh](./build/build-vscode.sh) (`yarn build:vscode`) + - Builds vscode into `./lib/vscode/out-vscode`. +- [./ci/build/build-release.sh](./build/build-release.sh) (`yarn release`) - Bundles the output of the above two scripts into a single node module at `./release`. -- [./build/build-static-release.sh](./build/build-static-release.sh) (`yarn release:static`) - - Requires a release already built in `./release`. - - Will build a static release with node bundled into `./release-static` -- [./build/clean.sh](./build/clean.sh) (`yarn clean`) - - Removes all build artifacts - - Will also `git reset --hard lib/vscode` - - Useful to do a clean build -- [./build/code-server.sh](./build/code-server.sh) +- [./ci/build/build-static-release.sh](./build/build-static-release.sh) (`yarn release:static`) + - Requires a node module already built into `./release` with the above script. + - Will build a static release with node and native modules bundled into `./release-static`. +- [./ci/build/clean.sh](./build/clean.sh) (`yarn clean`) + - Removes all build artifacts. + - Will also `git reset --hard lib/vscode`. + - Useful to do a clean build. +- [./ci/build/code-server.sh](./build/code-server.sh) - Copied into static releases to run code-server with the bundled node binary. -- [./build/test-static-release.sh](./build/test-static-release.sh) (`yarn test:static-release`) - - Ensures code-server in the `./release-static` directory runs -- [./build/build-packages.sh](./build/build-packages.sh) (`yarn package`) - - Packages `./release-static` into an archive in `./release-packages` - - If on linux, [nfpm](https://github.com/goreleaser/nfpm) is used to generate .deb and .rpm -- [./build/nfpm.yaml](./build/nfpm.yaml) - - Used to configure [nfpm](https://github.com/goreleaser/nfpm) to generate .deb and .rpm -- [./build/code-server-nfpm.sh](./build/code-server-nfpm.sh) - - Entrypoint script for code-server for .deb and .rpm -- [./build/code-server.service](./build/code-server.service) - - systemd user service packaged into the debs and rpms -- [./build/release-github-draft.sh](./build/release-github-draft.sh) (`yarn release:github-draft`) - - Uses [hub](https://github.com/github/hub) to create a draft release with a template description -- [./build/release-github-assets.sh](./build/release-github-assets.sh) (`yarn release:github-assets`) - - Downloads the release-package artifacts for the current commit from CI +- [./ci/build/test-static-release.sh](./build/test-static-release.sh) (`yarn test:static-release`) + - Ensures code-server in the `./release-static` directory works by installing an extension. +- [./ci/build/build-packages.sh](./build/build-packages.sh) (`yarn package`) + - Packages `./release-static` into a `.tar.gz` archive in `./release-packages`. + - If on linux, [nfpm](https://github.com/goreleaser/nfpm) is used to generate `.deb` and `.rpm`. +- [./ci/build/nfpm.yaml](./build/nfpm.yaml) + - Used to configure [nfpm](https://github.com/goreleaser/nfpm) to generate `.deb` and `.rpm`. +- [./ci/build/code-server-nfpm.sh](./build/code-server-nfpm.sh) + - Entrypoint script for code-server for `.deb` and .rpm`. +- [./ci/build/code-server.service](./build/code-server.service) + - systemd user service packaged into the `.deb` and `.rpm`. +- [./ci/build/release-github-draft.sh](./build/release-github-draft.sh) (`yarn release:github-draft`) + - Uses [hub](https://github.com/github/hub) to create a draft release with a template description. +- [./ci/build/release-github-assets.sh](./build/release-github-assets.sh) (`yarn release:github-assets`) + - Downloads the release-package artifacts for the current commit from CI. - Uses [hub](https://github.com/github/hub) to upload the artifacts to the release - specified in `package.json` -- [./build/npm-postinstall.sh](./build/npm-postinstall.sh) - - Post install script for the npm package - - Bundled by`yarn release` + specified in `package.json`. +- [./ci/build/npm-postinstall.sh](./build/npm-postinstall.sh) + - Post install script for the npm package. + - Bundled by`yarn release`. ## release-container This directory contains the release docker container. - [./release-container/build.sh](./release-container/build.sh) - - Builds the release container with the tag `codercom/code-server:$VERSION-$ARCH` - - Assumes debian releases are ready in `./release-packages` + - Builds the release container with the tag `codercom/code-server-$ARCH:$VERSION`. + - Assumes debian releases are ready in `./release-packages`. ## container @@ -108,27 +110,26 @@ This directory contains the container for CI. ## steps -This directory contains a few scripts used in CI. -Just helps avoid clobbering the CI configuration. +This directory contains the scripts used in CI. +Helps avoid clobbering the CI configuration. - [./steps/fmt.sh](./steps/fmt.sh) - - Runs `yarn fmt` after ensuring VS Code is patched + - Runs `yarn fmt` after ensuring VS Code is patched. - [./steps/lint.sh](./steps/lint.sh) - - Runs `yarn lint` after ensuring VS Code is patched + - Runs `yarn lint` after ensuring VS Code is patched. - [./steps/test.sh](./steps/test.sh) - - Runs `yarn test` after ensuring VS Code is patched + - Runs `yarn test` after ensuring VS Code is patched. - [./steps/release.sh](./steps/release.sh) - - Runs the full release process - - Generates the npm package at `./release` -- [./steps/static-release.sh](./steps/static-release.sh) - - Takes the output of the previous script and generates a static release and packages -- [./steps/lib.sh](./steps/lib.sh) - - Contains helpers to download artifacts from github actions workflow runs + - Runs the release process. + - Generates the npm package at `./release`. +- [./steps/release-static.sh](./steps/release-static.sh) + - Takes the output of the previous script and generates a self-contained release and + release packages into `release-packages`. - [./steps/publish-npm.sh](./steps/publish-npm.sh) - - Grabs the `npm-package` release artifact for the current commit and publishes it on NPM + - Grabs the `npm-package` release artifact for the current commit and publishes it on npm. - [./steps/build-docker-image.sh](./steps/build-docker-image.sh) - - Builds the docker image and then saves it into `./release-images/$ARCH.tar` + - Builds the docker image and then saves it into `./release-images/code-server-$ARCH-$VERSION.tar`. - [./steps/push-docker-manifest.sh](./steps/push-docker-manifest.sh) - Loads all images in `./release-images` and then builds and pushes a multi architecture docker manifest for the amd64 and arm64 images to `codercom/code-server:$VERSION` and - `codercom/code-server:latest` + `codercom/code-server:latest`. diff --git a/ci/dev/diff-vscode.sh b/ci/dev/diff-vscode.sh index a1c1f027..98c955df 100755 --- a/ci/dev/diff-vscode.sh +++ b/ci/dev/diff-vscode.sh @@ -5,6 +5,7 @@ main() { cd "$(dirname "$0")/../.." cd ./lib/vscode + git add -A git diff HEAD > ../../ci/dev/vscode.patch } diff --git a/ci/steps/build-docker-image.sh b/ci/steps/build-docker-image.sh index c9344354..692fa40f 100755 --- a/ci/steps/build-docker-image.sh +++ b/ci/steps/build-docker-image.sh @@ -8,7 +8,7 @@ main() { ./ci/release-container/build.sh mkdir -p release-images - docker save "codercom/code-server-$ARCH:$VERSION" > "release-images/code-server-$ARCH:$VERSION.tar" + docker save "codercom/code-server-$ARCH:$VERSION" > "release-images/code-server-$ARCH-$VERSION.tar" } main "$@" diff --git a/doc/CONTRIBUTING.md b/doc/CONTRIBUTING.md index 95d47f33..23956059 100644 --- a/doc/CONTRIBUTING.md +++ b/doc/CONTRIBUTING.md @@ -1,7 +1,6 @@ # Contributing - [Detailed CI and build process docs](../ci) -- [Our VS Code Web docs](../src/node/app) ## Requirements @@ -9,15 +8,17 @@ Please refer to [VS Code's prerequisites](https://github.com/Microsoft/vscode/wi Differences: -- We require at least node v12 but later versions should work -- We use [fnpm](https://github.com/goreleaser/nfpm) to build .deb and .rpm packages +- We require a minimum of node v12 but later versions should work. +- We use [fnpm](https://github.com/goreleaser/nfpm) to build `.deb` and `.rpm` packages. +- The [CI container](../ci/container/Dockerfile) is a useful reference for all our dependencies. ## Development Workflow ```shell yarn yarn vscode -yarn watch # Visit http://localhost:8080 once completed. +yarn watch +# Visit http://localhost:8080 once the build completed. ``` To develop inside of an isolated docker container: @@ -35,9 +36,6 @@ Any changes made to the source will be live reloaded. If changes are made to the patch and you've built previously you must manually reset VS Code then run `yarn vscode:patch`. -Some docs are available at [../src/node/app](../src/node/app) on how code-server -works internally. - ## Build ```shell @@ -48,7 +46,8 @@ yarn build:vscode yarn release cd release yarn --production -node . # Run the built JavaScript with Node. +# Runs the built JavaScript with Node. +node . ``` Now you can make it static and build packages with: @@ -57,6 +56,62 @@ Now you can make it static and build packages with: yarn release:static yarn test:static-release yarn package +# The static release is in ./release-static +# .deb, .rpm and the self-contained archive are in ./release-packages ``` -The static release will be in `./release-static` and .deb, .rpm and self-contained release in `./release-packages`. +## Structure + +The `code-server` script serves an HTTP API to login and start a remote VS Code process. + +The CLI code is in [./src/node](./src/node) and the HTTP routes are implemented in +[./src/node/app](./src/node/app). + +Most of the meaty parts are in our VS Code patch which is described next. + +### VS Code Patch + +Back in v1 of code-server, we had an extensive patch of VS Code that split the codebase +into a frontend and server. The frontend consisted of all UI code and the server ran +the extensions and exposed an API to the frontend for file access and everything else +that the UI needed. + +This worked but eventually Microsoft added support to VS Code to run it in the web. +They have open sourced the frontend but have kept the server closed source. + +So in interest of piggy backing off their work, v2 and beyond use the VS Code +web frontend and fill in the server. This is contained in our +[./ci/dev/vscode.patch](../ci/dev/vscode.patch) under the path `src/vs/server`. + +Other notable changes in our patch include: + +- Add our own build file which includes our code and VS Code's web code. +- Allow multiple extension directories (both user and built-in). +- Modify the loader, websocket, webview, service worker, and asset requests to + use the URL of the page as a base (and TLS if necessary for the websocket). +- Send client-side telemetry through the server. +- Allow modification of the display language. +- Make it possible for us to load code on the client. +- Make extensions work in the browser. +- Make it possible to install extensions of any kind. +- Fix getting permanently disconnected when you sleep or hibernate for a while. +- Add connection type to web socket query parameters. + +Some known issues presently: + +- Creating custom VS Code extensions and debugging them doesn't work. +- Extension profiling and tips are currently disabled. + +As the web portion of VS Code matures, we'll be able to shrink and maybe even entirely +eliminate our patch. In the meantime, however, upgrading the VS Code version requires +ensuring that the patch still applies and has the intended effects. + +To generate a new patch run `yarn vscode:diff`. + +**note**: We have extension docs on the CI and build system at [./ci/README.md](../ci/README.md) + +If functionality doesn't depend on code from VS Code then it should be moved +into code-server otherwise it should be in the patch. + +In the future we'd like to run VS Code unit tests against our builds to ensure features +work as expected. diff --git a/doc/FAQ.md b/doc/FAQ.md index c5569e1c..6a933964 100644 --- a/doc/FAQ.md +++ b/doc/FAQ.md @@ -54,6 +54,9 @@ See below for installing an extension from the cli. Feel free to file an issue to add a missing extension to the marketplace. +If you have your own custom marketplace, it is possible to point code-server to it by setting +`$SERVICE_URL` and `$ITEM_URL` to point to it. + ## Where are extensions stored? Defaults to `~/.local/share/code-server/extensions`. diff --git a/doc/assets/droplet.svg b/doc/assets/droplet.svg deleted file mode 100644 index ecbb3f24..00000000 --- a/doc/assets/droplet.svg +++ /dev/null @@ -1,24 +0,0 @@ - - - - do-btn-blue-ghost - Created with Sketch. - - - - - - - - - - - - - - Create a Droplet - - - - - diff --git a/src/node/app/README.md b/src/node/app/README.md deleted file mode 100644 index 02d2c438..00000000 --- a/src/node/app/README.md +++ /dev/null @@ -1,63 +0,0 @@ -# app - -Implementation of [VS Code](https://code.visualstudio.com/) remote/web for use -in `code-server`. - -## Docker - -To debug Golang in VS Code using the -[ms-vscode-go extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode.Go), -you need to add `--security-opt seccomp=unconfined` to your `docker run` -arguments when launching code-server with Docker. See -[#725](https://github.com/cdr/code-server/issues/725) for details. - -## Known Issues - -- Creating custom VS Code extensions and debugging them doesn't work. -- Extension profiling and tips are currently disabled. - -## Extensions - -`code-server` does not provide access to the official -[Visual Studio Marketplace](https://marketplace.visualstudio.com/vscode). Instead, -Coder has created a custom extension marketplace that we manage for open-source -extensions. If you want to use an extension with code-server that we do not have -in our marketplace please look for a release in the extension’s repository, -contact us to see if we have one in the works or, if you build an extension -locally from open source, you can copy it to the `extensions` folder. If you -build one locally from open-source please contribute it to the project and let -us know so we can give you props! If you have your own custom marketplace, it is -possible to point code-server to it by setting the `SERVICE_URL` and `ITEM_URL` -environment variables. - -## Development: upgrading VS Code - -We patch VS Code to provide and fix some functionality. As the web portion of VS -Code matures, we'll be able to shrink and maybe even entirely eliminate our -patch. In the meantime, however, upgrading the VS Code version requires ensuring -that the patch still applies and has the intended effects. - -If functionality doesn't depend on code from VS Code then it should be moved -into code-server otherwise it should be in the patch. - -To generate a new patch, **stage all the changes** you want to be included in -the patch in the VS Code source, then run `yarn vscode:diff` in this -directory. - -Notable changes include: - -- Add our own build file which includes our code and VS Code's web code. -- Allow multiple extension directories (both user and built-in). -- Modify the loader, websocket, webview, service worker, and asset requests to - use the URL of the page as a base (and TLS if necessary for the websocket). -- Send client-side telemetry through the server. -- Make changing the display language work. -- Make it possible for us to load code on the client. -- Make extensions work in the browser. -- Make it possible to install extensions of any kind. -- Fix getting permanently disconnected when you sleep or hibernate for a while. -- Add connection type to web socket query parameters. - -## Future - -- Run VS Code unit tests against our builds to ensure features work as expected.