- Manual installation:
- install/install-manual-prerequisites.html.textile.liquid
- install/packages.html.textile.liquid
- - admin/upgrading.html.textile.liquid
- Configuration:
- install/config.html.textile.liquid
- - admin/config-migration.html.textile.liquid
- - admin/config.html.textile.liquid
- admin/config-urls.html.textile.liquid
+ - admin/config.html.textile.liquid
+ - Maintenance and upgrading:
+ - admin/upgrading.html.textile.liquid
+ - admin/maintenance-and-upgrading.html.textile.liquid
- Core:
- install/install-api-server.html.textile.liquid
- Keep:
+++ /dev/null
----
-layout: default
-navsection: installguide
-title: Migrating Configuration from v1.4 to v2.0
-...
-
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-{% include 'notebox_begin_warning' %}
-_New installations of Arvados 2.0+ can skip this section_
-{% include 'notebox_end' %}
-
-Arvados 2.0 migrates to a centralized configuration file for all components. The centralized Arvados configuration is @/etc/arvados/config.yml@. Components that support the new centralized configuration are listed below. During the migration period, legacy configuration files are still loaded and take precedence over the centralized configuration file.
-
-h2. API server
-
-The legacy API server configuration is stored in @config/application.yml@ and @config/database.yml@. After migration to @/etc/arvados/config.yml@, both of these files should be moved out of the way and/or deleted.
-
-Change to the API server directory and use the following commands:
-
-<pre>
-$ RAILS_ENV=production bin/rake config:migrate > config.yml
-$ cp config.yml /etc/arvados/config.yml
-</pre>
-
-This will print the contents of @config.yml@ after merging the legacy @application.yml@ and @database.yml@ into the existing systemwide @config.yml@. It may be redirected to a file and copied to @/etc/arvados/config.yml@ (it is safe to copy over, all configuration items from the existing @/etc/arvados/config.yml@ will be included in the migrated output).
-
-If you wish to update @config.yml@ configuration by hand, or check that everything has been migrated, use @config:diff@ to print configuration items that differ between @application.yml@ and the system @config.yml@.
-
-<pre>
-$ RAILS_ENV=production bin/rake config:diff
-</pre>
-
-This command will also report if no migrations are required.
-
-h2. Workbench
-
-The legacy workbench configuration is stored in @config/application.yml@. After migration to @/etc/arvados/config.yml@, this file should be moved out of the way and/or deleted.
-
-Change to the workbench server directory and use the following commands:
-
-<pre>
-$ RAILS_ENV=production bin/rake config:migrate > config.yml
-$ cp config.yml /etc/arvados/config.yml
-</pre>
-
-This will print the contents of @config.yml@ after merging the legacy @application.yml@ into the existing systemwide @config.yml@. It may be redirected to a file and copied to @/etc/arvados/config.yml@ (it is safe to copy over, all configuration items from the existing @/etc/arvados/config.yml@ will be included in the migrated output).
-
-If you wish to update @config.yml@ configuration by hand, or check that everything has been migrated, use @config:diff@ to print configuration items that differ between @application.yml@ and the system @config.yml@.
-
-<pre>
-$ RAILS_ENV=production bin/rake config:diff
-</pre>
-
-This command will also report if no migrations are required.
-
-h2. keepstore, keep-web, crunch-dispatch-slurm, arvados-ws, keepproxy, arv-git-httpd, keep-balance
-
-The legacy config for each component (loaded from @/etc/arvados/component/component.yml@ or a different location specified via the -legacy-component-config command line argument) takes precedence over the centralized config. After you migrate everything from the legacy config to the centralized config, you should delete @/etc/arvados/component/component.yml@ and/or stop using the corresponding -legacy-component-config argument.
-
-To migrate a component configuration, do this on each node that runs an Arvados service:
-
-# Ensure that the latest @config.yml@ is installed on the current node
-# Install @arvados-server@ using @apt-get@ or @yum@.
-# Run @arvados-server config-check@, review and apply the recommended changes to @/etc/arvados/config.yml@
-# After applying changes, re-run @arvados-server config-check@ again to check for additional warnings and recommendations.
-# When you are satisfied, delete the legacy config file, restart the service, and check its startup logs.
-# Copy the updated @config.yml@ file to your next node, and repeat the process there.
-# When you have a @config.yml@ file that includes all volumes on all keepstores, it is important to add a 'Rendezvous' parameter to the InternalURLs entries to make sure the old volume identifiers line up with the new config. If you don't do this, @keep-balance@ will want to shuffle all the existing data around to match the new volume order. The 'Rendezvous' value should be the last 15 characters of the keepstore's UUID in the old configuration. Here's an example:
-
-<notextile>
-<pre><code>Clusters:
- xxxxx:
- Services:
- Keepstore:
- InternalURLs:
- "http://keep1.xxxxx.arvadosapi.com:25107": {Rendezvous: "eim6eefaibesh3i"}
- "http://keep2.xxxxx.arvadosapi.com:25107": {Rendezvous: "yequoodalai7ahg"}
- "http://keep3.xxxxx.arvadosapi.com:25107": {Rendezvous: "eipheho6re1shou"}
- "http://keep4.xxxxx.arvadosapi.com:25107": {Rendezvous: "ahk7chahthae3oo"}
-</code></pre>
-</notextile>
-
-In this example, the keepstore with the name `keep1` had the uuid `xxxxx-bi6l4-eim6eefaibesh3i` in the old configuration.
-
-After migrating and removing all legacy config files, make sure the @/etc/arvados/config.yml@ file is identical across all system nodes -- API server, keepstore, etc. -- and restart all services to make sure they are using the latest configuration.
-
-h2. Cloud installations only: node manager
-
-Node manager is deprecated and replaced by @arvados-dispatch-cloud@. No automated config migration is available. Follow the instructions to "install the cloud dispatcher":../install/crunch2-cloud/install-dispatch-cloud.html
-
-*Only one dispatch process should be running at a time.* If you are migrating a system that currently runs Node manager and @crunch-dispatch-slurm@, it is safest to remove the @crunch-dispatch-slurm@ service entirely before installing @arvados-dispatch-cloud@.
-
-<notextile>
-<pre><code>~$ <span class="userinput">sudo systemctl --now disable crunch-dispatch-slurm</span>
-~$ <span class="userinput">sudo apt-get remove crunch-dispatch-slurm</span>
-</code></pre>
-</notextile>
-
-h2. arvados-controller, arvados-dispatch-cloud
-
-Already uses centralized config exclusively. No migration needed.
The Arvados configuration is stored at @/etc/arvados/config.yml@
-See "Migrating Configuration":config-migration.html for information about migrating from legacy component-specific configuration files.
-
{% codeblock as yaml %}
{% include 'config_default_yml' %}
{% endcodeblock %}
--- /dev/null
+---
+layout: default
+navsection: installguide
+title: Maintenance and upgrading
+...
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+# "Commercial support":#commercial_support
+# "Maintaining Arvados":#maintaining
+## "Modification of the config.yml file":#configuration
+## "Distributing the configuration file":#distribution
+## "Restart the services affected by the change":#restart
+# "Upgrading Arvados":#upgrading
+
+h2(#commercial_support). Commercial support
+
+Arvados is "100% open source software":{{site.baseurl}}/user/copying/copying.html. Anyone can download, install, maintain and upgrade it. However, if this is not something you want to spend your time and energy doing, "Curii Corporation":https://curii.com provides managed Arvados installations as well as commercial support for Arvados. Please contact "info@curii.com":mailto:info@curii.com for more information.
+
+If you'd prefer to do things yourself, a few starting points for maintaining and upgrading Arvados can be found below.
+
+h2(#maintaining). Maintaining Arvados
+
+After Arvados is installed, periodic configuration changes may be required to adapt the software to your needs. Arvados uses a unified configuration file, which is normally found at @/etc/arvados/config.yml@.
+
+Making a configuration change to Arvados typically involves three steps:
+
+* modification of the @config.yml@ file
+* distribution of the modified file to the machines in the cluster
+* restarting of the services affected by the change
+
+h3(#configchange). Modification of the @config.yml@ file
+
+Consult the "configuration reference":{{site.baseurl}}/admin/config.html or another part of the documentation to identify the change to be made.
+
+Preserve a copy of your existing configuration file as a backup, and make the desired modification.
+
+Run @arvados-server config-check@ to make sure the configuration file has no errors and no warnings.
+
+h3(#distribution). Distribute the configuration file
+
+We recommend to keep the @config.yml@ file in sync between all the Arvados system nodes, to avoid issues with services running on different versions of the configuration.
+
+Distribution of the configuration file can be done in many ways, e.g. scp, configuration management software, etc.
+
+h3(#restart). Restart the services affected by the change
+
+If you know which Arvados service uses the specific configuration that was modified, restart those services. When in doubt, restart all Arvados system services.
+
+h2(#upgrading). Upgrading Arvados
+
+Upgrading Arvados typically involves the following steps:
+
+# consult the "upgrade notes":{{site.baseurl}}/admin/upgrading.html and the "release notes":https://arvados.org/releases/ for the release you want to upgrade to
+# Wait for the cluster to be idle and stop Arvados services.
+# Make a backup of your database, as a precaution.
+# update the configuration file for the new release, if necessary (see "Maintaining Arvados":#maintaining above)
+# rebuild and deploy the "compute node image":{{site.baseurl}}/install/crunch2-cloud/install-compute-node.html (cloud only)
+# Install new packages using @apt-get upgrade@ or @yum upgrade@.
+# Wait for package installation scripts as they perform any necessary data migrations.
+# Verify that the Arvados services were restarted as part of the package upgrades.
---
layout: default
navsection: installguide
-title: "Upgrading Arvados and Release notes"
+title: "Arvados upgrade notes"
...
{% comment %}
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-For Arvados administrators, this page will cover what you need to know and do in order to ensure a smooth upgrade of your Arvados installation. For general release notes covering features added and bugs fixed, see "Arvados releases":https://arvados.org/releases .
+For Arvados administrators, this page will cover what you need to know and do in order to ensure a smooth upgrade of your Arvados installation. For general release notes covering features added and bugs fixed, see "Arvados releases":https://arvados.org/releases.
-h2. General process
-
-# Consult upgrade notes below to see if any manual configuration updates are necessary.
-# Wait for the cluster to be idle and stop Arvados services.
-# Make a backup of your database, as a precaution.
-# Install new packages using @apt-get upgrade@ or @yum upgrade@.
-# Wait for package installation scripts as they perform any necessary data migrations.
-# Restart Arvados services.
+Upgrade instructions can be found at "Maintenance and upgrading":{{site.baseurl}}/admin/maintenance-and-upgrading.html#upgrading.
h2. Upgrade notes
-Some versions introduce changes that require special attention when upgrading: e.g., there is a new service to install, or there is a change to the default configuration that you might need to override in order to preserve the old behavior.
+Some versions introduce changes that require special attention when upgrading: e.g., there is a new service to install, or there is a change to the default configuration that you might need to override in order to preserve the old behavior. These notes are listed below, organized by release version. Scroll down to the version number you are upgrading to.
{% comment %}
Note to developers: Add new items at the top. Include the date, issue number, commit, and considerations/instructions for those about to upgrade.
h2(#main). development main (as of 2022-02-10)
-"previous: Upgrading from 2.3.0":#v2_3_0
+"previous: Upgrading to 2.3.0":#v2_3_0
h3. Anonymous token changes
h3. Migrating to centralized config.yml
-See "Migrating Configuration":config-migration.html for notes on migrating legacy per-component configuration files to the new centralized @/etc/arvados/config.yml@.
+See "Migrating Configuration":https://doc.arvados.org/v2.1/admin/config-migration.html for notes on migrating legacy per-component configuration files to the new centralized @/etc/arvados/config.yml@.
To ensure a smooth transition, the per-component config files continue to be read, and take precedence over the centralized configuration. Your cluster should continue to function after upgrade but before doing the full configuration migration. However, several services (keepstore, keep-web, keepproxy) require a minimal `/etc/arvados/config.yml` to start:
You can no longer specify individual keep services to balance via the @config.KeepServiceList@ command line option or @KeepServiceList@ legacy config option. Instead, keep-balance will operate on all keepstore servers with @service_type:disk@ as reported by the @arv keep_service list@ command. If you are still using the legacy config, @KeepServiceList@ should be removed or keep-balance will produce an error.
-Please see the "config migration guide":{{site.baseurl}}/admin/config-migration.html and "keep-balance install guide":{{site.baseurl}}/install/install-keep-balance.html for more details.
+Please see the "config migration guide":https://doc.arvados.org/v2.1/admin/config-migration.html and "keep-balance install guide":{{site.baseurl}}/install/install-keep-balance.html for more details.
h3. Arv-git-httpd configuration migration
-(feature "#14712":https://dev.arvados.org/issues/14712 ) The arv-git-httpd package can now be configured using the centralized configuration file at @/etc/arvados/config.yml@. Configuration via individual command line arguments is no longer available. Please see "arv-git-httpd's config migration guide":{{site.baseurl}}/admin/config-migration.html#arv-git-httpd for more details.
+(feature "#14712":https://dev.arvados.org/issues/14712 ) The arv-git-httpd package can now be configured using the centralized configuration file at @/etc/arvados/config.yml@. Configuration via individual command line arguments is no longer available. Please see "arv-git-httpd's config migration guide":https://doc.arvados.org/v2.1/admin/config-migration.html#arv-git-httpd for more details.
h3. Keepstore and keep-web configuration migration
keep-web now supports the legacy @keep-web.yml@ config format (used by Arvados 1.4) and the new cluster config file format. Please check "keep-web's install guide":{{site.baseurl}}/install/install-keep-web.html for more details.
-keepstore now supports the legacy @keepstore.yml@ config format (used by Arvados 1.4) and the new cluster config file format. Please check the "keepstore config migration notes":{{site.baseurl}}/admin/config-migration.html#keepstore and "keepstore install guide":{{site.baseurl}}/install/install-keepstore.html for more details.
+keepstore now supports the legacy @keepstore.yml@ config format (used by Arvados 1.4) and the new cluster config file format. Please check the "keepstore config migration notes":https://doc.arvados.org/v2.1/admin/config-migration.html#keepstore and "keepstore install guide":{{site.baseurl}}/install/install-keepstore.html for more details.
h3. Keepproxy configuration migration
-(feature "#14715":https://dev.arvados.org/issues/14715 ) Keepproxy can now be configured using the centralized config at @/etc/arvados/config.yml@. Configuration via individual command line arguments is no longer available and the @DisableGet@, @DisablePut@, and @PIDFile@ configuration options are no longer supported. If you are still using the legacy config and @DisableGet@ or @DisablePut@ are set to true or @PIDFile@ has a value, keepproxy will produce an error and fail to start. Please see "keepproxy's config migration guide":{{site.baseurl}}/admin/config-migration.html#keepproxy for more details.
+(feature "#14715":https://dev.arvados.org/issues/14715 ) Keepproxy can now be configured using the centralized config at @/etc/arvados/config.yml@. Configuration via individual command line arguments is no longer available and the @DisableGet@, @DisablePut@, and @PIDFile@ configuration options are no longer supported. If you are still using the legacy config and @DisableGet@ or @DisablePut@ are set to true or @PIDFile@ has a value, keepproxy will produce an error and fail to start. Please see "keepproxy's config migration guide":https://doc.arvados.org/v2.1/admin/config-migration.html#keepproxy for more details.
h3. Delete "keep_services" records
h3. New configuration
-Arvados is migrating to a centralized configuration file for all components. During the migration, legacy configuration files will continue to be loaded. See "Migrating Configuration":config-migration.html for details.
+Arvados is migrating to a centralized configuration file for all components. During the migration, legacy configuration files will continue to be loaded. See "Migrating Configuration":https://doc.arvados.org/v2.1/admin/config-migration.html for details.
h2(#v1_3_3). v1.3.3 (2019-05-14)
# "Run the provision.sh script":#run_provision_script
# "Initial user and login":#initial_user
# "Test the installed cluster running a simple workflow":#test_install
-
-
+# "After the installation":#post_install
h2(#introduction). Introduction
INFO Final process status is success
</code></pre>
</notextile>
+
+h2(#post_install). After the installation
+
+Once the installation is complete, it is recommended to keep a copy of your local configuration files. Committing them to version control is a good idea.
+
+Re-running the Salt-based installer is not recommended for maintaining and upgrading Arvados, please see "Maintenance and upgrading":{{site.baseurl}}/admin/maintenance-and-upgrading.html for more information.
## "DNS configuration (single host / multiple hostnames)":#single_host_multiple_hostnames_dns_configuration
# "Initial user and login":#initial_user
# "Test the installed cluster running a simple workflow":#test_install
+# "After the installation":#post_install
h2(#single_host). Single host install using the provision.sh script
INFO Final process status is success
</code></pre>
</notextile>
+
+h2(#post_install). After the installation
+
+Once the installation is complete, it is recommended to keep a copy of your local configuration files. Committing them to version control is a good idea.
+
+Re-running the Salt-based installer is not recommended for maintaining and upgrading Arvados, please see "Maintenance and upgrading":{{site.baseurl}}/admin/maintenance-and-upgrading.html for more information.
return errors.New("not implemented")
}
+func (fw FileWrapper) Snapshot() (*arvados.Subtree, error) {
+ return nil, errors.New("not implemented")
+}
+
+func (fw FileWrapper) Splice(*arvados.Subtree) error {
+ return errors.New("not implemented")
+}
+
func (client *KeepTestClient) ManifestFileReader(m manifest.Manifest, filename string) (arvados.File, error) {
if filename == hwImageID+".tar" {
rdr := ioutil.NopCloser(&bytes.Buffer{})
package mount
import (
+ "errors"
"io"
"log"
"os"
}
func (fs *keepFS) errCode(err error) int {
- if os.IsNotExist(err) {
+ if err == nil {
+ return 0
+ }
+ if errors.Is(err, os.ErrNotExist) {
return -fuse.ENOENT
}
- switch err {
- case os.ErrExist:
+ if errors.Is(err, os.ErrExist) {
return -fuse.EEXIST
- case arvados.ErrInvalidArgument:
+ }
+ if errors.Is(err, arvados.ErrInvalidArgument) {
return -fuse.EINVAL
- case arvados.ErrInvalidOperation:
+ }
+ if errors.Is(err, arvados.ErrInvalidOperation) {
return -fuse.ENOSYS
- case arvados.ErrDirectoryNotEmpty:
+ }
+ if errors.Is(err, arvados.ErrDirectoryNotEmpty) {
return -fuse.ENOTEMPTY
- case nil:
- return 0
- default:
- return -fuse.EIO
}
+ return -fuse.EIO
}
func (fs *keepFS) Mkdir(path string, mode uint32) int {
# file to determine what version of cwltool and schema-salad to
# build.
install_requires=[
- 'cwltool==3.1.20220210171524',
+ 'cwltool==3.1.20220217190813',
'schema-salad==8.2.20211116214159',
'arvados-python-client{}'.format(pysdk_dep),
'setuptools',
Stat() (os.FileInfo, error)
Truncate(int64) error
Sync() error
+ // Create a snapshot of a file or directory tree, which can
+ // then be spliced onto a different path or a different
+ // collection.
+ Snapshot() (*Subtree, error)
+ // Replace this file or directory with the given snapshot.
+ // The target must be inside a collection: Splice returns an
+ // error if the File is a virtual file or directory like
+ // by_id, a project directory, .arvados#collection,
+ // etc. Splice can replace directories with regular files and
+ // vice versa, except it cannot replace the root directory of
+ // a collection with a regular file.
+ Splice(snapshot *Subtree) error
+}
+
+// A Subtree is a detached part of a filesystem tree that can be
+// spliced into a filesystem via (File)Splice().
+type Subtree struct {
+ inode inode
}
// A FileSystem is an http.Filesystem plus Stat() and support for
Readdir() ([]os.FileInfo, error)
Size() int64
FileInfo() os.FileInfo
+ // Create a snapshot of this node and its descendants.
+ Snapshot() (inode, error)
+ // Replace this node with a copy of the provided snapshot.
+ // Caller may provide the same snapshot to multiple Splice
+ // calls, but must not modify the snapshot concurrently.
+ Splice(inode) error
// Child() performs lookups and updates of named child nodes.
//
return 64
}
+func (*nullnode) Snapshot() (inode, error) {
+ return nil, ErrInvalidOperation
+}
+
+func (*nullnode) Splice(inode) error {
+ return ErrInvalidOperation
+}
+
type treenode struct {
fs FileSystem
parent inode
// supported. Locking inodes from different
// filesystems could deadlock, so we must error out
// now.
- return ErrInvalidArgument
+ return ErrInvalidOperation
}
// To ensure we can test reliably whether we're about to move
func permittedName(name string) bool {
return name != "" && name != "." && name != ".." && !strings.Contains(name, "/")
}
+
+// Snapshot returns a Subtree that's a copy of the given path. It
+// returns an error if the path is not inside a collection.
+func Snapshot(fs FileSystem, path string) (*Subtree, error) {
+ f, err := fs.OpenFile(path, os.O_RDONLY, 0)
+ if err != nil {
+ return nil, err
+ }
+ defer f.Close()
+ return f.Snapshot()
+}
+
+// Splice inserts newsubtree at the indicated target path.
+//
+// Splice returns an error if target is not inside a collection.
+//
+// Splice returns an error if target is the root of a collection and
+// newsubtree is a snapshot of a file.
+func Splice(fs FileSystem, target string, newsubtree *Subtree) error {
+ f, err := fs.OpenFile(target, os.O_WRONLY, 0)
+ if os.IsNotExist(err) {
+ f, err = fs.OpenFile(target, os.O_CREATE|os.O_WRONLY, 0700)
+ }
+ if err != nil {
+ return err
+ }
+ defer f.Close()
+ return f.Splice(newsubtree)
+}
return fs.fileSystem.root.(*dirnode).TreeSize()
}
+func (fs *collectionFileSystem) Snapshot() (inode, error) {
+ return fs.fileSystem.root.Snapshot()
+}
+
+func (fs *collectionFileSystem) Splice(r inode) error {
+ return fs.fileSystem.root.Splice(r)
+}
+
// filenodePtr is an offset into a file that is (usually) efficient to
// seek to. Specifically, if filenode.repacked==filenodePtr.repacked
// then
}
}
+func (fn *filenode) Snapshot() (inode, error) {
+ fn.RLock()
+ defer fn.RUnlock()
+ segments := make([]segment, 0, len(fn.segments))
+ for _, seg := range fn.segments {
+ segments = append(segments, seg.Slice(0, seg.Len()))
+ }
+ return &filenode{
+ fileinfo: fn.fileinfo,
+ segments: segments,
+ }, nil
+}
+
+func (fn *filenode) Splice(repl inode) error {
+ repl, err := repl.Snapshot()
+ if err != nil {
+ return err
+ }
+ fn.parent.Lock()
+ defer fn.parent.Unlock()
+ fn.Lock()
+ defer fn.Unlock()
+ _, err = fn.parent.Child(fn.fileinfo.name, func(inode) (inode, error) { return repl, nil })
+ if err != nil {
+ return err
+ }
+ switch repl := repl.(type) {
+ case *dirnode:
+ repl.parent = fn.parent
+ repl.fileinfo.name = fn.fileinfo.name
+ repl.setTreeFS(fn.fs)
+ case *filenode:
+ repl.parent = fn.parent
+ repl.fileinfo.name = fn.fileinfo.name
+ repl.fs = fn.fs
+ default:
+ return fmt.Errorf("cannot splice snapshot containing %T: %w", repl, ErrInvalidArgument)
+ }
+ return nil
+}
+
type dirnode struct {
fs *collectionFileSystem
treenode
return
}
+func (dn *dirnode) Snapshot() (inode, error) {
+ return dn.snapshot()
+}
+
+func (dn *dirnode) snapshot() (*dirnode, error) {
+ dn.RLock()
+ defer dn.RUnlock()
+ snap := &dirnode{
+ treenode: treenode{
+ inodes: make(map[string]inode, len(dn.inodes)),
+ fileinfo: dn.fileinfo,
+ },
+ }
+ for name, child := range dn.inodes {
+ dupchild, err := child.Snapshot()
+ if err != nil {
+ return nil, err
+ }
+ snap.inodes[name] = dupchild
+ dupchild.SetParent(snap, name)
+ }
+ return snap, nil
+}
+
+func (dn *dirnode) Splice(repl inode) error {
+ repl, err := repl.Snapshot()
+ if err != nil {
+ return err
+ }
+ switch repl := repl.(type) {
+ default:
+ return fmt.Errorf("cannot splice snapshot containing %T: %w", repl, ErrInvalidArgument)
+ case *dirnode:
+ dn.Lock()
+ defer dn.Unlock()
+ dn.inodes = repl.inodes
+ dn.setTreeFS(dn.fs)
+ case *filenode:
+ dn.parent.Lock()
+ defer dn.parent.Unlock()
+ removing, err := dn.parent.Child(dn.fileinfo.name, nil)
+ if err != nil {
+ return fmt.Errorf("cannot use Splice to replace a top-level directory with a file: %w", ErrInvalidOperation)
+ } else if removing != dn {
+ // If ../thisdirname is not this dirnode, it
+ // must be an inode that wraps a dirnode, like
+ // a collectionFileSystem or deferrednode.
+ if deferred, ok := removing.(*deferrednode); ok {
+ // More useful to report the type of
+ // the wrapped node rather than just
+ // *deferrednode. (We know the real
+ // inode is already loaded because dn
+ // is inside it.)
+ removing = deferred.realinode()
+ }
+ return fmt.Errorf("cannot use Splice to attach a file at top level of %T: %w", removing, ErrInvalidOperation)
+ }
+ dn.Lock()
+ defer dn.Unlock()
+ _, err = dn.parent.Child(dn.fileinfo.name, func(inode) (inode, error) { return repl, nil })
+ if err != nil {
+ return err
+ }
+ repl.fs = dn.fs
+ }
+ return nil
+}
+
+func (dn *dirnode) setTreeFS(fs *collectionFileSystem) {
+ dn.fs = fs
+ for _, child := range dn.inodes {
+ switch child := child.(type) {
+ case *dirnode:
+ child.setTreeFS(fs)
+ case *filenode:
+ child.fs = fs
+ }
+ }
+}
+
type segment interface {
io.ReaderAt
Len() int
func (dn *deferrednode) FS() FileSystem { return dn.currentinode().FS() }
func (dn *deferrednode) Parent() inode { return dn.currentinode().Parent() }
func (dn *deferrednode) MemorySize() int64 { return dn.currentinode().MemorySize() }
+func (dn *deferrednode) Snapshot() (inode, error) { return dn.realinode().Snapshot() }
+func (dn *deferrednode) Splice(repl inode) error { return dn.realinode().Splice(repl) }
// Sync the containing filesystem.
return f.FS().Sync()
}
+
+func (f *filehandle) Snapshot() (*Subtree, error) {
+ if !f.readable {
+ return nil, ErrInvalidOperation
+ }
+ node, err := f.inode.Snapshot()
+ return &Subtree{inode: node}, err
+}
+
+func (f *filehandle) Splice(r *Subtree) error {
+ if !f.writable {
+ return ErrReadOnlyFile
+ }
+ return f.inode.Splice(r.inode)
+}
}
func (*getternode) Child(string, func(inode) (inode, error)) (inode, error) {
- return nil, ErrInvalidArgument
+ return nil, ErrInvalidOperation
}
func (gn *getternode) get() error {
return ln.treenode.Readdir()
}
-// Child rejects (with ErrInvalidArgument) calls to add/replace
+// Child rejects (with ErrInvalidOperation) calls to add/replace
// children, instead calling loadOne when a non-existing child is
// looked up.
func (ln *lookupnode) Child(name string, replace func(inode) (inode, error)) (inode, error) {
if replace != nil {
// Let the callback try to delete or replace the
// existing node; if it does, return
- // ErrInvalidArgument.
+ // ErrInvalidOperation.
if tryRepl, err := replace(existing); err != nil {
// Propagate error from callback
return existing, err
} else if tryRepl != existing {
- return existing, ErrInvalidArgument
+ return existing, ErrInvalidOperation
}
}
// Return original error from ln.treenode.Child() (it might be
import (
"bytes"
"encoding/json"
+ "errors"
"io"
"os"
"path/filepath"
s.fs.MountProject("home", "")
_, err := s.fs.OpenFile("/home/A Project/newfilename", os.O_CREATE|os.O_RDWR, 0)
- c.Check(err, check.ErrorMatches, "invalid argument")
+ c.Check(err, ErrorIs, ErrInvalidOperation)
err = s.fs.Mkdir("/home/A Project/newdirname", 0)
- c.Check(err, check.ErrorMatches, "invalid argument")
+ c.Check(err, ErrorIs, ErrInvalidOperation)
err = s.fs.Mkdir("/by_id/newdirname", 0)
- c.Check(err, check.ErrorMatches, "invalid argument")
+ c.Check(err, ErrorIs, ErrInvalidOperation)
err = s.fs.Mkdir("/by_id/"+fixtureAProjectUUID+"/newdirname", 0)
- c.Check(err, check.ErrorMatches, "invalid argument")
+ c.Check(err, ErrorIs, ErrInvalidOperation)
_, err = s.fs.OpenFile("/home/A Project", 0, 0)
c.Check(err, check.IsNil)
}
+
+type errorIsChecker struct {
+ *check.CheckerInfo
+}
+
+var ErrorIs check.Checker = errorIsChecker{
+ &check.CheckerInfo{Name: "ErrorIs", Params: []string{"value", "target"}},
+}
+
+func (checker errorIsChecker) Check(params []interface{}, names []string) (result bool, errStr string) {
+ err, ok := params[0].(error)
+ if !ok {
+ return false, ""
+ }
+ target, ok := params[1].(error)
+ if !ok {
+ return false, ""
+ }
+ return errors.Is(err, target), ""
+}
}
func (fs *customFileSystem) newNode(name string, perm os.FileMode, modTime time.Time) (node inode, err error) {
- return nil, ErrInvalidArgument
+ return nil, ErrInvalidOperation
}
func (fs *customFileSystem) mountByID(parent inode, id string) inode {
}
}
-// vdirnode wraps an inode by rejecting (with ErrInvalidArgument)
+// vdirnode wraps an inode by rejecting (with ErrInvalidOperation)
// calls that add/replace children directly, instead calling a
// create() func when a non-existing child is looked up.
//
} else if tryRepl, err := replace(existing); err != nil {
return existing, err
} else if tryRepl != existing {
- return existing, ErrInvalidArgument
+ return existing, ErrInvalidOperation
} else {
return existing, nil
}
package arvados
import (
+ "fmt"
+ "io"
+ "io/ioutil"
"net/http"
"os"
+ "syscall"
"time"
check "gopkg.in/check.v1"
c.Check(names, check.DeepEquals, []string{"baz"})
_, err = s.fs.OpenFile("/by_id/"+fixtureNonexistentCollection, os.O_RDWR|os.O_CREATE, 0755)
- c.Check(err, check.Equals, ErrInvalidArgument)
+ c.Check(err, ErrorIs, ErrInvalidOperation)
err = s.fs.Rename("/by_id/"+fixtureFooCollection, "/by_id/beep")
- c.Check(err, check.Equals, ErrInvalidArgument)
+ c.Check(err, ErrorIs, ErrInvalidOperation)
err = s.fs.Rename("/by_id/"+fixtureFooCollection+"/foo", "/by_id/beep")
- c.Check(err, check.Equals, ErrInvalidArgument)
+ c.Check(err, ErrorIs, ErrInvalidOperation)
_, err = s.fs.Stat("/by_id/beep")
c.Check(err, check.Equals, os.ErrNotExist)
err = s.fs.Rename("/by_id/"+fixtureFooCollection+"/foo", "/by_id/"+fixtureFooCollection+"/bar")
c.Check(err, check.IsNil)
err = s.fs.Rename("/by_id", "/beep")
- c.Check(err, check.Equals, ErrInvalidArgument)
+ c.Check(err, ErrorIs, ErrInvalidOperation)
+}
+
+// Copy subtree from OS src to dst path inside fs. If src is a
+// directory, dst must exist and be a directory.
+func copyFromOS(fs FileSystem, dst, src string) error {
+ inf, err := os.Open(src)
+ if err != nil {
+ return err
+ }
+ defer inf.Close()
+ dirents, err := inf.Readdir(-1)
+ if e, ok := err.(*os.PathError); ok {
+ if e, ok := e.Err.(syscall.Errno); ok {
+ if e == syscall.ENOTDIR {
+ err = syscall.ENOTDIR
+ }
+ }
+ }
+ if err == syscall.ENOTDIR {
+ outf, err := fs.OpenFile(dst, os.O_CREATE|os.O_EXCL|os.O_TRUNC|os.O_WRONLY, 0700)
+ if err != nil {
+ return fmt.Errorf("open %s: %s", dst, err)
+ }
+ defer outf.Close()
+ _, err = io.Copy(outf, inf)
+ if err != nil {
+ return fmt.Errorf("%s: copying data from %s: %s", dst, src, err)
+ }
+ err = outf.Close()
+ if err != nil {
+ return err
+ }
+ } else if err != nil {
+ return fmt.Errorf("%s: readdir: %T %s", src, err, err)
+ } else {
+ {
+ d, err := fs.Open(dst)
+ if err != nil {
+ return fmt.Errorf("opendir(%s): %s", dst, err)
+ }
+ d.Close()
+ }
+ for _, ent := range dirents {
+ if ent.Name() == "." || ent.Name() == ".." {
+ continue
+ }
+ dstname := dst + "/" + ent.Name()
+ if ent.IsDir() {
+ err = fs.Mkdir(dstname, 0700)
+ if err != nil {
+ return fmt.Errorf("mkdir %s: %s", dstname, err)
+ }
+ }
+ err = copyFromOS(fs, dstname, src+"/"+ent.Name())
+ if err != nil {
+ return err
+ }
+ }
+ }
+ return nil
+}
+
+func (s *SiteFSSuite) TestSnapshotSplice(c *check.C) {
+ s.fs.MountProject("home", "")
+ thisfile, err := ioutil.ReadFile("fs_site_test.go")
+ c.Assert(err, check.IsNil)
+
+ var src1 Collection
+ err = s.client.RequestAndDecode(&src1, "POST", "arvados/v1/collections", nil, map[string]interface{}{
+ "collection": map[string]string{
+ "name": "TestSnapshotSplice src1",
+ "owner_uuid": fixtureAProjectUUID,
+ },
+ })
+ c.Assert(err, check.IsNil)
+ defer s.client.RequestAndDecode(nil, "DELETE", "arvados/v1/collections/"+src1.UUID, nil, nil)
+ err = s.fs.Sync()
+ c.Assert(err, check.IsNil)
+ err = copyFromOS(s.fs, "/home/A Project/TestSnapshotSplice src1", "..") // arvados.git/sdk/go
+ c.Assert(err, check.IsNil)
+
+ var src2 Collection
+ err = s.client.RequestAndDecode(&src2, "POST", "arvados/v1/collections", nil, map[string]interface{}{
+ "collection": map[string]string{
+ "name": "TestSnapshotSplice src2",
+ "owner_uuid": fixtureAProjectUUID,
+ },
+ })
+ c.Assert(err, check.IsNil)
+ defer s.client.RequestAndDecode(nil, "DELETE", "arvados/v1/collections/"+src2.UUID, nil, nil)
+ err = s.fs.Sync()
+ c.Assert(err, check.IsNil)
+ err = copyFromOS(s.fs, "/home/A Project/TestSnapshotSplice src2", "..") // arvados.git/sdk/go
+ c.Assert(err, check.IsNil)
+
+ var dst Collection
+ err = s.client.RequestAndDecode(&dst, "POST", "arvados/v1/collections", nil, map[string]interface{}{
+ "collection": map[string]string{
+ "name": "TestSnapshotSplice dst",
+ "owner_uuid": fixtureAProjectUUID,
+ },
+ })
+ c.Assert(err, check.IsNil)
+ defer s.client.RequestAndDecode(nil, "DELETE", "arvados/v1/collections/"+dst.UUID, nil, nil)
+ err = s.fs.Sync()
+ c.Assert(err, check.IsNil)
+
+ dstPath := "/home/A Project/TestSnapshotSplice dst"
+ err = copyFromOS(s.fs, dstPath, "..") // arvados.git/sdk/go
+ c.Assert(err, check.IsNil)
+
+ // Snapshot directory
+ snap1, err := Snapshot(s.fs, "/home/A Project/TestSnapshotSplice src1/ctxlog")
+ c.Check(err, check.IsNil)
+ // Attach same snapshot twice, at paths that didn't exist before
+ err = Splice(s.fs, dstPath+"/ctxlog-copy", snap1)
+ c.Check(err, check.IsNil)
+ err = Splice(s.fs, dstPath+"/ctxlog-copy2", snap1)
+ c.Check(err, check.IsNil)
+ // Splicing a snapshot twice results in two independent copies
+ err = s.fs.Rename(dstPath+"/ctxlog-copy2/log.go", dstPath+"/ctxlog-copy/log2.go")
+ c.Check(err, check.IsNil)
+ _, err = s.fs.Open(dstPath + "/ctxlog-copy2/log.go")
+ c.Check(err, check.Equals, os.ErrNotExist)
+ f, err := s.fs.Open(dstPath + "/ctxlog-copy/log.go")
+ if c.Check(err, check.IsNil) {
+ buf, err := ioutil.ReadAll(f)
+ c.Check(err, check.IsNil)
+ c.Check(string(buf), check.Not(check.Equals), "")
+ f.Close()
+ }
+
+ // Snapshot regular file
+ snapFile, err := Snapshot(s.fs, "/home/A Project/TestSnapshotSplice src1/arvados/fs_site_test.go")
+ c.Check(err, check.IsNil)
+ // Replace dir with file
+ err = Splice(s.fs, dstPath+"/ctxlog-copy2", snapFile)
+ c.Check(err, check.IsNil)
+ if f, err := s.fs.Open(dstPath + "/ctxlog-copy2"); c.Check(err, check.IsNil) {
+ buf, err := ioutil.ReadAll(f)
+ c.Check(err, check.IsNil)
+ c.Check(string(buf), check.Equals, string(thisfile))
+ }
+
+ // Cannot splice a file onto a collection root, or anywhere
+ // outside a collection
+ for _, badpath := range []string{
+ dstPath,
+ "/home/A Project/newnodename",
+ "/home/A Project",
+ "/home/newnodename",
+ "/home",
+ "/newnodename",
+ } {
+ err = Splice(s.fs, badpath, snapFile)
+ c.Check(err, check.NotNil)
+ c.Check(err, ErrorIs, ErrInvalidOperation, check.Commentf("badpath %s"))
+ if badpath == dstPath {
+ c.Check(err, check.ErrorMatches, `cannot use Splice to attach a file at top level of \*arvados.collectionFileSystem: invalid operation`, check.Commentf("badpath: %s", badpath))
+ continue
+ }
+ err = Splice(s.fs, badpath, snap1)
+ c.Check(err, ErrorIs, ErrInvalidOperation, check.Commentf("badpath %s"))
+ }
+
+ // Destination cannot have trailing slash
+ for _, badpath := range []string{
+ dstPath + "/ctxlog/",
+ dstPath + "/",
+ "/home/A Project/",
+ "/home/",
+ "/",
+ "",
+ } {
+ err = Splice(s.fs, badpath, snap1)
+ c.Check(err, ErrorIs, ErrInvalidArgument, check.Commentf("badpath %s", badpath))
+ err = Splice(s.fs, badpath, snapFile)
+ c.Check(err, ErrorIs, ErrInvalidArgument, check.Commentf("badpath %s", badpath))
+ }
+
+ // Destination's parent must already exist
+ for _, badpath := range []string{
+ dstPath + "/newdirname/",
+ dstPath + "/newdirname/foobar",
+ "/foo/bar",
+ } {
+ err = Splice(s.fs, badpath, snap1)
+ c.Check(err, ErrorIs, os.ErrNotExist, check.Commentf("badpath %s", badpath))
+ err = Splice(s.fs, badpath, snapFile)
+ c.Check(err, ErrorIs, os.ErrNotExist, check.Commentf("badpath %s", badpath))
+ }
+
+ snap2, err := Snapshot(s.fs, dstPath+"/ctxlog-copy")
+ c.Check(err, check.IsNil)
+ err = Splice(s.fs, dstPath+"/ctxlog-copy-copy", snap2)
+ c.Check(err, check.IsNil)
+
+ // Snapshot entire collection, splice into same collection at
+ // a new path, remove file from original location, verify
+ // spliced content survives
+ snapDst, err := Snapshot(s.fs, dstPath+"")
+ c.Check(err, check.IsNil)
+ err = Splice(s.fs, dstPath+"", snapDst)
+ c.Check(err, check.IsNil)
+ err = Splice(s.fs, dstPath+"/copy1", snapDst)
+ c.Check(err, check.IsNil)
+ err = Splice(s.fs, dstPath+"/copy2", snapDst)
+ c.Check(err, check.IsNil)
+ err = s.fs.RemoveAll(dstPath + "/arvados/fs_site_test.go")
+ c.Check(err, check.IsNil)
+ err = s.fs.RemoveAll(dstPath + "/arvados")
+ c.Check(err, check.IsNil)
+ _, err = s.fs.Open(dstPath + "/arvados/fs_site_test.go")
+ c.Check(err, check.Equals, os.ErrNotExist)
+ f, err = s.fs.Open(dstPath + "/copy2/arvados/fs_site_test.go")
+ c.Check(err, check.IsNil)
+ defer f.Close()
+ buf, err := ioutil.ReadAll(f)
+ c.Check(err, check.IsNil)
+ c.Check(string(buf), check.Equals, string(thisfile))
}
return true
}
err = fs.Mkdir(dir, 0755)
- if err == arvados.ErrInvalidArgument {
+ if errors.Is(err, arvados.ErrInvalidArgument) || errors.Is(err, arvados.ErrInvalidOperation) {
// Cannot create a directory
// here.
err = fmt.Errorf("mkdir %q failed: %w", dir, err)
err = bucket.PutReader(trial.path, bytes.NewReader(buf), int64(len(buf)), trial.contentType, s3.Private, s3.Options{})
c.Check(err.(*s3.Error).StatusCode, check.Equals, 400)
c.Check(err.(*s3.Error).Code, check.Equals, `InvalidArgument`)
- c.Check(err, check.ErrorMatches, `(mkdir "/by_id/zzzzz-j7d0g-[a-z0-9]{15}/newdir2?"|open "/zzzzz-j7d0g-[a-z0-9]{15}/newfile") failed: invalid argument`)
+ c.Check(err, check.ErrorMatches, `(mkdir "/by_id/zzzzz-j7d0g-[a-z0-9]{15}/newdir2?"|open "/zzzzz-j7d0g-[a-z0-9]{15}/newfile") failed: invalid (argument|operation)`)
_, err = bucket.GetReader(trial.path)
c.Check(err.(*s3.Error).StatusCode, check.Equals, 404)
"--volume=$PG_DATA:/var/lib/postgresql:rw" \
"--volume=$VAR_DATA:$ARVADOS_CONTAINER_PATH:rw" \
"--volume=$PASSENGER:/var/lib/passenger:rw" \
- "--volume=$GEMS:/var/lib/arvados/lib/ruby/gems:rw" \
+ "--volume=$GEMS:/var/lib/arvados-arvbox/.gem:rw" \
"--volume=$PIPCACHE:/var/lib/pip:rw" \
"--volume=$NPMCACHE:/var/lib/npm:rw" \
"--volume=$GOSTUFF:/var/lib/gopath:rw" \
export R_LIBS=/var/lib/Rlibs
export HOME=$(getent passwd arvbox | cut -d: -f6)
export ARVADOS_CONTAINER_PATH=/var/lib/arvados-arvbox
-GEMLOCK=/var/lib/arvados/lib/ruby/gems/gems.lock
+export GEM_HOME=$HOME/.gem
+GEMLOCK=$GEM_HOME/gems.lock
defaultdev=$(/sbin/ip route|awk '/default/ { print $5 }')
dockerip=$(/sbin/ip route | grep default | awk '{ print $3 }')
fi
run_bundler() {
- flock $GEMLOCK /var/lib/arvados/bin/gem install --no-document bundler:$BUNDLER_VERSION
+ flock $GEMLOCK /var/lib/arvados/bin/gem install --no-document --user bundler:$BUNDLER_VERSION
if test -f Gemfile.lock ; then
frozen=--frozen
else
if ! [[ -z "$waiting" ]] ; then
if ps x | grep -v grep | grep "bundle install" > /dev/null; then
- gemcount=$(ls /var/lib/arvados/lib/ruby/gems/*/gems 2>/dev/null | wc -l)
+ gemcount=$(ls /var/lib/arvados/lib/ruby/gems/*/gems /var/lib/arvados-arvbox/.gem/ruby/*/gems 2>/dev/null | wc -l)
gemlockcount=0
for l in /usr/src/arvados/services/api/Gemfile.lock \
projects / arvados.git / commitdiff