Closes #21453.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
arv-put --version >/dev/null
-PYTHON=`ls /usr/share/python3*/dist/python3-arvados-python-client/bin/python3 |head -n1`
-
-$PYTHON << EOF
+/usr/lib/python3-arvados-python-client/bin/python <<EOF
import arvados
print("Successfully imported arvados")
EOF
cd $WORKSPACE/$PKG_DIR
rm -rf dist/*
- local venv_dir="dist/build/usr/share/python$PYTHON3_VERSION/dist/$PYTHON_PKG"
+ local venv_dir="dist/build/usr/lib/$PYTHON_PKG"
echo "Creating virtualenv..."
if ! "$PYTHON3_EXECUTABLE" -m venv "$venv_dir"; then
printf "Error, unable to run\n %s -m venv %s\n" "$PYTHON3_EXECUTABLE" "$venv_dir"
# make sure the systemd service file ends up in the right place
# used by arvados-docker-cleaner
if [[ -e "${systemd_unit}" ]]; then
- COMMAND_ARR+=("usr/share/python$PYTHON3_VERSION/dist/$PKG/share/doc/$PKG/$PKG.service=/lib/systemd/system/$PKG.service")
+ COMMAND_ARR+=("$sys_venv_dir/share/doc/$PKG/$PKG.service=/lib/systemd/system/$PKG.service")
fi
COMMAND_ARR+=("${fpm_args[@]}")
- # Make sure to install all our package binaries in /usr/bin.
- # We have to walk $WORKSPACE/$PKG_DIR/bin rather than
- # $WORKSPACE/build/usr/share/$python/dist/$PYTHON_PKG/bin/ to get the list
- # because the latter also includes all the python binaries for the virtualenv.
- # We have to take the copies of our binaries from the latter directory, though,
- # because those are the ones we rewrote the shebang line of, above.
+ # Make sure to install all our package binaries in /usr/bin. We have to
+ # walk $WORKSPACE/$PKG_DIR/bin rather than $venv_dir/bin to get the list
+ # because the latter also includes scripts installed by all the
+ # dependencies in the virtualenv, which may conflict with other
+ # packages. We have to take the copies of our binaries from the latter
+ # directory, though, because those are the ones we rewrote the shebang
+ # line of, above.
if [[ -e "$WORKSPACE/$PKG_DIR/bin" ]]; then
for binary in `ls $WORKSPACE/$PKG_DIR/bin`; do
COMMAND_ARR+=("$sys_venv_dir/bin/$binary=/usr/bin/")
"previous: Upgrading to 2.7.1":#v2_7_1
+h3. Virtual environments inside distribution Python packages have moved
+
+The distribution packages that we publish for Python packages include an entire virtualenv with all required libraries. In Arvados 3.0 these virtualenvs have moved from @/usr/share/python3/dist/PACKAGE_NAME@ to @/usr/lib/PACKAGE_NAME@ to prevent conflicts with distribution packages and better conform to filesystem standards.
+
+If you only run the executables installed by these packages, you don't need to change anything. Those are still installed under @/usr/bin@ and will use the new location when you upgrade. If you have written your own scripts or tools that rely on these virtualenvs, you may need to update those with the new location. For example, if you have a shell script that activates the virtualenv by running:
+
+<pre><code class="shell">source /usr/share/python3/dist/python3-arvados-python-client/bin/activate</code></pre>
+
+You must update it to:
+
+<notextile>
+<pre><code class="shell">source <span class="userinput">/usr/lib/python3-arvados-python-client</span>/bin/activate</code></pre>
+</notextile>
+
+If you have a Python script with this shebang line:
+
+<pre><code class="shell">#!/usr/share/python3/dist/python3-arvados-python-client/bin/python</code></pre>
+
+You must update it to:
+
+<notextile>
+<pre><code class="shell">#!<span class="userinput">/usr/lib/python3-arvados-python-client</span>/bin/python</code></pre>
+</notextile>
+
h3. WebDAV service uses @/var/cache@ for file content
@keep-web@ now stores copies of recently accessed data blocks in @/var/cache/arvados/keep@ instead of in memory. That directory will be created automatically. The default cache size is 10% of the filesystem size. Use the new @Collections.WebDAVCache.DiskCacheSize@ config to specify a different percentage or an absolute size.
To use the Python SDK elsewhere, you can install it "from an Arvados distribution package":#package-install or "from PyPI using pip":#pip-install.
{% include 'notebox_begin_warning' %}
-As of Arvados 2.2, the Python SDK requires Python 3.6+. The last version to support Python 2.7 is Arvados 2.0.4.
+As of Arvados 3.0, the Python SDK requires Python 3.8+.
{% include 'notebox_end' %}
h2(#package-install). Install from a distribution package
{% include 'install_packages' %}
-{% include 'notebox_begin_warning' %}
-If you are on Ubuntu 18.04, please note that the Arvados packages that use Python depend on the python-3.8 package. This means they are installed under @/usr/share/python3.8@, not @/usr/share/python3@. You will need to update the commands below accordingly.
-{% include 'notebox_end' %}
-
The package includes a virtualenv, which means the correct Python environment needs to be loaded before the Arvados SDK can be imported. You can test the installation by doing that, then creating a client object. Ensure your "@ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ credentials are set up correctly":{{site.baseurl}}/user/reference/api-tokens.html. Then you should be able to run the following without any errors:
<notextile>
-<pre>~$ <code class="userinput">source /usr/share/python3/dist/python3-arvados-python-client/bin/activate</code>
+<pre>~$ <code class="userinput">source /usr/lib/python3-arvados-python-client/bin/activate</code>
(python-arvados-python-client) ~$ <code class="userinput">python</code>
Python 3.7.3 (default, Jul 25 2020, 13:03:44)
[GCC 8.3.0] on linux
Alternatively, you can run the Python executable inside the @virtualenv@ directly:
<notextile>
-<pre>~$ <code class="userinput">/usr/share/python3/dist/python3-arvados-python-client/bin/python</code>
+<pre>~$ <code class="userinput">/usr/lib/python3-arvados-python-client/bin/python</code>
Python 3.7.3 (default, Jul 25 2020, 13:03:44)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
/usr/bin/install --directory --owner=crunch --group=crunch --mode=0700 /keep /tmp/crunch-src /tmp/crunch-job
USER crunch
-ENV PATH=/usr/share/python3.9/dist/python3-arvados-cwl-runner/bin:/usr/local/bin:/usr/bin:/bin
+ENV PATH=/usr/lib/python3-arvados-cwl-runner/bin:/usr/local/bin:/usr/bin:/bin
fi
# Use the python executor associated with the installed OS package, if present.
-python=$(((ls /usr/share/python3*/dist/python3-arvados-cwl-runner/bin/python || ls /opt/arvados-py/bin/python || echo python3) | head -n1) 2>/dev/null)
+python="$(PATH="/usr/lib/python3-arvados-cwl-runner/bin:/opt/arvados-py/bin:$PATH" command -v python3)"
# Test for #18888
# This is a standalone test because the bug was observed with this
projects / arvados.git / commitdiff