From: Tomasz Marciniak Date: Wed, 8 Aug 2018 11:45:06 +0000 (+0200) Subject: Arvados SDK Java - release 2.0.0 X-Git-Tag: 1.4.0~55^2~18^2~2 X-Git-Url: https://git.arvados.org/arvados.git/commitdiff_plain/22ae07f51e144baa27ddc56d56a8c6ac6f63ad58 Arvados SDK Java - release 2.0.0 Arvados-DCO-1.1-Signed-off-by: Tomasz Marciniak --- 22ae07f51e144baa27ddc56d56a8c6ac6f63ad58 diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000000..c928081f78 --- /dev/null +++ b/.gitignore @@ -0,0 +1,9 @@ +/.gradle/ +/bin/ +/build/ +.project +.classpath +/.settings/ +.DS_Store +/.idea/ +/out/ diff --git a/README.md b/README.md new file mode 100644 index 0000000000..ca5aef91c1 --- /dev/null +++ b/README.md @@ -0,0 +1,115 @@ +``` +Copyright (C) The Arvados Authors. All rights reserved. + +SPDX-License-Identifier: CC-BY-SA-3.0 +``` + +# Arvados Java SDK + +##### About +Arvados Java Client allows to access Arvados servers and uses two APIs: +* lower level [Keep Server API](https://doc.arvados.org/api/index.html) +* higher level [Keep-Web API](https://godoc.org/github.com/curoverse/arvados/services/keep-web) (when needed) + +##### Required Java version +This SDK requires Java 8+ + +##### Logging + +SLF4J is used for logging. Concrete logging framework and configuration must be provided by a client. + +##### Configuration + +[TypeSafe Configuration](https://github.com/lightbend/config) is used for configuring this library. + +Please, have a look at java/resources/reference.conf for default values provided with this library. + +* **keepweb-host** - change to host of your Keep-Web installation +* **keepweb-port** - change to port of your Keep-Web installation +* **host** - change to host of your Arvados installation +* **port** - change to port of your Arvados installation +* **token** - authenticates registered user, one must provide + [token obtained from Arvados Workbench](https://doc.arvados.org/user/reference/api-tokens.html) +* **protocol** - don't change to unless really needed +* **host-insecure** - insecure communication with Arvados (ignores SSL certificate verification), + don't change to *true* unless really needed +* **split-size** - size of chunk files in megabytes +* **temp-dir** - temporary chunk files storage +* **copies** - amount of chunk files duplicates per Keep server +* **retries** - in case of chunk files send failure this should allow to repeat send + (*NOTE*: this parameter is not used at the moment but was left for future improvements) + +In order to override default settings one can create application.conf file in an application. +Example: src/test/resources/application.conf. + +Alternatively ExternalConfigProvider class can be used to pass configuration via code. +ExternalConfigProvider comes with a builder and all of the above values must be provided in order for it to work properly. + +ArvadosFacade has two constructors, one without arguments that uses values from reference.conf and second one +taking ExternalConfigProvider as an argument. + +##### API clients + +All API clients inherit from BaseStandardApiClient. This class contains implementation of all +common methods as described in http://doc.arvados.org/api/methods.html. + +Parameters provided to common or specific methods are String UUID or fields wrapped in Java objects. For example: + +```java +String uuid = "ardev-4zz18-rxcql7qwyakg1r1"; + +Collection actual = client.get(uuid); +``` + +```java +ListArgument listArgument = ListArgument.builder() + .filters(Arrays.asList( + Filter.of("owner_uuid", Operator.LIKE, "ardev%"), + Filter.of("name", Operator.LIKE, "Super%"), + Filter.of("portable_data_hash", Operator.IN, Lists.newArrayList("54f6d9f59065d3c009d4306660989379+65") + ))) + .build(); + +CollectionList actual = client.list(listArgument); +``` + +Non-standard API clients must inherit from BaseApiClient. +For example: KeepServerApiClient communicates directly with Keep servers using exclusively non-common methods. + +##### Business logic + +More advanced API data handling could be implemented as *Facade* classes. +In current version functionalities provided by SDK are handled by *ArvadosFacade*. +They include: +* **downloading single file from collection** - using Keep-Web +* **downloading whole collection** - using Keep-Web or Keep Server API +* **listing file info from certain collection** - information is returned as list of *FileTokens* providing file details +* **uploading single file** - to either new or existing collection +* **uploading list of files** - to either new or existing collection +* **creating an empty collection** +* **getting current user info** +* **listing current user's collections** +* **creating new project** +* **deleting certain collection** + +##### Note regarding Keep-Web + +Current version requires both Keep Web and standard Keep Server API configured in order to use Keep-Web functionalities. + +##### Integration tests + +In order to run integration tests all fields within following configuration file must be provided: +```java +src/test/resources/integration-test-appliation.conf +``` +Parameter **integration-tests.project-uuid** should contain UUID of one project available to user, +whose token was provided within configuration file. + +Integration tests require connection to real Arvados server. + +##### Note regarding file naming + +While uploading via this SDK all uploaded files within single collection must have different names. +This applies also to uploading files to already existing collection. +Renaming files with duplicate names is not implemented in current version. + diff --git a/build.gradle b/build.gradle new file mode 100644 index 0000000000..eeec33369b --- /dev/null +++ b/build.gradle @@ -0,0 +1,50 @@ +apply plugin: 'java-library' +apply plugin: 'eclipse' +apply plugin: 'idea' +apply plugin: 'maven' + +version = '2.0.0' + +repositories { + mavenCentral() +} + +dependencies { + api 'com.squareup.okhttp3:okhttp:3.9.1' + api 'com.fasterxml.jackson.core:jackson-databind:2.9.2' + api 'com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.9.2' + api 'com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.9.2' + api 'commons-codec:commons-codec:1.11' + api 'commons-io:commons-io:2.6' + api 'com.google.guava:guava:23.4-jre' + api 'org.slf4j:slf4j-api:1.7.25' + api 'com.typesafe:config:1.3.2' + + testImplementation 'junit:junit:4.12' + testImplementation 'org.mockito:mockito-core:2.12.0' + testImplementation 'org.assertj:assertj-core:3.8.0' + testImplementation 'com.squareup.okhttp3:mockwebserver:3.9.1' +} + +test { + useJUnit { + excludeCategories 'org.arvados.client.junit.categories.IntegrationTests' + } + + testLogging { + events "passed", "skipped", "failed" + afterSuite { desc, result -> + if (!desc.parent) { // will match the outermost suite + println "\n---- Test results ----" + println "${result.resultType} (${result.testCount} tests, ${result.successfulTestCount} successes, ${result.failedTestCount} failures, ${result.skippedTestCount} skipped)" + println "" + } + } + } +} + +task integrationTest(type: Test) { + useJUnit { + includeCategories 'org.arvados.client.junit.categories.IntegrationTests' + } +} diff --git a/gradle/wrapper/gradle-wrapper.jar b/gradle/wrapper/gradle-wrapper.jar new file mode 100644 index 0000000000..27768f1bba Binary files /dev/null and b/gradle/wrapper/gradle-wrapper.jar differ diff --git a/gradle/wrapper/gradle-wrapper.properties b/gradle/wrapper/gradle-wrapper.properties new file mode 100644 index 0000000000..9d2dc020a2 --- /dev/null +++ b/gradle/wrapper/gradle-wrapper.properties @@ -0,0 +1,5 @@ +distributionBase=GRADLE_USER_HOME +distributionPath=wrapper/dists +zipStoreBase=GRADLE_USER_HOME +zipStorePath=wrapper/dists +distributionUrl=https\://services.gradle.org/distributions/gradle-4.8-bin.zip diff --git a/gradlew b/gradlew new file mode 100755 index 0000000000..cccdd3d517 --- /dev/null +++ b/gradlew @@ -0,0 +1,172 @@ +#!/usr/bin/env sh + +############################################################################## +## +## Gradle start up script for UN*X +## +############################################################################## + +# Attempt to set APP_HOME +# Resolve links: $0 may be a link +PRG="$0" +# Need this for relative symlinks. +while [ -h "$PRG" ] ; do + ls=`ls -ld "$PRG"` + link=`expr "$ls" : '.*-> \(.*\)$'` + if expr "$link" : '/.*' > /dev/null; then + PRG="$link" + else + PRG=`dirname "$PRG"`"/$link" + fi +done +SAVED="`pwd`" +cd "`dirname \"$PRG\"`/" >/dev/null +APP_HOME="`pwd -P`" +cd "$SAVED" >/dev/null + +APP_NAME="Gradle" +APP_BASE_NAME=`basename "$0"` + +# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script. +DEFAULT_JVM_OPTS="" + +# Use the maximum available, or set MAX_FD != -1 to use that value. +MAX_FD="maximum" + +warn () { + echo "$*" +} + +die () { + echo + echo "$*" + echo + exit 1 +} + +# OS specific support (must be 'true' or 'false'). +cygwin=false +msys=false +darwin=false +nonstop=false +case "`uname`" in + CYGWIN* ) + cygwin=true + ;; + Darwin* ) + darwin=true + ;; + MINGW* ) + msys=true + ;; + NONSTOP* ) + nonstop=true + ;; +esac + +CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar + +# Determine the Java command to use to start the JVM. +if [ -n "$JAVA_HOME" ] ; then + if [ -x "$JAVA_HOME/jre/sh/java" ] ; then + # IBM's JDK on AIX uses strange locations for the executables + JAVACMD="$JAVA_HOME/jre/sh/java" + else + JAVACMD="$JAVA_HOME/bin/java" + fi + if [ ! -x "$JAVACMD" ] ; then + die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME + +Please set the JAVA_HOME variable in your environment to match the +location of your Java installation." + fi +else + JAVACMD="java" + which java >/dev/null 2>&1 || die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. + +Please set the JAVA_HOME variable in your environment to match the +location of your Java installation." +fi + +# Increase the maximum file descriptors if we can. +if [ "$cygwin" = "false" -a "$darwin" = "false" -a "$nonstop" = "false" ] ; then + MAX_FD_LIMIT=`ulimit -H -n` + if [ $? -eq 0 ] ; then + if [ "$MAX_FD" = "maximum" -o "$MAX_FD" = "max" ] ; then + MAX_FD="$MAX_FD_LIMIT" + fi + ulimit -n $MAX_FD + if [ $? -ne 0 ] ; then + warn "Could not set maximum file descriptor limit: $MAX_FD" + fi + else + warn "Could not query maximum file descriptor limit: $MAX_FD_LIMIT" + fi +fi + +# For Darwin, add options to specify how the application appears in the dock +if $darwin; then + GRADLE_OPTS="$GRADLE_OPTS \"-Xdock:name=$APP_NAME\" \"-Xdock:icon=$APP_HOME/media/gradle.icns\"" +fi + +# For Cygwin, switch paths to Windows format before running java +if $cygwin ; then + APP_HOME=`cygpath --path --mixed "$APP_HOME"` + CLASSPATH=`cygpath --path --mixed "$CLASSPATH"` + JAVACMD=`cygpath --unix "$JAVACMD"` + + # We build the pattern for arguments to be converted via cygpath + ROOTDIRSRAW=`find -L / -maxdepth 1 -mindepth 1 -type d 2>/dev/null` + SEP="" + for dir in $ROOTDIRSRAW ; do + ROOTDIRS="$ROOTDIRS$SEP$dir" + SEP="|" + done + OURCYGPATTERN="(^($ROOTDIRS))" + # Add a user-defined pattern to the cygpath arguments + if [ "$GRADLE_CYGPATTERN" != "" ] ; then + OURCYGPATTERN="$OURCYGPATTERN|($GRADLE_CYGPATTERN)" + fi + # Now convert the arguments - kludge to limit ourselves to /bin/sh + i=0 + for arg in "$@" ; do + CHECK=`echo "$arg"|egrep -c "$OURCYGPATTERN" -` + CHECK2=`echo "$arg"|egrep -c "^-"` ### Determine if an option + + if [ $CHECK -ne 0 ] && [ $CHECK2 -eq 0 ] ; then ### Added a condition + eval `echo args$i`=`cygpath --path --ignore --mixed "$arg"` + else + eval `echo args$i`="\"$arg\"" + fi + i=$((i+1)) + done + case $i in + (0) set -- ;; + (1) set -- "$args0" ;; + (2) set -- "$args0" "$args1" ;; + (3) set -- "$args0" "$args1" "$args2" ;; + (4) set -- "$args0" "$args1" "$args2" "$args3" ;; + (5) set -- "$args0" "$args1" "$args2" "$args3" "$args4" ;; + (6) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" ;; + (7) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" ;; + (8) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" ;; + (9) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" "$args8" ;; + esac +fi + +# Escape application args +save () { + for i do printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/" ; done + echo " " +} +APP_ARGS=$(save "$@") + +# Collect all arguments for the java command, following the shell quoting and substitution rules +eval set -- $DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS "\"-Dorg.gradle.appname=$APP_BASE_NAME\"" -classpath "\"$CLASSPATH\"" org.gradle.wrapper.GradleWrapperMain "$APP_ARGS" + +# by default we should be in the correct project dir, but when run from Finder on Mac, the cwd is wrong +if [ "$(uname)" = "Darwin" ] && [ "$HOME" = "$PWD" ]; then + cd "$(dirname "$0")" +fi + +exec "$JAVACMD" "$@" diff --git a/gradlew.bat b/gradlew.bat new file mode 100644 index 0000000000..f9553162f1 --- /dev/null +++ b/gradlew.bat @@ -0,0 +1,84 @@ +@if "%DEBUG%" == "" @echo off +@rem ########################################################################## +@rem +@rem Gradle startup script for Windows +@rem +@rem ########################################################################## + +@rem Set local scope for the variables with windows NT shell +if "%OS%"=="Windows_NT" setlocal + +set DIRNAME=%~dp0 +if "%DIRNAME%" == "" set DIRNAME=. +set APP_BASE_NAME=%~n0 +set APP_HOME=%DIRNAME% + +@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script. +set DEFAULT_JVM_OPTS= + +@rem Find java.exe +if defined JAVA_HOME goto findJavaFromJavaHome + +set JAVA_EXE=java.exe +%JAVA_EXE% -version >NUL 2>&1 +if "%ERRORLEVEL%" == "0" goto init + +echo. +echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. +echo. +echo Please set the JAVA_HOME variable in your environment to match the +echo location of your Java installation. + +goto fail + +:findJavaFromJavaHome +set JAVA_HOME=%JAVA_HOME:"=% +set JAVA_EXE=%JAVA_HOME%/bin/java.exe + +if exist "%JAVA_EXE%" goto init + +echo. +echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% +echo. +echo Please set the JAVA_HOME variable in your environment to match the +echo location of your Java installation. + +goto fail + +:init +@rem Get command-line arguments, handling Windows variants + +if not "%OS%" == "Windows_NT" goto win9xME_args + +:win9xME_args +@rem Slurp the command line arguments. +set CMD_LINE_ARGS= +set _SKIP=2 + +:win9xME_args_slurp +if "x%~1" == "x" goto execute + +set CMD_LINE_ARGS=%* + +:execute +@rem Setup the command line + +set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar + +@rem Execute Gradle +"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %CMD_LINE_ARGS% + +:end +@rem End local scope for the variables with windows NT shell +if "%ERRORLEVEL%"=="0" goto mainEnd + +:fail +rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of +rem the _cmd.exe /c_ return code! +if not "" == "%GRADLE_EXIT_CONSOLE%" exit 1 +exit /b 1 + +:mainEnd +if "%OS%"=="Windows_NT" endlocal + +:omega diff --git a/settings.gradle b/settings.gradle new file mode 100644 index 0000000000..be8ccc6ca4 --- /dev/null +++ b/settings.gradle @@ -0,0 +1 @@ +rootProject.name = 'arvados-java' diff --git a/src/main/java/org/arvados/client/api/client/BaseApiClient.java b/src/main/java/org/arvados/client/api/client/BaseApiClient.java new file mode 100644 index 0000000000..7e8a2979be --- /dev/null +++ b/src/main/java/org/arvados/client/api/client/BaseApiClient.java @@ -0,0 +1,83 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.client; + +import com.fasterxml.jackson.databind.ObjectMapper; +import org.arvados.client.exception.ArvadosApiException; +import org.arvados.client.api.client.factory.OkHttpClientFactory; +import org.arvados.client.api.model.ApiError; +import org.arvados.client.config.ConfigProvider; +import okhttp3.OkHttpClient; +import okhttp3.Request; +import okhttp3.Response; +import okhttp3.ResponseBody; +import org.slf4j.Logger; + +import java.io.IOException; +import java.io.UnsupportedEncodingException; +import java.net.URLDecoder; +import java.nio.charset.StandardCharsets; +import java.util.Objects; + +abstract class BaseApiClient { + + static final ObjectMapper MAPPER = new ObjectMapper().findAndRegisterModules(); + + final OkHttpClient client; + final ConfigProvider config; + private final Logger log = org.slf4j.LoggerFactory.getLogger(BaseApiClient.class); + + BaseApiClient(ConfigProvider config) { + this.config = config; + client = OkHttpClientFactory.builder() + .build() + .create(config.isApiHostInsecure()); + } + + Request.Builder getRequestBuilder() { + return new Request.Builder() + .addHeader("authorization", String.format("OAuth2 %s", config.getApiToken())) + .addHeader("cache-control", "no-cache"); + } + + String newCall(Request request) { + return (String) getResponseBody(request, body -> body.string().trim()); + } + + byte[] newFileCall(Request request) { + return (byte[]) getResponseBody(request, ResponseBody::bytes); + } + + private Object getResponseBody(Request request, Command command) { + try { + log.debug(URLDecoder.decode(request.toString(), StandardCharsets.UTF_8.name())); + } catch (UnsupportedEncodingException e) { + throw new ArvadosApiException(e); + } + + try (Response response = client.newCall(request).execute()) { + ResponseBody responseBody = response.body(); + + if (!response.isSuccessful()) { + String errorBody = Objects.requireNonNull(responseBody).string(); + if (errorBody == null || errorBody.length() == 0) { + throw new ArvadosApiException(String.format("Error code %s with message: %s", response.code(), response.message())); + } + ApiError apiError = MAPPER.readValue(errorBody, ApiError.class); + throw new ArvadosApiException(String.format("Error code %s with messages: %s", response.code(), apiError.getErrors())); + } + return command.readResponseBody(responseBody); + } catch (IOException e) { + throw new ArvadosApiException(e); + } + } + + private interface Command { + Object readResponseBody(ResponseBody body) throws IOException; + } +} diff --git a/src/main/java/org/arvados/client/api/client/BaseStandardApiClient.java b/src/main/java/org/arvados/client/api/client/BaseStandardApiClient.java new file mode 100644 index 0000000000..ab03d34f19 --- /dev/null +++ b/src/main/java/org/arvados/client/api/client/BaseStandardApiClient.java @@ -0,0 +1,153 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.client; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.databind.ObjectWriter; +import okhttp3.MediaType; +import okhttp3.HttpUrl; +import okhttp3.HttpUrl.Builder; +import okhttp3.Request; +import okhttp3.RequestBody; +import org.arvados.client.exception.ArvadosApiException; +import org.arvados.client.api.model.Item; +import org.arvados.client.api.model.ItemList; +import org.arvados.client.api.model.argument.ListArgument; +import org.arvados.client.config.ConfigProvider; +import org.slf4j.Logger; + +import java.io.IOException; +import java.util.Map; + +public abstract class BaseStandardApiClient extends BaseApiClient { + + private static final MediaType JSON = MediaType.parse(com.google.common.net.MediaType.JSON_UTF_8.toString()); + private final Logger log = org.slf4j.LoggerFactory.getLogger(BaseStandardApiClient.class); + + BaseStandardApiClient(ConfigProvider config) { + super(config); + } + + public L list(ListArgument listArguments) { + log.debug("Get list of {}", getType().getSimpleName()); + Builder urlBuilder = getUrlBuilder(); + addQueryParameters(urlBuilder, listArguments); + HttpUrl url = urlBuilder.build(); + Request request = getRequestBuilder().url(url).build(); + return callForList(request); + } + + public L list() { + return list(ListArgument.builder().build()); + } + + public T get(String uuid) { + log.debug("Get {} by UUID {}", getType().getSimpleName(), uuid); + HttpUrl url = getUrlBuilder().addPathSegment(uuid).build(); + Request request = getRequestBuilder().get().url(url).build(); + return callForType(request); + } + + public T create(T type) { + log.debug("Create {}", getType().getSimpleName()); + String json = mapToJson(type); + RequestBody body = RequestBody.create(JSON, json); + Request request = getRequestBuilder().post(body).build(); + return callForType(request); + } + + public T delete(String uuid) { + log.debug("Delete {} by UUID {}", getType().getSimpleName(), uuid); + HttpUrl url = getUrlBuilder().addPathSegment(uuid).build(); + Request request = getRequestBuilder().delete().url(url).build(); + return callForType(request); + } + + public T update(T type) { + String uuid = type.getUuid(); + log.debug("Update {} by UUID {}", getType().getSimpleName(), uuid); + String json = mapToJson(type); + RequestBody body = RequestBody.create(JSON, json); + HttpUrl url = getUrlBuilder().addPathSegment(uuid).build(); + Request request = getRequestBuilder().put(body).url(url).build(); + return callForType(request); + } + + @Override + Request.Builder getRequestBuilder() { + return super.getRequestBuilder().url(getUrlBuilder().build()); + } + + HttpUrl.Builder getUrlBuilder() { + return new HttpUrl.Builder() + .scheme(config.getApiProtocol()) + .host(config.getApiHost()) + .port(config.getApiPort()) + .addPathSegment("arvados") + .addPathSegment("v1") + .addPathSegment(getResource()); + } + + TL call(Request request, Class cls) { + String bodyAsString = newCall(request); + try { + return mapToObject(bodyAsString, cls); + } catch (IOException e) { + throw new ArvadosApiException("A problem occurred while parsing JSON data", e); + } + } + + private TL mapToObject(String content, Class cls) throws IOException { + return MAPPER.readValue(content, cls); + } + + private String mapToJson(TL type) { + ObjectWriter writer = MAPPER.writer().withDefaultPrettyPrinter(); + try { + return writer.writeValueAsString(type); + } catch (JsonProcessingException e) { + log.error(e.getMessage()); + return null; + } + } + + T callForType(Request request) { + return call(request, getType()); + } + + L callForList(Request request) { + return call(request, getListType()); + } + + abstract String getResource(); + + abstract Class getType(); + + abstract Class getListType(); + + Request getNoArgumentMethodRequest(String method) { + HttpUrl url = getUrlBuilder().addPathSegment(method).build(); + return getRequestBuilder().get().url(url).build(); + } + + RequestBody getJsonRequestBody(Object object) { + return RequestBody.create(JSON, mapToJson(object)); + } + + void addQueryParameters(Builder urlBuilder, Object object) { + Map queryMap = MAPPER.convertValue(object, new TypeReference>() {}); + queryMap.keySet().forEach(key -> { + Object type = queryMap.get(key); + if (!(type instanceof String)) { + type = mapToJson(type); + } + urlBuilder.addQueryParameter(key, (String) type); + }); + } +} diff --git a/src/main/java/org/arvados/client/api/client/CollectionsApiClient.java b/src/main/java/org/arvados/client/api/client/CollectionsApiClient.java new file mode 100644 index 0000000000..141f02deba --- /dev/null +++ b/src/main/java/org/arvados/client/api/client/CollectionsApiClient.java @@ -0,0 +1,45 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.client; + +import org.arvados.client.api.model.Collection; +import org.arvados.client.api.model.CollectionList; +import org.arvados.client.config.ConfigProvider; +import org.slf4j.Logger; + +public class CollectionsApiClient extends BaseStandardApiClient { + + private static final String RESOURCE = "collections"; + private final Logger log = org.slf4j.LoggerFactory.getLogger(CollectionsApiClient.class); + + public CollectionsApiClient(ConfigProvider config) { + super(config); + } + + @Override + public Collection create(Collection type) { + Collection newCollection = super.create(type); + log.debug(String.format("New collection '%s' with UUID %s has been created", newCollection.getName(), newCollection.getUuid())); + return newCollection; + } + + @Override + String getResource() { + return RESOURCE; + } + + @Override + Class getType() { + return Collection.class; + } + + @Override + Class getListType() { + return CollectionList.class; + } +} diff --git a/src/main/java/org/arvados/client/api/client/CountingFileRequestBody.java b/src/main/java/org/arvados/client/api/client/CountingFileRequestBody.java new file mode 100644 index 0000000000..43fcdba5c6 --- /dev/null +++ b/src/main/java/org/arvados/client/api/client/CountingFileRequestBody.java @@ -0,0 +1,84 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.client; + +import okhttp3.MediaType; +import okhttp3.RequestBody; +import okio.BufferedSink; +import okio.Okio; +import okio.Source; +import org.slf4j.Logger; + +import java.io.File; + +/** + * Based on: + * {@link} https://gist.github.com/eduardb/dd2dc530afd37108e1ac + */ +public class CountingFileRequestBody extends RequestBody { + + private static final int SEGMENT_SIZE = 2048; // okio.Segment.SIZE + private static final MediaType CONTENT_BINARY = MediaType.parse(com.google.common.net.MediaType.OCTET_STREAM.toString()); + + private final File file; + private final ProgressListener listener; + + CountingFileRequestBody(final File file, final ProgressListener listener) { + this.file = file; + this.listener = listener; + } + + @Override + public long contentLength() { + return file.length(); + } + + @Override + public MediaType contentType() { + return CONTENT_BINARY; + } + + @Override + public void writeTo(BufferedSink sink) { + try (Source source = Okio.source(file)) { + long total = 0; + long read; + + while ((read = source.read(sink.buffer(), SEGMENT_SIZE)) != -1) { + total += read; + sink.flush(); + listener.updateProgress(total); + + } + } catch (RuntimeException rethrown) { + throw rethrown; + } catch (Exception ignored) { + //ignore + } + } + + static class TransferData { + + private final Logger log = org.slf4j.LoggerFactory.getLogger(TransferData.class); + private int progressValue; + private long totalSize; + + TransferData(long totalSize) { + this.progressValue = 0; + this.totalSize = totalSize; + } + + void updateTransferProgress(long transferred) { + float progress = (transferred / (float) totalSize) * 100; + if (progressValue != (int) progress) { + progressValue = (int) progress; + log.debug("{} / {} / {}%", transferred, totalSize, progressValue); + } + } + } +} \ No newline at end of file diff --git a/src/main/java/org/arvados/client/api/client/GroupsApiClient.java b/src/main/java/org/arvados/client/api/client/GroupsApiClient.java new file mode 100644 index 0000000000..75aa9ca309 --- /dev/null +++ b/src/main/java/org/arvados/client/api/client/GroupsApiClient.java @@ -0,0 +1,61 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.client; + +import okhttp3.HttpUrl; +import okhttp3.HttpUrl.Builder; +import okhttp3.Request; +import okhttp3.RequestBody; +import org.arvados.client.api.model.Group; +import org.arvados.client.api.model.GroupList; +import org.arvados.client.api.model.argument.ContentsGroup; +import org.arvados.client.api.model.argument.UntrashGroup; +import org.arvados.client.config.ConfigProvider; +import org.slf4j.Logger; + +public class GroupsApiClient extends BaseStandardApiClient { + + private static final String RESOURCE = "groups"; + private final Logger log = org.slf4j.LoggerFactory.getLogger(GroupsApiClient.class); + + public GroupsApiClient(ConfigProvider config) { + super(config); + } + + public GroupList contents(ContentsGroup contentsGroup) { + log.debug("Get {} contents", getType().getSimpleName()); + Builder urlBuilder = getUrlBuilder().addPathSegment("contents"); + addQueryParameters(urlBuilder, contentsGroup); + HttpUrl url = urlBuilder.build(); + Request request = getRequestBuilder().url(url).build(); + return callForList(request); + } + + public Group untrash(UntrashGroup untrashGroup) { + log.debug("Untrash {} by UUID {}", getType().getSimpleName(), untrashGroup.getUuid()); + HttpUrl url = getUrlBuilder().addPathSegment(untrashGroup.getUuid()).addPathSegment("untrash").build(); + RequestBody requestBody = getJsonRequestBody(untrashGroup); + Request request = getRequestBuilder().post(requestBody).url(url).build(); + return callForType(request); + } + + @Override + String getResource() { + return RESOURCE; + } + + @Override + Class getType() { + return Group.class; + } + + @Override + Class getListType() { + return GroupList.class; + } +} diff --git a/src/main/java/org/arvados/client/api/client/KeepServerApiClient.java b/src/main/java/org/arvados/client/api/client/KeepServerApiClient.java new file mode 100644 index 0000000000..a9306ca2ec --- /dev/null +++ b/src/main/java/org/arvados/client/api/client/KeepServerApiClient.java @@ -0,0 +1,54 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.client; + +import okhttp3.Request; +import okhttp3.RequestBody; +import org.arvados.client.api.client.CountingFileRequestBody.TransferData; +import org.arvados.client.common.Headers; +import org.arvados.client.config.ConfigProvider; +import org.slf4j.Logger; + +import java.io.File; +import java.util.Map; + +public class KeepServerApiClient extends BaseApiClient { + + private final Logger log = org.slf4j.LoggerFactory.getLogger(KeepServerApiClient.class); + + public KeepServerApiClient(ConfigProvider config) { + super(config); + } + + public String upload(String url, Map headers, File body) { + + log.debug("Upload file {} to server location {}", body, url); + + final TransferData transferData = new TransferData(body.length()); + + RequestBody requestBody = new CountingFileRequestBody(body, transferData::updateTransferProgress); + + Request request = getRequestBuilder() + .url(url) + .addHeader(Headers.X_KEEP_DESIRED_REPLICAS, headers.get(Headers.X_KEEP_DESIRED_REPLICAS)) + .put(requestBody) + .build(); + + return newCall(request); + } + + public byte[] download(String url) { + + Request request = getRequestBuilder() + .url(url) + .get() + .build(); + + return newFileCall(request); + } +} diff --git a/src/main/java/org/arvados/client/api/client/KeepServicesApiClient.java b/src/main/java/org/arvados/client/api/client/KeepServicesApiClient.java new file mode 100644 index 0000000000..81a9d6f5da --- /dev/null +++ b/src/main/java/org/arvados/client/api/client/KeepServicesApiClient.java @@ -0,0 +1,43 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.client; + +import org.arvados.client.api.model.KeepService; +import org.arvados.client.api.model.KeepServiceList; +import org.arvados.client.config.ConfigProvider; +import org.slf4j.Logger; + +public class KeepServicesApiClient extends BaseStandardApiClient { + + private static final String RESOURCE = "keep_services"; + private final Logger log = org.slf4j.LoggerFactory.getLogger(KeepServicesApiClient.class); + + public KeepServicesApiClient(ConfigProvider config) { + super(config); + } + + public KeepServiceList accessible() { + log.debug("Get list of accessible {}", getType().getSimpleName()); + return callForList(getNoArgumentMethodRequest("accessible")); + } + + @Override + String getResource() { + return RESOURCE; + } + + @Override + Class getType() { + return KeepService.class; + } + + @Override + Class getListType() { + return KeepServiceList.class; + } +} diff --git a/src/main/java/org/arvados/client/api/client/KeepWebApiClient.java b/src/main/java/org/arvados/client/api/client/KeepWebApiClient.java new file mode 100644 index 0000000000..4cd08b7832 --- /dev/null +++ b/src/main/java/org/arvados/client/api/client/KeepWebApiClient.java @@ -0,0 +1,37 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.client; + +import okhttp3.HttpUrl; +import okhttp3.Request; +import org.arvados.client.config.ConfigProvider; + +public class KeepWebApiClient extends BaseApiClient { + + public KeepWebApiClient(ConfigProvider config) { + super(config); + } + + public byte[] download(String collectionUuid, String filePathName) { + Request request = getRequestBuilder() + .url(getUrlBuilder(collectionUuid,filePathName).build()) + .get() + .build(); + + return newFileCall(request); + } + + private HttpUrl.Builder getUrlBuilder(String collectionUuid, String filePathName) { + return new HttpUrl.Builder() + .scheme(config.getApiProtocol()) + .host(config.getKeepWebHost()) + .port(config.getKeepWebPort()) + .addPathSegment("c=" + collectionUuid) + .addPathSegment(filePathName); + } +} diff --git a/src/main/java/org/arvados/client/api/client/ProgressListener.java b/src/main/java/org/arvados/client/api/client/ProgressListener.java new file mode 100644 index 0000000000..8563adcc76 --- /dev/null +++ b/src/main/java/org/arvados/client/api/client/ProgressListener.java @@ -0,0 +1,14 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.client; + +@FunctionalInterface +public interface ProgressListener { + + void updateProgress(long num); +} \ No newline at end of file diff --git a/src/main/java/org/arvados/client/api/client/UsersApiClient.java b/src/main/java/org/arvados/client/api/client/UsersApiClient.java new file mode 100644 index 0000000000..5bf1d07458 --- /dev/null +++ b/src/main/java/org/arvados/client/api/client/UsersApiClient.java @@ -0,0 +1,51 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.client; + +import okhttp3.Request; +import org.arvados.client.api.model.User; +import org.arvados.client.api.model.UserList; +import org.arvados.client.config.ConfigProvider; +import org.slf4j.Logger; + +public class UsersApiClient extends BaseStandardApiClient { + + private static final String RESOURCE = "users"; + private final Logger log = org.slf4j.LoggerFactory.getLogger(UsersApiClient.class); + + public UsersApiClient(ConfigProvider config) { + super(config); + } + + public User current() { + log.debug("Get current {}", getType().getSimpleName()); + Request request = getNoArgumentMethodRequest("current"); + return callForType(request); + } + + public User system() { + log.debug("Get system {}", getType().getSimpleName()); + Request request = getNoArgumentMethodRequest("system"); + return callForType(request); + } + + @Override + String getResource() { + return RESOURCE; + } + + @Override + Class getType() { + return User.class; + } + + @Override + Class getListType() { + return UserList.class; + } +} diff --git a/src/main/java/org/arvados/client/api/client/factory/OkHttpClientFactory.java b/src/main/java/org/arvados/client/api/client/factory/OkHttpClientFactory.java new file mode 100644 index 0000000000..0e95e661e7 --- /dev/null +++ b/src/main/java/org/arvados/client/api/client/factory/OkHttpClientFactory.java @@ -0,0 +1,89 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.client.factory; + +import okhttp3.OkHttpClient; +import org.arvados.client.exception.ArvadosClientException; +import org.slf4j.Logger; + +import javax.net.ssl.SSLContext; +import javax.net.ssl.SSLSocketFactory; +import javax.net.ssl.TrustManager; +import javax.net.ssl.X509TrustManager; +import java.security.KeyManagementException; +import java.security.NoSuchAlgorithmException; +import java.security.SecureRandom; +import java.security.cert.X509Certificate; + +public class OkHttpClientFactory { + + private final Logger log = org.slf4j.LoggerFactory.getLogger(OkHttpClientFactory.class); + + OkHttpClientFactory() { + } + + public static OkHttpClientFactoryBuilder builder() { + return new OkHttpClientFactoryBuilder(); + } + + public OkHttpClient create(boolean apiHostInsecure) { + OkHttpClient.Builder builder = new OkHttpClient.Builder(); + if (apiHostInsecure) { + trustAllCertificates(builder); + } + return builder.build(); + } + + private void trustAllCertificates(OkHttpClient.Builder builder) { + log.warn("Creating unsafe OkHttpClient. All SSL certificates will be accepted."); + try { + // Create a trust manager that does not validate certificate chains + final TrustManager[] trustAllCerts = new TrustManager[] { createX509TrustManager() }; + + // Install the all-trusting trust manager + SSLContext sslContext = SSLContext.getInstance("SSL"); + sslContext.init(null, trustAllCerts, new SecureRandom()); + // Create an ssl socket factory with our all-trusting manager + final SSLSocketFactory sslSocketFactory = sslContext.getSocketFactory(); + + builder.sslSocketFactory(sslSocketFactory, (X509TrustManager) trustAllCerts[0]); + builder.hostnameVerifier((hostname, session) -> true); + } catch (NoSuchAlgorithmException | KeyManagementException e) { + throw new ArvadosClientException("Error establishing SSL context", e); + } + } + + private static X509TrustManager createX509TrustManager() { + return new X509TrustManager() { + + @Override + public void checkClientTrusted(X509Certificate[] chain, String authType) {} + + @Override + public void checkServerTrusted(X509Certificate[] chain, String authType) {} + + @Override + public X509Certificate[] getAcceptedIssuers() { + return new X509Certificate[] {}; + } + }; + } + + public static class OkHttpClientFactoryBuilder { + OkHttpClientFactoryBuilder() { + } + + public OkHttpClientFactory build() { + return new OkHttpClientFactory(); + } + + public String toString() { + return "OkHttpClientFactory.OkHttpClientFactoryBuilder()"; + } + } +} diff --git a/src/main/java/org/arvados/client/api/model/ApiError.java b/src/main/java/org/arvados/client/api/model/ApiError.java new file mode 100644 index 0000000000..1529f9c30c --- /dev/null +++ b/src/main/java/org/arvados/client/api/model/ApiError.java @@ -0,0 +1,42 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.model; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.annotation.JsonInclude; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonPropertyOrder; + +import java.util.List; + +@JsonInclude(JsonInclude.Include.NON_NULL) +@JsonIgnoreProperties(ignoreUnknown = true) +@JsonPropertyOrder({ "errors", "error_token" }) +public class ApiError { + + @JsonProperty("errors") + private List errors; + @JsonProperty("error_token") + private String errorToken; + + public List getErrors() { + return this.errors; + } + + public String getErrorToken() { + return this.errorToken; + } + + public void setErrors(List errors) { + this.errors = errors; + } + + public void setErrorToken(String errorToken) { + this.errorToken = errorToken; + } +} diff --git a/src/main/java/org/arvados/client/api/model/Collection.java b/src/main/java/org/arvados/client/api/model/Collection.java new file mode 100644 index 0000000000..b1652e2a3b --- /dev/null +++ b/src/main/java/org/arvados/client/api/model/Collection.java @@ -0,0 +1,137 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.model; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.annotation.JsonInclude; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonPropertyOrder; + +import java.time.LocalDateTime; + +@JsonInclude(JsonInclude.Include.NON_NULL) +@JsonIgnoreProperties(ignoreUnknown = true) +@JsonPropertyOrder({ "portable_data_hash", "replication_desired", "replication_confirmed_at", "replication_confirmed", "manifest_text", + "name", "description", "properties", "delete_at", "trash_at", "is_trashed" }) +public class Collection extends Item { + + @JsonProperty("portable_data_hash") + private String portableDataHash; + @JsonProperty("replication_desired") + private Integer replicationDesired; + @JsonProperty("replication_confirmed_at") + private LocalDateTime replicationConfirmedAt; + @JsonProperty("replication_confirmed") + private Integer replicationConfirmed; + @JsonProperty("manifest_text") + private String manifestText; + @JsonProperty("name") + private String name; + @JsonProperty("description") + private String description; + @JsonProperty("properties") + private Object properties; + @JsonProperty("delete_at") + private LocalDateTime deleteAt; + @JsonProperty("trash_at") + private LocalDateTime trashAt; + @JsonProperty("is_trashed") + private Boolean trashed; + + public String getPortableDataHash() { + return this.portableDataHash; + } + + public Integer getReplicationDesired() { + return this.replicationDesired; + } + + public LocalDateTime getReplicationConfirmedAt() { + return this.replicationConfirmedAt; + } + + public Integer getReplicationConfirmed() { + return this.replicationConfirmed; + } + + public String getManifestText() { + return this.manifestText; + } + + public String getName() { + return this.name; + } + + public String getDescription() { + return this.description; + } + + public Object getProperties() { + return this.properties; + } + + public LocalDateTime getDeleteAt() { + return this.deleteAt; + } + + public LocalDateTime getTrashAt() { + return this.trashAt; + } + + public Boolean getTrashed() { + return this.trashed; + } + + public void setPortableDataHash(String portableDataHash) { + this.portableDataHash = portableDataHash; + } + + public void setReplicationDesired(Integer replicationDesired) { + this.replicationDesired = replicationDesired; + } + + public void setReplicationConfirmedAt(LocalDateTime replicationConfirmedAt) { + this.replicationConfirmedAt = replicationConfirmedAt; + } + + public void setReplicationConfirmed(Integer replicationConfirmed) { + this.replicationConfirmed = replicationConfirmed; + } + + public void setManifestText(String manifestText) { + this.manifestText = manifestText; + } + + public void setName(String name) { + this.name = name; + } + + public void setDescription(String description) { + this.description = description; + } + + public void setProperties(Object properties) { + this.properties = properties; + } + + public void setDeleteAt(LocalDateTime deleteAt) { + this.deleteAt = deleteAt; + } + + public void setTrashAt(LocalDateTime trashAt) { + this.trashAt = trashAt; + } + + public void setTrashed(Boolean trashed) { + this.trashed = trashed; + } + + public String toString() { + return "Collection(portableDataHash=" + this.getPortableDataHash() + ", replicationDesired=" + this.getReplicationDesired() + ", replicationConfirmedAt=" + this.getReplicationConfirmedAt() + ", replicationConfirmed=" + this.getReplicationConfirmed() + ", manifestText=" + this.getManifestText() + ", name=" + this.getName() + ", description=" + this.getDescription() + ", properties=" + this.getProperties() + ", deleteAt=" + this.getDeleteAt() + ", trashAt=" + this.getTrashAt() + ", trashed=" + this.getTrashed() + ")"; + } +} \ No newline at end of file diff --git a/src/main/java/org/arvados/client/api/model/CollectionList.java b/src/main/java/org/arvados/client/api/model/CollectionList.java new file mode 100644 index 0000000000..4dae7f630c --- /dev/null +++ b/src/main/java/org/arvados/client/api/model/CollectionList.java @@ -0,0 +1,32 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.model; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.annotation.JsonInclude; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonPropertyOrder; + +import java.util.List; + +@JsonInclude(JsonInclude.Include.NON_NULL) +@JsonIgnoreProperties(ignoreUnknown = true) +@JsonPropertyOrder({ "items" }) +public class CollectionList extends ItemList { + + @JsonProperty("items") + private List items; + + public List getItems() { + return this.items; + } + + public void setItems(List items) { + this.items = items; + } +} diff --git a/src/main/java/org/arvados/client/api/model/Group.java b/src/main/java/org/arvados/client/api/model/Group.java new file mode 100644 index 0000000000..e9fbdb744d --- /dev/null +++ b/src/main/java/org/arvados/client/api/model/Group.java @@ -0,0 +1,319 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.model; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.annotation.JsonInclude; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonPropertyOrder; + +import java.time.LocalDateTime; +import java.util.List; + +@JsonInclude(JsonInclude.Include.NON_NULL) +@JsonIgnoreProperties(ignoreUnknown = true) +@JsonPropertyOrder({ "command", "container_count", "container_count_max", "container_image", "container_uuid", "cwd", "environment", "expires_at", + "filters", "log_uuid", "mounts", "output_name", "output_path", "output_uuid", "output_ttl", "priority", "properties", "requesting_container_uuid", + "runtime_constraints", "scheduling_parameters", "state", "use_existing" }) +public class Group extends Item { + + @JsonProperty("name") + private String name; + @JsonProperty("group_class") + private String groupClass; + @JsonProperty("description") + private String description; + @JsonProperty("writable_by") + private List writableBy; + @JsonProperty("delete_at") + private LocalDateTime deleteAt; + @JsonProperty("trash_at") + private LocalDateTime trashAt; + @JsonProperty("is_trashed") + private Boolean isTrashed; + @JsonProperty("command") + private List command; + @JsonProperty("container_count") + private Integer containerCount; + @JsonProperty("container_count_max") + private Integer containerCountMax; + @JsonProperty("container_image") + private String containerImage; + @JsonProperty("container_uuid") + private String containerUuid; + @JsonProperty("cwd") + private String cwd; + @JsonProperty("environment") + private Object environment; + @JsonProperty("expires_at") + private LocalDateTime expiresAt; + @JsonProperty("filters") + private List filters; + @JsonProperty("log_uuid") + private String logUuid; + @JsonProperty("mounts") + private Object mounts; + @JsonProperty("output_name") + private String outputName; + @JsonProperty("output_path") + private String outputPath; + @JsonProperty("output_uuid") + private String outputUuid; + @JsonProperty("output_ttl") + private Integer outputTtl; + @JsonProperty("priority") + private Integer priority; + @JsonProperty("properties") + private Object properties; + @JsonProperty("requesting_container_uuid") + private String requestingContainerUuid; + @JsonProperty("runtime_constraints") + private RuntimeConstraints runtimeConstraints; + @JsonProperty("scheduling_parameters") + private Object schedulingParameters; + @JsonProperty("state") + private String state; + @JsonProperty("use_existing") + private Boolean useExisting; + + public String getName() { + return this.name; + } + + public String getGroupClass() { + return this.groupClass; + } + + public String getDescription() { + return this.description; + } + + public List getWritableBy() { + return this.writableBy; + } + + public LocalDateTime getDeleteAt() { + return this.deleteAt; + } + + public LocalDateTime getTrashAt() { + return this.trashAt; + } + + public Boolean getIsTrashed() { + return this.isTrashed; + } + + public List getCommand() { + return this.command; + } + + public Integer getContainerCount() { + return this.containerCount; + } + + public Integer getContainerCountMax() { + return this.containerCountMax; + } + + public String getContainerImage() { + return this.containerImage; + } + + public String getContainerUuid() { + return this.containerUuid; + } + + public String getCwd() { + return this.cwd; + } + + public Object getEnvironment() { + return this.environment; + } + + public LocalDateTime getExpiresAt() { + return this.expiresAt; + } + + public List getFilters() { + return this.filters; + } + + public String getLogUuid() { + return this.logUuid; + } + + public Object getMounts() { + return this.mounts; + } + + public String getOutputName() { + return this.outputName; + } + + public String getOutputPath() { + return this.outputPath; + } + + public String getOutputUuid() { + return this.outputUuid; + } + + public Integer getOutputTtl() { + return this.outputTtl; + } + + public Integer getPriority() { + return this.priority; + } + + public Object getProperties() { + return this.properties; + } + + public String getRequestingContainerUuid() { + return this.requestingContainerUuid; + } + + public RuntimeConstraints getRuntimeConstraints() { + return this.runtimeConstraints; + } + + public Object getSchedulingParameters() { + return this.schedulingParameters; + } + + public String getState() { + return this.state; + } + + public Boolean getUseExisting() { + return this.useExisting; + } + + public void setName(String name) { + this.name = name; + } + + public void setGroupClass(String groupClass) { + this.groupClass = groupClass; + } + + public void setDescription(String description) { + this.description = description; + } + + public void setWritableBy(List writableBy) { + this.writableBy = writableBy; + } + + public void setDeleteAt(LocalDateTime deleteAt) { + this.deleteAt = deleteAt; + } + + public void setTrashAt(LocalDateTime trashAt) { + this.trashAt = trashAt; + } + + public void setIsTrashed(Boolean isTrashed) { + this.isTrashed = isTrashed; + } + + public void setCommand(List command) { + this.command = command; + } + + public void setContainerCount(Integer containerCount) { + this.containerCount = containerCount; + } + + public void setContainerCountMax(Integer containerCountMax) { + this.containerCountMax = containerCountMax; + } + + public void setContainerImage(String containerImage) { + this.containerImage = containerImage; + } + + public void setContainerUuid(String containerUuid) { + this.containerUuid = containerUuid; + } + + public void setCwd(String cwd) { + this.cwd = cwd; + } + + public void setEnvironment(Object environment) { + this.environment = environment; + } + + public void setExpiresAt(LocalDateTime expiresAt) { + this.expiresAt = expiresAt; + } + + public void setFilters(List filters) { + this.filters = filters; + } + + public void setLogUuid(String logUuid) { + this.logUuid = logUuid; + } + + public void setMounts(Object mounts) { + this.mounts = mounts; + } + + public void setOutputName(String outputName) { + this.outputName = outputName; + } + + public void setOutputPath(String outputPath) { + this.outputPath = outputPath; + } + + public void setOutputUuid(String outputUuid) { + this.outputUuid = outputUuid; + } + + public void setOutputTtl(Integer outputTtl) { + this.outputTtl = outputTtl; + } + + public void setPriority(Integer priority) { + this.priority = priority; + } + + public void setProperties(Object properties) { + this.properties = properties; + } + + public void setRequestingContainerUuid(String requestingContainerUuid) { + this.requestingContainerUuid = requestingContainerUuid; + } + + public void setRuntimeConstraints(RuntimeConstraints runtimeConstraints) { + this.runtimeConstraints = runtimeConstraints; + } + + public void setSchedulingParameters(Object schedulingParameters) { + this.schedulingParameters = schedulingParameters; + } + + public void setState(String state) { + this.state = state; + } + + public void setUseExisting(Boolean useExisting) { + this.useExisting = useExisting; + } + + public String toString() { + return "Group(name=" + this.getName() + ", groupClass=" + this.getGroupClass() + ", description=" + this.getDescription() + ", writableBy=" + this.getWritableBy() + ", deleteAt=" + this.getDeleteAt() + ", trashAt=" + this.getTrashAt() + ", isTrashed=" + this.getIsTrashed() + ", command=" + this.getCommand() + ", containerCount=" + this.getContainerCount() + ", containerCountMax=" + this.getContainerCountMax() + ", containerImage=" + this.getContainerImage() + ", containerUuid=" + this.getContainerUuid() + ", cwd=" + this.getCwd() + ", environment=" + this.getEnvironment() + ", expiresAt=" + this.getExpiresAt() + ", filters=" + this.getFilters() + ", logUuid=" + this.getLogUuid() + ", mounts=" + this.getMounts() + ", outputName=" + this.getOutputName() + ", outputPath=" + this.getOutputPath() + ", outputUuid=" + this.getOutputUuid() + ", outputTtl=" + this.getOutputTtl() + ", priority=" + this.getPriority() + ", properties=" + this.getProperties() + ", requestingContainerUuid=" + this.getRequestingContainerUuid() + ", runtimeConstraints=" + this.getRuntimeConstraints() + ", schedulingParameters=" + this.getSchedulingParameters() + ", state=" + this.getState() + ", useExisting=" + this.getUseExisting() + ")"; + } +} diff --git a/src/main/java/org/arvados/client/api/model/GroupList.java b/src/main/java/org/arvados/client/api/model/GroupList.java new file mode 100644 index 0000000000..c78d8ff145 --- /dev/null +++ b/src/main/java/org/arvados/client/api/model/GroupList.java @@ -0,0 +1,32 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.model; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.annotation.JsonInclude; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonPropertyOrder; + +import java.util.List; + +@JsonInclude(JsonInclude.Include.NON_NULL) +@JsonIgnoreProperties(ignoreUnknown = true) +@JsonPropertyOrder({ "items" }) +public class GroupList extends ItemList { + + @JsonProperty("items") + private List items; + + public List getItems() { + return this.items; + } + + public void setItems(List items) { + this.items = items; + } +} diff --git a/src/main/java/org/arvados/client/api/model/Item.java b/src/main/java/org/arvados/client/api/model/Item.java new file mode 100644 index 0000000000..be30e57843 --- /dev/null +++ b/src/main/java/org/arvados/client/api/model/Item.java @@ -0,0 +1,123 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.model; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.annotation.JsonInclude; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonPropertyOrder; + +import java.time.LocalDateTime; + +@JsonInclude(JsonInclude.Include.NON_NULL) +@JsonIgnoreProperties(ignoreUnknown = true) +@JsonPropertyOrder({ "href", "kind", "etag", "uuid", "owner_uuid", "created_at", "modified_by_client_uuid", + "modified_by_user_uuid", "modified_at", "updated_at" }) +public abstract class Item { + + @JsonProperty("href") + private String href; + @JsonProperty("kind") + private String kind; + @JsonProperty("etag") + private String etag; + @JsonProperty("uuid") + private String uuid; + @JsonProperty("owner_uuid") + private String ownerUuid; + @JsonProperty("created_at") + private LocalDateTime createdAt; + @JsonProperty("modified_by_client_uuid") + private String modifiedByClientUuid; + @JsonProperty("modified_by_user_uuid") + private String modifiedByUserUuid; + @JsonProperty("modified_at") + private LocalDateTime modifiedAt; + @JsonProperty("updated_at") + private LocalDateTime updatedAt; + + public String getHref() { + return this.href; + } + + public String getKind() { + return this.kind; + } + + public String getEtag() { + return this.etag; + } + + public String getUuid() { + return this.uuid; + } + + public String getOwnerUuid() { + return this.ownerUuid; + } + + public LocalDateTime getCreatedAt() { + return this.createdAt; + } + + public String getModifiedByClientUuid() { + return this.modifiedByClientUuid; + } + + public String getModifiedByUserUuid() { + return this.modifiedByUserUuid; + } + + public LocalDateTime getModifiedAt() { + return this.modifiedAt; + } + + public LocalDateTime getUpdatedAt() { + return this.updatedAt; + } + + public void setHref(String href) { + this.href = href; + } + + public void setKind(String kind) { + this.kind = kind; + } + + public void setEtag(String etag) { + this.etag = etag; + } + + public void setUuid(String uuid) { + this.uuid = uuid; + } + + public void setOwnerUuid(String ownerUuid) { + this.ownerUuid = ownerUuid; + } + + public void setCreatedAt(LocalDateTime createdAt) { + this.createdAt = createdAt; + } + + public void setModifiedByClientUuid(String modifiedByClientUuid) { + this.modifiedByClientUuid = modifiedByClientUuid; + } + + public void setModifiedByUserUuid(String modifiedByUserUuid) { + this.modifiedByUserUuid = modifiedByUserUuid; + } + + public void setModifiedAt(LocalDateTime modifiedAt) { + this.modifiedAt = modifiedAt; + } + + public void setUpdatedAt(LocalDateTime updatedAt) { + this.updatedAt = updatedAt; + } +} diff --git a/src/main/java/org/arvados/client/api/model/ItemList.java b/src/main/java/org/arvados/client/api/model/ItemList.java new file mode 100644 index 0000000000..b15a3628f2 --- /dev/null +++ b/src/main/java/org/arvados/client/api/model/ItemList.java @@ -0,0 +1,80 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.model; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.annotation.JsonInclude; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonPropertyOrder; + +@JsonInclude(JsonInclude.Include.NON_NULL) +@JsonIgnoreProperties(ignoreUnknown = true) +@JsonPropertyOrder({ "kind", "etag", "self_link", "offset", "limit", "items_available" }) +public class ItemList { + + @JsonProperty("kind") + private String kind; + @JsonProperty("etag") + private String etag; + @JsonProperty("self_link") + private String selfLink; + @JsonProperty("offset") + private Object offset; + @JsonProperty("limit") + private Object limit; + @JsonProperty("items_available") + private Integer itemsAvailable; + + public String getKind() { + return this.kind; + } + + public String getEtag() { + return this.etag; + } + + public String getSelfLink() { + return this.selfLink; + } + + public Object getOffset() { + return this.offset; + } + + public Object getLimit() { + return this.limit; + } + + public Integer getItemsAvailable() { + return this.itemsAvailable; + } + + public void setKind(String kind) { + this.kind = kind; + } + + public void setEtag(String etag) { + this.etag = etag; + } + + public void setSelfLink(String selfLink) { + this.selfLink = selfLink; + } + + public void setOffset(Object offset) { + this.offset = offset; + } + + public void setLimit(Object limit) { + this.limit = limit; + } + + public void setItemsAvailable(Integer itemsAvailable) { + this.itemsAvailable = itemsAvailable; + } +} \ No newline at end of file diff --git a/src/main/java/org/arvados/client/api/model/KeepService.java b/src/main/java/org/arvados/client/api/model/KeepService.java new file mode 100644 index 0000000000..c29b44cb67 --- /dev/null +++ b/src/main/java/org/arvados/client/api/model/KeepService.java @@ -0,0 +1,77 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.model; + +import com.fasterxml.jackson.annotation.*; + +@JsonInclude(JsonInclude.Include.NON_NULL) +@JsonIgnoreProperties(ignoreUnknown = true) +@JsonPropertyOrder({ "service_host", "service_port", "service_ssl_flag", "service_type", "read_only" }) +public class KeepService extends Item { + + @JsonProperty("service_host") + private String serviceHost; + @JsonProperty("service_port") + private Integer servicePort; + @JsonProperty("service_ssl_flag") + private Boolean serviceSslFlag; + @JsonProperty("service_type") + private String serviceType; + @JsonProperty("read_only") + private Boolean readOnly; + @JsonIgnore + private String serviceRoot; + + public String getServiceHost() { + return this.serviceHost; + } + + public Integer getServicePort() { + return this.servicePort; + } + + public Boolean getServiceSslFlag() { + return this.serviceSslFlag; + } + + public String getServiceType() { + return this.serviceType; + } + + public Boolean getReadOnly() { + return this.readOnly; + } + + public String getServiceRoot() { + return this.serviceRoot; + } + + public void setServiceHost(String serviceHost) { + this.serviceHost = serviceHost; + } + + public void setServicePort(Integer servicePort) { + this.servicePort = servicePort; + } + + public void setServiceSslFlag(Boolean serviceSslFlag) { + this.serviceSslFlag = serviceSslFlag; + } + + public void setServiceType(String serviceType) { + this.serviceType = serviceType; + } + + public void setReadOnly(Boolean readOnly) { + this.readOnly = readOnly; + } + + public void setServiceRoot(String serviceRoot) { + this.serviceRoot = serviceRoot; + } +} \ No newline at end of file diff --git a/src/main/java/org/arvados/client/api/model/KeepServiceList.java b/src/main/java/org/arvados/client/api/model/KeepServiceList.java new file mode 100644 index 0000000000..bbc09dc289 --- /dev/null +++ b/src/main/java/org/arvados/client/api/model/KeepServiceList.java @@ -0,0 +1,32 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.model; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.annotation.JsonInclude; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonPropertyOrder; + +import java.util.List; + +@JsonInclude(JsonInclude.Include.NON_NULL) +@JsonIgnoreProperties(ignoreUnknown = true) +@JsonPropertyOrder({ "items" }) +public class KeepServiceList extends ItemList { + + @JsonProperty("items") + private List items; + + public List getItems() { + return this.items; + } + + public void setItems(List items) { + this.items = items; + } +} \ No newline at end of file diff --git a/src/main/java/org/arvados/client/api/model/RuntimeConstraints.java b/src/main/java/org/arvados/client/api/model/RuntimeConstraints.java new file mode 100644 index 0000000000..a23cd98eb4 --- /dev/null +++ b/src/main/java/org/arvados/client/api/model/RuntimeConstraints.java @@ -0,0 +1,60 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.model; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.annotation.JsonInclude; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonPropertyOrder; + +@JsonInclude(JsonInclude.Include.NON_NULL) +@JsonIgnoreProperties(ignoreUnknown = true) +@JsonPropertyOrder({ "API", "vcpus", "ram", "keep_cache_ram" }) +public class RuntimeConstraints { + + @JsonProperty("API") + private Boolean api; + @JsonProperty("vcpus") + private Integer vcpus; + @JsonProperty("ram") + private Long ram; + @JsonProperty("keep_cache_ram") + private Long keepCacheRam; + + public Boolean getApi() { + return this.api; + } + + public Integer getVcpus() { + return this.vcpus; + } + + public Long getRam() { + return this.ram; + } + + public Long getKeepCacheRam() { + return this.keepCacheRam; + } + + public void setApi(Boolean api) { + this.api = api; + } + + public void setVcpus(Integer vcpus) { + this.vcpus = vcpus; + } + + public void setRam(Long ram) { + this.ram = ram; + } + + public void setKeepCacheRam(Long keepCacheRam) { + this.keepCacheRam = keepCacheRam; + } +} diff --git a/src/main/java/org/arvados/client/api/model/User.java b/src/main/java/org/arvados/client/api/model/User.java new file mode 100644 index 0000000000..5c86a07bdf --- /dev/null +++ b/src/main/java/org/arvados/client/api/model/User.java @@ -0,0 +1,147 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.model; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.annotation.JsonInclude; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonPropertyOrder; + +import java.util.List; + +@JsonInclude(JsonInclude.Include.NON_NULL) +@JsonIgnoreProperties(ignoreUnknown = true) +@JsonPropertyOrder({ "email", "username", "full_name", "first_name", "last_name", "identity_url", "is_active", "is_admin", "is_invited", + "prefs", "writable_by", "default_owner_uuid" }) +public class User extends Item { + + @JsonProperty("email") + private String email; + @JsonProperty("username") + private String username; + @JsonProperty("full_name") + private String fullName; + @JsonProperty("first_name") + private String firstName; + @JsonProperty("last_name") + private String lastName; + @JsonProperty("identity_url") + private String identityUrl; + @JsonProperty("is_active") + private Boolean isActive; + @JsonProperty("is_admin") + private Boolean isAdmin; + @JsonProperty("is_invited") + private Boolean isInvited; + @JsonProperty("prefs") + private Object prefs; + @JsonProperty("writable_by") + private List writableBy; + @JsonProperty("default_owner_uuid") + private Boolean defaultOwnerUuid; + + public String getEmail() { + return this.email; + } + + public String getUsername() { + return this.username; + } + + public String getFullName() { + return this.fullName; + } + + public String getFirstName() { + return this.firstName; + } + + public String getLastName() { + return this.lastName; + } + + public String getIdentityUrl() { + return this.identityUrl; + } + + public Boolean getIsActive() { + return this.isActive; + } + + public Boolean getIsAdmin() { + return this.isAdmin; + } + + public Boolean getIsInvited() { + return this.isInvited; + } + + public Object getPrefs() { + return this.prefs; + } + + public List getWritableBy() { + return this.writableBy; + } + + public Boolean getDefaultOwnerUuid() { + return this.defaultOwnerUuid; + } + + public void setEmail(String email) { + this.email = email; + } + + public void setUsername(String username) { + this.username = username; + } + + public void setFullName(String fullName) { + this.fullName = fullName; + } + + public void setFirstName(String firstName) { + this.firstName = firstName; + } + + public void setLastName(String lastName) { + this.lastName = lastName; + } + + public void setIdentityUrl(String identityUrl) { + this.identityUrl = identityUrl; + } + + public void setIsActive(Boolean isActive) { + this.isActive = isActive; + } + + public void setIsAdmin(Boolean isAdmin) { + this.isAdmin = isAdmin; + } + + public void setIsInvited(Boolean isInvited) { + this.isInvited = isInvited; + } + + public void setPrefs(Object prefs) { + this.prefs = prefs; + } + + public void setWritableBy(List writableBy) { + this.writableBy = writableBy; + } + + public void setDefaultOwnerUuid(Boolean defaultOwnerUuid) { + this.defaultOwnerUuid = defaultOwnerUuid; + } + + public String toString() { + return "User(email=" + this.getEmail() + ", username=" + this.getUsername() + ", fullName=" + this.getFullName() + ", firstName=" + this.getFirstName() + ", lastName=" + this.getLastName() + ", identityUrl=" + this.getIdentityUrl() + ", isActive=" + this.getIsActive() + ", isAdmin=" + this.getIsAdmin() + ", isInvited=" + this.getIsInvited() + ", prefs=" + this.getPrefs() + ", writableBy=" + this.getWritableBy() + ", defaultOwnerUuid=" + this.getDefaultOwnerUuid() + ")"; + } +} diff --git a/src/main/java/org/arvados/client/api/model/UserList.java b/src/main/java/org/arvados/client/api/model/UserList.java new file mode 100644 index 0000000000..e148e72662 --- /dev/null +++ b/src/main/java/org/arvados/client/api/model/UserList.java @@ -0,0 +1,32 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.model; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.annotation.JsonInclude; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonPropertyOrder; + +import java.util.List; + +@JsonInclude(JsonInclude.Include.NON_NULL) +@JsonIgnoreProperties(ignoreUnknown = true) +@JsonPropertyOrder({ "items" }) +public class UserList extends ItemList { + + @JsonProperty("items") + private List items; + + public List getItems() { + return this.items; + } + + public void setItems(List items) { + this.items = items; + } +} diff --git a/src/main/java/org/arvados/client/api/model/argument/Argument.java b/src/main/java/org/arvados/client/api/model/argument/Argument.java new file mode 100644 index 0000000000..6da44088c2 --- /dev/null +++ b/src/main/java/org/arvados/client/api/model/argument/Argument.java @@ -0,0 +1,24 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.model.argument; + +import com.fasterxml.jackson.annotation.JsonIgnore; + +public abstract class Argument { + + @JsonIgnore + private String uuid; + + public String getUuid() { + return this.uuid; + } + + public void setUuid(String uuid) { + this.uuid = uuid; + } +} diff --git a/src/main/java/org/arvados/client/api/model/argument/ContentsGroup.java b/src/main/java/org/arvados/client/api/model/argument/ContentsGroup.java new file mode 100644 index 0000000000..16febf784c --- /dev/null +++ b/src/main/java/org/arvados/client/api/model/argument/ContentsGroup.java @@ -0,0 +1,63 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.model.argument; + +import com.fasterxml.jackson.annotation.JsonInclude; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonPropertyOrder; + +import java.util.List; + +@JsonInclude(JsonInclude.Include.NON_NULL) +@JsonPropertyOrder({ "limit", "order", "filters", "recursive" }) +public class ContentsGroup extends Argument { + + @JsonProperty("limit") + private Integer limit; + + @JsonProperty("order") + private String order; + + @JsonProperty("filters") + private List filters; + + @JsonProperty("recursive") + private Boolean recursive; + + public Integer getLimit() { + return this.limit; + } + + public String getOrder() { + return this.order; + } + + public List getFilters() { + return this.filters; + } + + public Boolean getRecursive() { + return this.recursive; + } + + public void setLimit(Integer limit) { + this.limit = limit; + } + + public void setOrder(String order) { + this.order = order; + } + + public void setFilters(List filters) { + this.filters = filters; + } + + public void setRecursive(Boolean recursive) { + this.recursive = recursive; + } +} diff --git a/src/main/java/org/arvados/client/api/model/argument/Filter.java b/src/main/java/org/arvados/client/api/model/argument/Filter.java new file mode 100644 index 0000000000..ae16dec4ed --- /dev/null +++ b/src/main/java/org/arvados/client/api/model/argument/Filter.java @@ -0,0 +1,118 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.model.argument; + +import com.fasterxml.jackson.annotation.JsonFormat; +import com.fasterxml.jackson.annotation.JsonInclude; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonPropertyOrder; + +@JsonFormat(shape = JsonFormat.Shape.ARRAY) +@JsonInclude(JsonInclude.Include.NON_NULL) +@JsonPropertyOrder({ "attribute", "operator", "operand" }) +public class Filter { + + @JsonProperty("attribute") + private String attribute; + + @JsonProperty("operator") + private Operator operator; + + @JsonProperty("operand") + private Object operand; + + private Filter(String attribute, Operator operator, Object operand) { + this.attribute = attribute; + this.operator = operator; + this.operand = operand; + } + + public static Filter of(String attribute, Operator operator, Object operand) { + return new Filter(attribute, operator, operand); + } + + public String getAttribute() { + return this.attribute; + } + + public Operator getOperator() { + return this.operator; + } + + public Object getOperand() { + return this.operand; + } + + public boolean equals(Object o) { + if (o == this) return true; + if (!(o instanceof Filter)) return false; + final Filter other = (Filter) o; + final Object this$attribute = this.getAttribute(); + final Object other$attribute = other.getAttribute(); + if (this$attribute == null ? other$attribute != null : !this$attribute.equals(other$attribute)) return false; + final Object this$operator = this.getOperator(); + final Object other$operator = other.getOperator(); + if (this$operator == null ? other$operator != null : !this$operator.equals(other$operator)) return false; + final Object this$operand = this.getOperand(); + final Object other$operand = other.getOperand(); + if (this$operand == null ? other$operand != null : !this$operand.equals(other$operand)) return false; + return true; + } + + public int hashCode() { + final int PRIME = 59; + int result = 1; + final Object $attribute = this.getAttribute(); + result = result * PRIME + ($attribute == null ? 43 : $attribute.hashCode()); + final Object $operator = this.getOperator(); + result = result * PRIME + ($operator == null ? 43 : $operator.hashCode()); + final Object $operand = this.getOperand(); + result = result * PRIME + ($operand == null ? 43 : $operand.hashCode()); + return result; + } + + public String toString() { + return "Filter(attribute=" + this.getAttribute() + ", operator=" + this.getOperator() + ", operand=" + this.getOperand() + ")"; + } + + public enum Operator { + + @JsonProperty("<") + LESS, + + @JsonProperty("<=") + LESS_EQUALS, + + @JsonProperty(">=") + MORE_EQUALS, + + @JsonProperty(">") + MORE, + + @JsonProperty("like") + LIKE, + + @JsonProperty("ilike") + ILIKE, + + @JsonProperty("=") + EQUALS, + + @JsonProperty("!=") + NOT_EQUALS, + + @JsonProperty("in") + IN, + + @JsonProperty("not in") + NOT_IN, + + @JsonProperty("is_a") + IS_A + } +} diff --git a/src/main/java/org/arvados/client/api/model/argument/ListArgument.java b/src/main/java/org/arvados/client/api/model/argument/ListArgument.java new file mode 100644 index 0000000000..70231e6766 --- /dev/null +++ b/src/main/java/org/arvados/client/api/model/argument/ListArgument.java @@ -0,0 +1,123 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.model.argument; + +import com.fasterxml.jackson.annotation.JsonInclude; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonPropertyOrder; + +import java.util.List; + +@JsonInclude(JsonInclude.Include.NON_NULL) +@JsonPropertyOrder({ "limit", "offset", "filters", "order", "select", "distinct", "count" }) +public class ListArgument extends Argument { + + @JsonProperty("limit") + private Integer limit; + + @JsonProperty("offset") + private Integer offset; + + @JsonProperty("filters") + private List filters; + + @JsonProperty("order") + private List order; + + @JsonProperty("select") + private List select; + + @JsonProperty("distinct") + private Boolean distinct; + + @JsonProperty("count") + private Count count; + + + ListArgument(Integer limit, Integer offset, List filters, List order, List select, Boolean distinct, Count count) { + this.limit = limit; + this.offset = offset; + this.filters = filters; + this.order = order; + this.select = select; + this.distinct = distinct; + this.count = count; + } + + public static ListArgumentBuilder builder() { + return new ListArgumentBuilder(); + } + + public enum Count { + + @JsonProperty("exact") + EXACT, + + @JsonProperty("none") + NONE + } + + public static class ListArgumentBuilder { + private Integer limit; + private Integer offset; + private List filters; + private List order; + private List select; + private Boolean distinct; + private Count count; + + ListArgumentBuilder() { + } + + public ListArgumentBuilder limit(Integer limit) { + this.limit = limit; + return this; + } + + public ListArgumentBuilder offset(Integer offset) { + this.offset = offset; + return this; + } + + public ListArgumentBuilder filters(List filters) { + this.filters = filters; + return this; + } + + public ListArgumentBuilder order(List order) { + this.order = order; + return this; + } + + public ListArgumentBuilder select(List select) { + this.select = select; + return this; + } + + public ListArgumentBuilder distinct(Boolean distinct) { + this.distinct = distinct; + return this; + } + + public ListArgumentBuilder count(Count count) { + this.count = count; + return this; + } + + public ListArgument build() { + return new ListArgument(limit, offset, filters, order, select, distinct, count); + } + + public String toString() { + return "ListArgument.ListArgumentBuilder(limit=" + this.limit + + ", offset=" + this.offset + ", filters=" + this.filters + + ", order=" + this.order + ", select=" + this.select + + ", distinct=" + this.distinct + ", count=" + this.count + ")"; + } + } +} diff --git a/src/main/java/org/arvados/client/api/model/argument/UntrashGroup.java b/src/main/java/org/arvados/client/api/model/argument/UntrashGroup.java new file mode 100644 index 0000000000..027dbf7275 --- /dev/null +++ b/src/main/java/org/arvados/client/api/model/argument/UntrashGroup.java @@ -0,0 +1,28 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.model.argument; + +import com.fasterxml.jackson.annotation.JsonInclude; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonPropertyOrder; + +@JsonInclude(JsonInclude.Include.NON_NULL) +@JsonPropertyOrder({ "ensure_unique_name" }) +public class UntrashGroup extends Argument { + + @JsonProperty("ensure_unique_name") + private Boolean ensureUniqueName; + + public Boolean getEnsureUniqueName() { + return this.ensureUniqueName; + } + + public void setEnsureUniqueName(Boolean ensureUniqueName) { + this.ensureUniqueName = ensureUniqueName; + } +} diff --git a/src/main/java/org/arvados/client/common/Characters.java b/src/main/java/org/arvados/client/common/Characters.java new file mode 100644 index 0000000000..1e49a71bcb --- /dev/null +++ b/src/main/java/org/arvados/client/common/Characters.java @@ -0,0 +1,21 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.common; + +public final class Characters { + + private Characters() {} + + public static final String SPACE = "\\040"; + public static final String NEW_LINE = "\n"; + public static final String SLASH = "/"; + public static final String DOT = "."; + public static final String COLON = ":"; + public static final String PERCENT = "%"; + public static final String QUOTE = "\""; +} diff --git a/src/main/java/org/arvados/client/common/Headers.java b/src/main/java/org/arvados/client/common/Headers.java new file mode 100644 index 0000000000..4b43ed9bb1 --- /dev/null +++ b/src/main/java/org/arvados/client/common/Headers.java @@ -0,0 +1,15 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.common; + +public final class Headers { + + private Headers() {} + + public static final String X_KEEP_DESIRED_REPLICAS = "X-Keep-Desired-Replicas"; +} diff --git a/src/main/java/org/arvados/client/common/Patterns.java b/src/main/java/org/arvados/client/common/Patterns.java new file mode 100644 index 0000000000..c852cb070c --- /dev/null +++ b/src/main/java/org/arvados/client/common/Patterns.java @@ -0,0 +1,19 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.common; + +public final class Patterns { + + public static final String HINT_PATTERN = "^[A-Z][A-Za-z0-9@_-]+$"; + public static final String FILE_TOKEN_PATTERN = "(\\d+:\\d+:\\S+)"; + public static final String LOCATOR_PATTERN = "([0-9a-f]{32})\\+([0-9]+)(\\+[A-Z][-A-Za-z0-9@_]*)*"; + public static final String GROUP_UUID_PATTERN = "[a-z0-9]{5}-j7d0g-[a-z0-9]{15}"; + public static final String USER_UUID_PATTERN = "[a-z0-9]{5}-tpzed-[a-z0-9]{15}"; + + private Patterns() {} +} diff --git a/src/main/java/org/arvados/client/config/ConfigProvider.java b/src/main/java/org/arvados/client/config/ConfigProvider.java new file mode 100644 index 0000000000..c9a4109313 --- /dev/null +++ b/src/main/java/org/arvados/client/config/ConfigProvider.java @@ -0,0 +1,40 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.config; + +import java.io.File; + +public interface ConfigProvider { + + //API + boolean isApiHostInsecure(); + + String getKeepWebHost(); + + int getKeepWebPort(); + + String getApiHost(); + + int getApiPort(); + + String getApiToken(); + + String getApiProtocol(); + + + //FILE UPLOAD + int getFileSplitSize(); + + File getFileSplitDirectory(); + + int getNumberOfCopies(); + + int getNumberOfRetries(); + + +} diff --git a/src/main/java/org/arvados/client/config/ExternalConfigProvider.java b/src/main/java/org/arvados/client/config/ExternalConfigProvider.java new file mode 100644 index 0000000000..17e06966fa --- /dev/null +++ b/src/main/java/org/arvados/client/config/ExternalConfigProvider.java @@ -0,0 +1,181 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.config; + +import java.io.File; + +public class ExternalConfigProvider implements ConfigProvider { + + private boolean apiHostInsecure; + private String keepWebHost; + private int keepWebPort; + private String apiHost; + private int apiPort; + private String apiToken; + private String apiProtocol; + private int fileSplitSize; + private File fileSplitDirectory; + private int numberOfCopies; + private int numberOfRetries; + + ExternalConfigProvider(boolean apiHostInsecure, String keepWebHost, int keepWebPort, String apiHost, int apiPort, String apiToken, String apiProtocol, int fileSplitSize, File fileSplitDirectory, int numberOfCopies, int numberOfRetries) { + this.apiHostInsecure = apiHostInsecure; + this.keepWebHost = keepWebHost; + this.keepWebPort = keepWebPort; + this.apiHost = apiHost; + this.apiPort = apiPort; + this.apiToken = apiToken; + this.apiProtocol = apiProtocol; + this.fileSplitSize = fileSplitSize; + this.fileSplitDirectory = fileSplitDirectory; + this.numberOfCopies = numberOfCopies; + this.numberOfRetries = numberOfRetries; + } + + public static ExternalConfigProviderBuilder builder() { + return new ExternalConfigProviderBuilder(); + } + + @Override + public String toString() { + return "ExternalConfigProvider{" + + "apiHostInsecure=" + apiHostInsecure + + ", keepWebHost='" + keepWebHost + '\'' + + ", keepWebPort=" + keepWebPort + + ", apiHost='" + apiHost + '\'' + + ", apiPort=" + apiPort + + ", apiToken='" + apiToken + '\'' + + ", apiProtocol='" + apiProtocol + '\'' + + ", fileSplitSize=" + fileSplitSize + + ", fileSplitDirectory=" + fileSplitDirectory + + ", numberOfCopies=" + numberOfCopies + + ", numberOfRetries=" + numberOfRetries + + '}'; + } + + public boolean isApiHostInsecure() { + return this.apiHostInsecure; + } + + public String getKeepWebHost() { + return this.keepWebHost; + } + + public int getKeepWebPort() { + return this.keepWebPort; + } + + public String getApiHost() { + return this.apiHost; + } + + public int getApiPort() { + return this.apiPort; + } + + public String getApiToken() { + return this.apiToken; + } + + public String getApiProtocol() { + return this.apiProtocol; + } + + public int getFileSplitSize() { + return this.fileSplitSize; + } + + public File getFileSplitDirectory() { + return this.fileSplitDirectory; + } + + public int getNumberOfCopies() { + return this.numberOfCopies; + } + + public int getNumberOfRetries() { + return this.numberOfRetries; + } + + public static class ExternalConfigProviderBuilder { + private boolean apiHostInsecure; + private String keepWebHost; + private int keepWebPort; + private String apiHost; + private int apiPort; + private String apiToken; + private String apiProtocol; + private int fileSplitSize; + private File fileSplitDirectory; + private int numberOfCopies; + private int numberOfRetries; + + ExternalConfigProviderBuilder() { + } + + public ExternalConfigProvider.ExternalConfigProviderBuilder apiHostInsecure(boolean apiHostInsecure) { + this.apiHostInsecure = apiHostInsecure; + return this; + } + + public ExternalConfigProvider.ExternalConfigProviderBuilder keepWebHost(String keepWebHost) { + this.keepWebHost = keepWebHost; + return this; + } + + public ExternalConfigProvider.ExternalConfigProviderBuilder keepWebPort(int keepWebPort) { + this.keepWebPort = keepWebPort; + return this; + } + + public ExternalConfigProvider.ExternalConfigProviderBuilder apiHost(String apiHost) { + this.apiHost = apiHost; + return this; + } + + public ExternalConfigProvider.ExternalConfigProviderBuilder apiPort(int apiPort) { + this.apiPort = apiPort; + return this; + } + + public ExternalConfigProvider.ExternalConfigProviderBuilder apiToken(String apiToken) { + this.apiToken = apiToken; + return this; + } + + public ExternalConfigProvider.ExternalConfigProviderBuilder apiProtocol(String apiProtocol) { + this.apiProtocol = apiProtocol; + return this; + } + + public ExternalConfigProvider.ExternalConfigProviderBuilder fileSplitSize(int fileSplitSize) { + this.fileSplitSize = fileSplitSize; + return this; + } + + public ExternalConfigProvider.ExternalConfigProviderBuilder fileSplitDirectory(File fileSplitDirectory) { + this.fileSplitDirectory = fileSplitDirectory; + return this; + } + + public ExternalConfigProvider.ExternalConfigProviderBuilder numberOfCopies(int numberOfCopies) { + this.numberOfCopies = numberOfCopies; + return this; + } + + public ExternalConfigProvider.ExternalConfigProviderBuilder numberOfRetries(int numberOfRetries) { + this.numberOfRetries = numberOfRetries; + return this; + } + + public ExternalConfigProvider build() { + return new ExternalConfigProvider(apiHostInsecure, keepWebHost, keepWebPort, apiHost, apiPort, apiToken, apiProtocol, fileSplitSize, fileSplitDirectory, numberOfCopies, numberOfRetries); + } + + } +} diff --git a/src/main/java/org/arvados/client/config/FileConfigProvider.java b/src/main/java/org/arvados/client/config/FileConfigProvider.java new file mode 100644 index 0000000000..589c3346b2 --- /dev/null +++ b/src/main/java/org/arvados/client/config/FileConfigProvider.java @@ -0,0 +1,107 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.config; + +import com.typesafe.config.Config; +import com.typesafe.config.ConfigFactory; + +import java.io.File; + +public class FileConfigProvider implements ConfigProvider { + + private static final String DEFAULT_PATH = "arvados"; + private final Config config; + + public FileConfigProvider() { + config = ConfigFactory.load().getConfig(DEFAULT_PATH); + } + + public FileConfigProvider(final String configFile) { + config = (configFile != null) ? + ConfigFactory.load(configFile).getConfig(DEFAULT_PATH) : ConfigFactory.load().getConfig(DEFAULT_PATH); + } + + public Config getConfig() { + return config; + } + + private File getFile(String path) { + return new File(config.getString(path)); + } + + private int getInt(String path) { + return config.getInt(path); + } + + private boolean getBoolean(String path) { + return config.getBoolean(path); + } + + private String getString(String path) { + return config.getString(path); + } + + @Override + public boolean isApiHostInsecure() { + return this.getBoolean("api.host-insecure"); + } + + @Override + public String getKeepWebHost() { + return this.getString("api.keepweb-host"); + } + + @Override + public int getKeepWebPort() { + return this.getInt("api.keepweb-port"); + } + + @Override + public String getApiHost() { + return this.getString("api.host"); + } + + @Override + public int getApiPort() { + return this.getInt("api.port"); + } + + @Override + public String getApiToken() { + return this.getString("api.token"); + } + + @Override + public String getApiProtocol() { + return this.getString("api.protocol"); + } + + @Override + public int getFileSplitSize() { + return this.getInt("split-size"); + } + + @Override + public File getFileSplitDirectory() { + return this.getFile("temp-dir"); + } + + @Override + public int getNumberOfCopies() { + return this.getInt("copies"); + } + + @Override + public int getNumberOfRetries() { + return this.getInt("retries"); + } + + public String getIntegrationTestProjectUuid() { + return this.getString("integration-tests.project-uuid"); + } +} diff --git a/src/main/java/org/arvados/client/exception/ArvadosApiException.java b/src/main/java/org/arvados/client/exception/ArvadosApiException.java new file mode 100644 index 0000000000..51a9962487 --- /dev/null +++ b/src/main/java/org/arvados/client/exception/ArvadosApiException.java @@ -0,0 +1,25 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.exception; + +public class ArvadosApiException extends ArvadosClientException { + + private static final long serialVersionUID = 1L; + + public ArvadosApiException(String message) { + super(message); + } + + public ArvadosApiException(String message, Throwable cause) { + super(message, cause); + } + + public ArvadosApiException(Throwable cause) { + super(cause); + } +} diff --git a/src/main/java/org/arvados/client/exception/ArvadosClientException.java b/src/main/java/org/arvados/client/exception/ArvadosClientException.java new file mode 100644 index 0000000000..e93028d75c --- /dev/null +++ b/src/main/java/org/arvados/client/exception/ArvadosClientException.java @@ -0,0 +1,27 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.exception; + +/** + * Parent exception for all exceptions in library. + * More specific exceptions like ArvadosApiException extend this class. + */ +public class ArvadosClientException extends RuntimeException { + + public ArvadosClientException(String message) { + super(message); + } + + public ArvadosClientException(String message, Throwable cause) { + super(message, cause); + } + + public ArvadosClientException(Throwable cause) { + super(cause); + } +} diff --git a/src/main/java/org/arvados/client/facade/ArvadosFacade.java b/src/main/java/org/arvados/client/facade/ArvadosFacade.java new file mode 100644 index 0000000000..b80b528fe5 --- /dev/null +++ b/src/main/java/org/arvados/client/facade/ArvadosFacade.java @@ -0,0 +1,299 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.facade; + +import com.google.common.collect.Lists; +import org.arvados.client.api.client.CollectionsApiClient; +import org.arvados.client.api.client.GroupsApiClient; +import org.arvados.client.api.client.KeepWebApiClient; +import org.arvados.client.api.client.UsersApiClient; +import org.arvados.client.api.model.*; +import org.arvados.client.api.model.argument.Filter; +import org.arvados.client.api.model.argument.ListArgument; +import org.arvados.client.config.FileConfigProvider; +import org.arvados.client.config.ConfigProvider; +import org.arvados.client.logic.collection.FileToken; +import org.arvados.client.logic.collection.ManifestDecoder; +import org.arvados.client.logic.keep.FileDownloader; +import org.arvados.client.logic.keep.FileUploader; +import org.arvados.client.logic.keep.KeepClient; +import org.slf4j.Logger; + +import java.io.File; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; + +public class ArvadosFacade { + + private final ConfigProvider config; + private final Logger log = org.slf4j.LoggerFactory.getLogger(ArvadosFacade.class); + private CollectionsApiClient collectionsApiClient; + private GroupsApiClient groupsApiClient; + private UsersApiClient usersApiClient; + private FileDownloader fileDownloader; + private FileUploader fileUploader; + private static final String PROJECT = "project"; + private static final String SUBPROJECT = "sub-project"; + + public ArvadosFacade(ConfigProvider config) { + this.config = config; + setFacadeFields(); + } + + public ArvadosFacade() { + this.config = new FileConfigProvider(); + setFacadeFields(); + } + + private void setFacadeFields() { + collectionsApiClient = new CollectionsApiClient(config); + groupsApiClient = new GroupsApiClient(config); + usersApiClient = new UsersApiClient(config); + KeepClient keepClient = new KeepClient(config); + ManifestDecoder manifestDecoder = new ManifestDecoder(); + KeepWebApiClient keepWebApiClient = new KeepWebApiClient(config); + fileDownloader = new FileDownloader(keepClient, manifestDecoder, collectionsApiClient, keepWebApiClient); + fileUploader = new FileUploader(keepClient, collectionsApiClient, config); + } + + /** + * This method downloads single file from collection using Arvados Keep-Web. + * File is saved on a drive in specified location and returned. + * + * @param filePathName path to the file in collection. If requested file is stored + * directly in collection (not within its subdirectory) this + * would be just the name of file (ex. 'file.txt'). + * Otherwise full file path must be passed (ex. 'folder/file.txt') + * @param collectionUuid uuid of collection containing requested file + * @param pathToDownloadFolder path to location in which file should be saved. + * Passed location must be a directory in which file of + * that name does not already exist. + * @return downloaded file + */ + public File downloadFile(String filePathName, String collectionUuid, String pathToDownloadFolder) { + return fileDownloader.downloadSingleFileUsingKeepWeb(filePathName, collectionUuid, pathToDownloadFolder); + } + + /** + * This method downloads all files from collection. + * Directory named by collection uuid is created in specified location, + * files are saved on a drive in this directory and list with downloaded + * files is returned. + * + * @param collectionUuid uuid of collection from which files are downloaded + * @param pathToDownloadFolder path to location in which files should be saved. + * New folder named by collection uuid, containing + * downloaded files, is created in this location. + * Passed location must be a directory in which folder + * of that name does not already exist. + * @param usingKeepWeb if set to true files will be downloaded using Keep Web. + * If set to false files will be downloaded using Keep Server API. + * @return list containing downloaded files + */ + public List downloadCollectionFiles(String collectionUuid, String pathToDownloadFolder, boolean usingKeepWeb) { + if (usingKeepWeb) + return fileDownloader.downloadFilesFromCollectionUsingKeepWeb(collectionUuid, pathToDownloadFolder); + return fileDownloader.downloadFilesFromCollection(collectionUuid, pathToDownloadFolder); + } + + /** + * Lists all FileTokens (objects containing information about files) for + * specified collection. + * Information in each FileToken includes file path, name, size and position + * in data stream + * + * @param collectionUuid uuid of collection for which FileTokens are listed + * @return list containing FileTokens for each file in specified collection + */ + public List listFileInfoFromCollection(String collectionUuid) { + return fileDownloader.listFileInfoFromCollection(collectionUuid); + } + + /** + * Creates and uploads new collection containing passed files. + * Created collection has a default name and is uploaded to user's 'Home' project. + * + * @see ArvadosFacade#upload(List, String, String) + */ + public Collection upload(List files) { + return upload(files, null, null); + } + + /** + * Creates and uploads new collection containing a single file. + * Created collection has a default name and is uploaded to user's 'Home' project. + * + * @see ArvadosFacade#upload(List, String, String) + */ + public Collection upload(File file) { + return upload(Collections.singletonList(file), null, null); + } + + /** + * Uploads new collection with specified name and containing selected files + * to an existing project. + * + * @param sourceFiles list of files to be uploaded within new collection + * @param collectionName name for the newly created collection. + * Collection with that name cannot be already created + * in specified project. If null is passed + * then collection name is set to default, containing + * phrase 'New Collection' and a timestamp. + * @param projectUuid uuid of the project in which created collection is to be included. + * If null is passed then collection is uploaded to user's 'Home' project. + * @return collection object mapped from JSON that is returned from server after successful upload + */ + public Collection upload(List sourceFiles, String collectionName, String projectUuid) { + return fileUploader.upload(sourceFiles, collectionName, projectUuid); + } + + /** + * Uploads a file to a specified collection. + * + * @see ArvadosFacade#uploadToExistingCollection(List, String) + */ + public Collection uploadToExistingCollection(File file, String collectionUUID) { + return fileUploader.uploadToExistingCollection(Collections.singletonList(file), collectionUUID); + } + + /** + * Uploads multiple files to an existing collection. + * + * @param files list of files to be uploaded to existing collection. + * File names must be unique - both within passed list and + * in comparison with files already existing within collection. + * @param collectionUUID UUID of collection to which files should be uploaded + * @return collection object mapped from JSON that is returned from server after successful upload + */ + public Collection uploadToExistingCollection(List files, String collectionUUID) { + return fileUploader.uploadToExistingCollection(files, collectionUUID); + } + + /** + * Creates and uploads new empty collection to specified project. + * + * @param collectionName name for the newly created collection. + * Collection with that name cannot be already created + * in specified project. + * @param projectUuid uuid of project that will contain uploaded empty collection. + * To select home project pass current user's uuid from getCurrentUser() + * @return collection object mapped from JSON that is returned from server after successful upload + * @see ArvadosFacade#getCurrentUser() + */ + public Collection createEmptyCollection(String collectionName, String projectUuid) { + Collection collection = new Collection(); + collection.setOwnerUuid(projectUuid); + collection.setName(collectionName); + return collectionsApiClient.create(collection); + } + + /** + * Returns current user information based on Api Token provided via configuration + * + * @return user object mapped from JSON that is returned from server based on provided Api Token. + * It contains information about user who has this token assigned. + */ + public User getCurrentUser() { + return usersApiClient.current(); + } + + /** + * Gets uuid of current user based on api Token provided in configuration and uses it to list all + * projects that this user owns in Arvados. + * + * @return GroupList containing all groups that current user is owner of. + * @see ArvadosFacade#getCurrentUser() + */ + public GroupList showGroupsOwnedByCurrentUser() { + ListArgument listArgument = ListArgument.builder() + .filters(Arrays.asList( + Filter.of("owner_uuid", Filter.Operator.LIKE, getCurrentUser().getUuid()), + Filter.of("group_class", Filter.Operator.IN, Lists.newArrayList(PROJECT, SUBPROJECT) + ))) + .build(); + GroupList groupList = groupsApiClient.list(listArgument); + log.debug("Groups owned by user:"); + groupList.getItems().forEach(m -> log.debug(m.getUuid() + " -- " + m.getName())); + + return groupList; + } + + /** + * Gets uuid of current user based on api Token provided in configuration and uses it to list all + * projects that this user has read access to in Arvados. + * + * @return GroupList containing all groups that current user has read access to. + */ + public GroupList showGroupsAccessibleByCurrentUser() { + ListArgument listArgument = ListArgument.builder() + .filters(Collections.singletonList( + Filter.of("group_class", Filter.Operator.IN, Lists.newArrayList(PROJECT, SUBPROJECT) + ))) + .build(); + GroupList groupList = groupsApiClient.list(listArgument); + log.debug("Groups accessible by user:"); + groupList.getItems().forEach(m -> log.debug(m.getUuid() + " -- " + m.getName())); + + return groupList; + } + + /** + * Filters all collections from selected project and returns list of those that contain passed String in their name. + * Operator "LIKE" is used so in order to obtain certain collection it is sufficient to pass just part of its name. + * Returned collections in collectionList are ordered by date of creation (starting from oldest one). + * + * @param collectionName collections containing this param in their name will be returned. + * Passing a wildcard is possible - for example passing "a%" searches for + * all collections starting with "a". + * @param projectUuid uuid of project in which will be searched for collections with given name. To search home + * project provide user uuid (from getCurrentUser()) + * @return object CollectionList containing all collections matching specified name criteria + * @see ArvadosFacade#getCurrentUser() + */ + public CollectionList getCollectionsFromProjectByName(String collectionName, String projectUuid) { + ListArgument listArgument = ListArgument.builder() + .filters(Arrays.asList( + Filter.of("owner_uuid", Filter.Operator.LIKE, projectUuid), + Filter.of("name", Filter.Operator.LIKE, collectionName) + )) + .order(Collections.singletonList("created_at")) + .build(); + + return collectionsApiClient.list(listArgument); + } + + /** + * Creates new project that will be a subproject of "home" for current user. + * + * @param projectName name for the newly created project + * @return Group object containing information about created project + * (mapped from JSON returned from server after creating the project) + */ + public Group createNewProject(String projectName) { + Group project = new Group(); + project.setName(projectName); + project.setGroupClass(PROJECT); + Group createdProject = groupsApiClient.create(project); + log.debug("Project " + createdProject.getName() + " created with UUID: " + createdProject.getUuid()); + return createdProject; + } + + /** + * Deletes collection with specified uuid. + * + * @param collectionUuid uuid of collection to be deleted. User whose token is provided in configuration + * must be authorized to delete such collection. + * @return collection object with deleted collection (mapped from JSON returned from server after deleting the collection) + */ + public Collection deleteCollection(String collectionUuid) { + Collection deletedCollection = collectionsApiClient.delete(collectionUuid); + log.debug("Collection: " + collectionUuid + " deleted."); + return deletedCollection; + } +} diff --git a/src/main/java/org/arvados/client/logic/collection/CollectionFactory.java b/src/main/java/org/arvados/client/logic/collection/CollectionFactory.java new file mode 100644 index 0000000000..25379f54b8 --- /dev/null +++ b/src/main/java/org/arvados/client/logic/collection/CollectionFactory.java @@ -0,0 +1,134 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.logic.collection; + +import org.arvados.client.api.client.GroupsApiClient; +import org.arvados.client.api.client.UsersApiClient; +import org.arvados.client.exception.ArvadosApiException; +import org.arvados.client.api.model.Collection; +import org.arvados.client.common.Patterns; +import org.arvados.client.config.FileConfigProvider; +import org.arvados.client.config.ConfigProvider; +import org.arvados.client.exception.ArvadosClientException; + +import java.io.File; +import java.time.LocalDateTime; +import java.time.format.DateTimeFormatter; +import java.util.List; +import java.util.Optional; + +public class CollectionFactory { + + private ConfigProvider config; + private UsersApiClient usersApiClient; + private GroupsApiClient groupsApiClient; + + private final String name; + private final String projectUuid; + private final List manifestFiles; + private final List manifestLocators; + + private CollectionFactory(ConfigProvider config, String name, String projectUuid, List manifestFiles, List manifestLocators) { + this.name = name; + this.projectUuid = projectUuid; + this.manifestFiles = manifestFiles; + this.manifestLocators = manifestLocators; + this.config = config; + setApiClients(); + } + + public static CollectionFactoryBuilder builder() { + return new CollectionFactoryBuilder(); + } + + private void setApiClients() { + if(this.config == null) this.config = new FileConfigProvider(); + + this.usersApiClient = new UsersApiClient(config); + this.groupsApiClient = new GroupsApiClient(config); + } + + public Collection create() { + ManifestFactory manifestFactory = ManifestFactory.builder() + .files(manifestFiles) + .locators(manifestLocators) + .build(); + String manifest = manifestFactory.create(); + + Collection newCollection = new Collection(); + newCollection.setName(getNameOrDefault(name)); + newCollection.setManifestText(manifest); + newCollection.setOwnerUuid(getDesiredProjectUuid(projectUuid)); + + return newCollection; + } + + private String getNameOrDefault(String name) { + return Optional.ofNullable(name).orElseGet(() -> { + LocalDateTime dateTime = LocalDateTime.now(); + DateTimeFormatter formatter = DateTimeFormatter.ofPattern("Y-MM-dd HH:mm:ss.SSS"); + return String.format("New Collection (%s)", dateTime.format(formatter)); + }); + } + + public String getDesiredProjectUuid(String projectUuid) { + try { + if (projectUuid == null || projectUuid.length() == 0){ + return usersApiClient.current().getUuid(); + } else if (projectUuid.matches(Patterns.USER_UUID_PATTERN)) { + return usersApiClient.get(projectUuid).getUuid(); + } else if (projectUuid.matches(Patterns.GROUP_UUID_PATTERN)) { + return groupsApiClient.get(projectUuid).getUuid(); + } + } catch (ArvadosApiException e) { + throw new ArvadosClientException(String.format("An error occurred while getting project by UUID %s", projectUuid)); + } + throw new ArvadosClientException(String.format("No project with %s UUID found", projectUuid)); + } + + public static class CollectionFactoryBuilder { + private ConfigProvider config; + private String name; + private String projectUuid; + private List manifestFiles; + private List manifestLocators; + + CollectionFactoryBuilder() { + } + + public CollectionFactoryBuilder config(ConfigProvider config) { + this.config = config; + return this; + } + + public CollectionFactoryBuilder name(String name) { + this.name = name; + return this; + } + + public CollectionFactoryBuilder projectUuid(String projectUuid) { + this.projectUuid = projectUuid; + return this; + } + + public CollectionFactoryBuilder manifestFiles(List manifestFiles) { + this.manifestFiles = manifestFiles; + return this; + } + + public CollectionFactoryBuilder manifestLocators(List manifestLocators) { + this.manifestLocators = manifestLocators; + return this; + } + + public CollectionFactory build() { + return new CollectionFactory(config, name, projectUuid, manifestFiles, manifestLocators); + } + + } +} diff --git a/src/main/java/org/arvados/client/logic/collection/FileToken.java b/src/main/java/org/arvados/client/logic/collection/FileToken.java new file mode 100644 index 0000000000..b41ccd3cdd --- /dev/null +++ b/src/main/java/org/arvados/client/logic/collection/FileToken.java @@ -0,0 +1,60 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.logic.collection; + +import com.google.common.base.Strings; +import org.arvados.client.common.Characters; + +public class FileToken { + + private int filePosition; + private int fileSize; + private String fileName; + private String path; + + public FileToken(String fileTokenInfo) { + splitFileTokenInfo(fileTokenInfo); + } + + public FileToken(String fileTokenInfo, String path) { + splitFileTokenInfo(fileTokenInfo); + this.path = path; + } + + private void splitFileTokenInfo(String fileTokenInfo) { + String[] tokenPieces = fileTokenInfo.split(":"); + this.filePosition = Integer.parseInt(tokenPieces[0]); + this.fileSize = Integer.parseInt(tokenPieces[1]); + this.fileName = tokenPieces[2].replace(Characters.SPACE, " "); + } + + @Override + public String toString() { + return filePosition + ":" + fileSize + ":" + fileName; + } + + public String getFullPath() { + return Strings.isNullOrEmpty(path) ? fileName : path + fileName; + } + + public int getFilePosition() { + return this.filePosition; + } + + public int getFileSize() { + return this.fileSize; + } + + public String getFileName() { + return this.fileName; + } + + public String getPath() { + return this.path; + } +} diff --git a/src/main/java/org/arvados/client/logic/collection/ManifestDecoder.java b/src/main/java/org/arvados/client/logic/collection/ManifestDecoder.java new file mode 100644 index 0000000000..6a76a4efbe --- /dev/null +++ b/src/main/java/org/arvados/client/logic/collection/ManifestDecoder.java @@ -0,0 +1,74 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.logic.collection; + +import org.arvados.client.common.Characters; +import org.arvados.client.exception.ArvadosClientException; +import org.arvados.client.logic.keep.KeepLocator; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.LinkedList; +import java.util.List; +import java.util.Objects; + +import static java.util.stream.Collectors.toList; +import static org.arvados.client.common.Patterns.FILE_TOKEN_PATTERN; +import static org.arvados.client.common.Patterns.LOCATOR_PATTERN; + +public class ManifestDecoder { + + public List decode(String manifestText) { + + if (manifestText == null || manifestText.isEmpty()) { + throw new ArvadosClientException("Manifest text cannot be empty."); + } + + List manifestStreams = new ArrayList<>(Arrays.asList(manifestText.split("\\n"))); + if (!manifestStreams.get(0).startsWith(". ")) { + throw new ArvadosClientException("Invalid first path component (expecting \".\")"); + } + + return manifestStreams.stream() + .map(this::decodeSingleManifestStream) + .collect(toList()); + } + + private ManifestStream decodeSingleManifestStream(String manifestStream) { + Objects.requireNonNull(manifestStream, "Manifest stream cannot be empty."); + + LinkedList manifestPieces = new LinkedList<>(Arrays.asList(manifestStream.split("\\s+"))); + String streamName = manifestPieces.poll(); + String path = ".".equals(streamName) ? "" : streamName.substring(2).concat(Characters.SLASH); + + List keepLocators = manifestPieces + .stream() + .filter(p -> p.matches(LOCATOR_PATTERN)) + .map(this::getKeepLocator) + .collect(toList()); + + + List fileTokens = manifestPieces.stream() + .skip(keepLocators.size()) + .filter(p -> p.matches(FILE_TOKEN_PATTERN)) + .map(p -> new FileToken(p, path)) + .collect(toList()); + + return new ManifestStream(streamName, keepLocators, fileTokens); + + } + + private KeepLocator getKeepLocator(String locatorString ) { + try { + return new KeepLocator(locatorString); + } catch (Exception e) { + throw new RuntimeException(e); + } + } + +} diff --git a/src/main/java/org/arvados/client/logic/collection/ManifestFactory.java b/src/main/java/org/arvados/client/logic/collection/ManifestFactory.java new file mode 100644 index 0000000000..96d605dd95 --- /dev/null +++ b/src/main/java/org/arvados/client/logic/collection/ManifestFactory.java @@ -0,0 +1,67 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.logic.collection; + +import com.google.common.collect.ImmutableList; +import org.arvados.client.common.Characters; + +import java.io.File; +import java.util.Collection; +import java.util.List; +import java.util.stream.Collectors; + +public class ManifestFactory { + + private Collection files; + private List locators; + + ManifestFactory(Collection files, List locators) { + this.files = files; + this.locators = locators; + } + + public static ManifestFactoryBuilder builder() { + return new ManifestFactoryBuilder(); + } + + public String create() { + ImmutableList.Builder builder = new ImmutableList.Builder() + .add(Characters.DOT) + .addAll(locators); + long filePosition = 0; + for (File file : files) { + builder.add(String.format("%d:%d:%s", filePosition, file.length(), file.getName().replace(" ", Characters.SPACE))); + filePosition += file.length(); + } + String manifest = builder.build().stream().collect(Collectors.joining(" ")).concat(Characters.NEW_LINE); + return manifest; + } + + public static class ManifestFactoryBuilder { + private Collection files; + private List locators; + + ManifestFactoryBuilder() { + } + + public ManifestFactory.ManifestFactoryBuilder files(Collection files) { + this.files = files; + return this; + } + + public ManifestFactory.ManifestFactoryBuilder locators(List locators) { + this.locators = locators; + return this; + } + + public ManifestFactory build() { + return new ManifestFactory(files, locators); + } + + } +} diff --git a/src/main/java/org/arvados/client/logic/collection/ManifestStream.java b/src/main/java/org/arvados/client/logic/collection/ManifestStream.java new file mode 100644 index 0000000000..30440300e4 --- /dev/null +++ b/src/main/java/org/arvados/client/logic/collection/ManifestStream.java @@ -0,0 +1,45 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.logic.collection; + +import org.arvados.client.logic.keep.KeepLocator; + +import java.util.List; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +public class ManifestStream { + + private String streamName; + private List keepLocators; + private List fileTokens; + + public ManifestStream(String streamName, List keepLocators, List fileTokens) { + this.streamName = streamName; + this.keepLocators = keepLocators; + this.fileTokens = fileTokens; + } + + @Override + public String toString() { + return streamName + " " + Stream.concat(keepLocators.stream().map(KeepLocator::toString), fileTokens.stream().map(FileToken::toString)) + .collect(Collectors.joining(" ")); + } + + public String getStreamName() { + return this.streamName; + } + + public List getKeepLocators() { + return this.keepLocators; + } + + public List getFileTokens() { + return this.fileTokens; + } +} diff --git a/src/main/java/org/arvados/client/logic/keep/FileDownloader.java b/src/main/java/org/arvados/client/logic/keep/FileDownloader.java new file mode 100644 index 0000000000..1f694f25c2 --- /dev/null +++ b/src/main/java/org/arvados/client/logic/keep/FileDownloader.java @@ -0,0 +1,256 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.logic.keep; + +import com.google.common.collect.Lists; +import org.arvados.client.api.client.CollectionsApiClient; +import org.arvados.client.api.client.KeepWebApiClient; +import org.arvados.client.api.model.Collection; +import org.arvados.client.common.Characters; +import org.arvados.client.exception.ArvadosClientException; +import org.arvados.client.logic.collection.FileToken; +import org.arvados.client.logic.collection.ManifestDecoder; +import org.arvados.client.logic.collection.ManifestStream; +import org.arvados.client.logic.keep.exception.DownloadFolderAlreadyExistsException; +import org.arvados.client.logic.keep.exception.FileAlreadyExistsException; +import org.slf4j.Logger; + +import java.io.File; +import java.io.FileOutputStream; +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.CompletableFuture; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +public class FileDownloader { + + private final KeepClient keepClient; + private final ManifestDecoder manifestDecoder; + private final CollectionsApiClient collectionsApiClient; + private final KeepWebApiClient keepWebApiClient; + private final Logger log = org.slf4j.LoggerFactory.getLogger(FileDownloader.class); + + public FileDownloader(KeepClient keepClient, ManifestDecoder manifestDecoder, CollectionsApiClient collectionsApiClient, KeepWebApiClient keepWebApiClient) { + this.keepClient = keepClient; + this.manifestDecoder = manifestDecoder; + this.collectionsApiClient = collectionsApiClient; + this.keepWebApiClient = keepWebApiClient; + } + + public List listFileInfoFromCollection(String collectionUuid) { + Collection requestedCollection = collectionsApiClient.get(collectionUuid); + String manifestText = requestedCollection.getManifestText(); + + // decode manifest text and get list of all FileTokens for this collection + return manifestDecoder.decode(manifestText) + .stream() + .flatMap(p -> p.getFileTokens().stream()) + .collect(Collectors.toList()); + } + + public File downloadSingleFileUsingKeepWeb(String filePathName, String collectionUuid, String pathToDownloadFolder) { + FileToken fileToken = getFileTokenFromCollection(filePathName, collectionUuid); + if (fileToken == null) { + throw new ArvadosClientException(String.format("%s not found in Collection with UUID %s", filePathName, collectionUuid)); + } + + File downloadedFile = checkIfFileExistsInTargetLocation(fileToken, pathToDownloadFolder); + try (FileOutputStream fos = new FileOutputStream(downloadedFile)) { + fos.write(keepWebApiClient.download(collectionUuid, filePathName)); + } catch (IOException e) { + throw new ArvadosClientException(String.format("Unable to write down file %s", fileToken.getFileName()), e); + } + return downloadedFile; + } + + public List downloadFilesFromCollectionUsingKeepWeb(String collectionUuid, String pathToDownloadFolder) { + String collectionTargetDir = setTargetDirectory(collectionUuid, pathToDownloadFolder).getAbsolutePath(); + List fileTokens = listFileInfoFromCollection(collectionUuid); + + List> futures = Lists.newArrayList(); + for (FileToken fileToken : fileTokens) { + futures.add(CompletableFuture.supplyAsync(() -> this.downloadOneFileFromCollectionUsingKeepWeb(fileToken, collectionUuid, collectionTargetDir))); + } + + @SuppressWarnings("unchecked") + CompletableFuture[] array = futures.toArray(new CompletableFuture[0]); + return Stream.of(array) + .map(CompletableFuture::join).collect(Collectors.toList()); + } + + private FileToken getFileTokenFromCollection(String filePathName, String collectionUuid) { + return listFileInfoFromCollection(collectionUuid) + .stream() + .filter(p -> (p.getFullPath()).equals(filePathName)) + .findFirst() + .orElse(null); + } + + private File checkIfFileExistsInTargetLocation(FileToken fileToken, String pathToDownloadFolder) { + String fileName = fileToken.getFileName(); + + File downloadFile = new File(pathToDownloadFolder + Characters.SLASH + fileName); + if (downloadFile.exists()) { + throw new FileAlreadyExistsException(String.format("File %s exists in location %s", fileName, pathToDownloadFolder)); + } else { + return downloadFile; + } + } + + private File downloadOneFileFromCollectionUsingKeepWeb(FileToken fileToken, String collectionUuid, String pathToDownloadFolder) { + String filePathName = fileToken.getPath() + fileToken.getFileName(); + File downloadedFile = new File(pathToDownloadFolder + Characters.SLASH + filePathName); + downloadedFile.getParentFile().mkdirs(); + + try (FileOutputStream fos = new FileOutputStream(downloadedFile)) { + fos.write(keepWebApiClient.download(collectionUuid, filePathName)); + } catch (IOException e) { + throw new RuntimeException(e); + } + return downloadedFile; + } + + public List downloadFilesFromCollection(String collectionUuid, String pathToDownloadFolder) { + + // download requested collection and extract manifest text + Collection requestedCollection = collectionsApiClient.get(collectionUuid); + String manifestText = requestedCollection.getManifestText(); + + // if directory with this collectionUUID does not exist - create one + // if exists - abort (throw exception) + File collectionTargetDir = setTargetDirectory(collectionUuid, pathToDownloadFolder); + + // decode manifest text and create list of ManifestStream objects containing KeepLocators and FileTokens + List manifestStreams = manifestDecoder.decode(manifestText); + + //list of all downloaded files that will be returned by this method + List downloadedFilesFromCollection = new ArrayList<>(); + + // download files for each manifest stream + for (ManifestStream manifestStream : manifestStreams) + downloadedFilesFromCollection.addAll(downloadFilesFromSingleManifestStream(manifestStream, collectionTargetDir)); + + log.debug(String.format("Total of: %d files downloaded", downloadedFilesFromCollection.size())); + return downloadedFilesFromCollection; + } + + private File setTargetDirectory(String collectionUUID, String pathToDownloadFolder) { + //local directory to save downloaded files + File collectionTargetDir = new File(pathToDownloadFolder + Characters.SLASH + collectionUUID); + if (collectionTargetDir.exists()) { + throw new DownloadFolderAlreadyExistsException(String.format("Directory for collection UUID %s already exists", collectionUUID)); + } else { + collectionTargetDir.mkdirs(); + } + return collectionTargetDir; + } + + private List downloadFilesFromSingleManifestStream(ManifestStream manifestStream, File collectionTargetDir){ + List downloadedFiles = new ArrayList<>(); + List keepLocators = manifestStream.getKeepLocators(); + DownloadHelper downloadHelper = new DownloadHelper(keepLocators); + + for (FileToken fileToken : manifestStream.getFileTokens()) { + File downloadedFile = new File(collectionTargetDir.getAbsolutePath() + Characters.SLASH + fileToken.getFullPath()); //create file + downloadedFile.getParentFile().mkdirs(); + + try (FileOutputStream fos = new FileOutputStream(downloadedFile, true)) { + downloadHelper.setBytesToDownload(fileToken.getFileSize()); //update file size info + + //this part needs to be repeated for each file until whole file is downloaded + do { + downloadHelper.requestNewDataChunk(); //check if new data chunk needs to be downloaded + downloadHelper.writeDownFile(fos); // download data from chunk + } while (downloadHelper.getBytesToDownload() != 0); + + } catch (IOException | ArvadosClientException e) { + throw new ArvadosClientException(String.format("Unable to write down file %s", fileToken.getFileName()), e); + } + + downloadedFiles.add(downloadedFile); + log.debug(String.format("File %d / %d downloaded from manifest stream", + manifestStream.getFileTokens().indexOf(fileToken) + 1, + manifestStream.getFileTokens().size())); + } + return downloadedFiles; + } + + private class DownloadHelper { + + // values for tracking file output streams and matching data chunks with initial files + int currentDataChunkNumber; + int bytesDownloadedFromChunk; + int bytesToDownload; + byte[] currentDataChunk; + boolean remainingDataInChunk; + final List keepLocators; + + private DownloadHelper(List keepLocators) { + currentDataChunkNumber = -1; + bytesDownloadedFromChunk = 0; + remainingDataInChunk = false; + this.keepLocators = keepLocators; + } + + private int getBytesToDownload() { + return bytesToDownload; + } + + private void setBytesToDownload(int bytesToDownload) { + this.bytesToDownload = bytesToDownload; + } + + private void requestNewDataChunk() { + if (!remainingDataInChunk) { + currentDataChunkNumber++; + if (currentDataChunkNumber < keepLocators.size()) { + //swap data chunk for next one + currentDataChunk = keepClient.getDataChunk(keepLocators.get(currentDataChunkNumber)); + log.debug(String.format("%d of %d data chunks from manifest stream downloaded", currentDataChunkNumber + 1, keepLocators.size())); + } else { + throw new ArvadosClientException("Data chunk required for download is missing."); + } + } + } + + private void writeDownFile(FileOutputStream fos) throws IOException { + //case 1: more bytes needed than available in current chunk (or whole current chunk needed) to download file + if (bytesToDownload >= currentDataChunk.length - bytesDownloadedFromChunk) { + writeDownWholeDataChunk(fos); + } + //case 2: current data chunk contains more bytes than is needed for this file + else { + writeDownDataChunkPartially(fos); + } + } + + private void writeDownWholeDataChunk(FileOutputStream fos) throws IOException { + // write all remaining bytes from current chunk + fos.write(currentDataChunk, bytesDownloadedFromChunk, currentDataChunk.length - bytesDownloadedFromChunk); + //update bytesToDownload + bytesToDownload -= (currentDataChunk.length - bytesDownloadedFromChunk); + // set remaining data in chunk to false + remainingDataInChunk = false; + //reset bytesDownloadedFromChunk so that its set to 0 for the next chunk + bytesDownloadedFromChunk = 0; + } + + private void writeDownDataChunkPartially(FileOutputStream fos) throws IOException { + //write all remaining bytes for this file from current chunk + fos.write(currentDataChunk, bytesDownloadedFromChunk, bytesToDownload); + // update number of bytes downloaded from this chunk + bytesDownloadedFromChunk += bytesToDownload; + // set remaining data in chunk to true + remainingDataInChunk = true; + // reset bytesToDownload to exit while loop and move to the next file + bytesToDownload = 0; + } + } +} \ No newline at end of file diff --git a/src/main/java/org/arvados/client/logic/keep/FileTransferHandler.java b/src/main/java/org/arvados/client/logic/keep/FileTransferHandler.java new file mode 100644 index 0000000000..c6a8ad3687 --- /dev/null +++ b/src/main/java/org/arvados/client/logic/keep/FileTransferHandler.java @@ -0,0 +1,55 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.logic.keep; + +import org.arvados.client.api.client.KeepServerApiClient; +import org.arvados.client.exception.ArvadosApiException; +import org.arvados.client.config.ConfigProvider; +import org.slf4j.Logger; + +import java.io.File; +import java.util.Map; + +public class FileTransferHandler { + + private final String host; + private final KeepServerApiClient keepServerApiClient; + private final Map headers; + private final Logger log = org.slf4j.LoggerFactory.getLogger(FileTransferHandler.class); + + public FileTransferHandler(String host, Map headers, ConfigProvider config) { + this.host = host; + this.headers = headers; + this.keepServerApiClient = new KeepServerApiClient(config); + } + + public String put(String hashString, File body) { + String url = host + hashString; + String locator = null; + try { + locator = keepServerApiClient.upload(url, headers, body); + } catch (ArvadosApiException e) { + log.error("Cannot upload file to Keep server.", e); + } + return locator; + } + + public byte[] get(KeepLocator locator) { + return get(locator.stripped(), locator.permissionHint()); + } + + public byte[] get(String blockLocator, String authToken) { + String url = host + blockLocator + "+" + authToken; + try { + return keepServerApiClient.download(url); + } catch (ArvadosApiException e) { + log.error("Cannot download file from Keep server.", e); + return null; + } + } +} diff --git a/src/main/java/org/arvados/client/logic/keep/FileUploader.java b/src/main/java/org/arvados/client/logic/keep/FileUploader.java new file mode 100644 index 0000000000..52e0f66caf --- /dev/null +++ b/src/main/java/org/arvados/client/logic/keep/FileUploader.java @@ -0,0 +1,101 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.logic.keep; + +import com.google.common.collect.Lists; +import org.arvados.client.api.client.CollectionsApiClient; +import org.arvados.client.api.model.Collection; +import org.arvados.client.common.Characters; +import org.arvados.client.config.ConfigProvider; +import org.arvados.client.exception.ArvadosClientException; +import org.arvados.client.logic.collection.CollectionFactory; +import org.arvados.client.utils.FileMerge; +import org.arvados.client.utils.FileSplit; +import org.slf4j.Logger; + +import java.io.File; +import java.io.IOException; +import java.util.List; +import java.util.Objects; +import java.util.UUID; + +import static java.util.stream.Collectors.toList; + +public class FileUploader { + + private final KeepClient keepClient; + private final CollectionsApiClient collectionsApiClient; + private final ConfigProvider config; + private final Logger log = org.slf4j.LoggerFactory.getLogger(FileUploader.class); + + public FileUploader(KeepClient keepClient, CollectionsApiClient collectionsApiClient, ConfigProvider config) { + this.keepClient = keepClient; + this.collectionsApiClient = collectionsApiClient; + this.config = config; + } + + public Collection upload(List sourceFiles, String collectionName, String projectUuid) { + List locators = uploadToKeep(sourceFiles); + CollectionFactory collectionFactory = CollectionFactory.builder() + .config(config) + .name(collectionName) + .projectUuid(projectUuid) + .manifestFiles(sourceFiles) + .manifestLocators(locators) + .build(); + + Collection newCollection = collectionFactory.create(); + return collectionsApiClient.create(newCollection); + } + + public Collection uploadToExistingCollection(List files, String collectionUuid) { + List locators = uploadToKeep(files); + Collection collectionBeforeUpload = collectionsApiClient.get(collectionUuid); + String oldManifest = collectionBeforeUpload.getManifestText(); + + CollectionFactory collectionFactory = CollectionFactory.builder() + .config(config) + .manifestFiles(files) + .manifestLocators(locators).build(); + + String newPartOfManifestText = collectionFactory.create().getManifestText(); + String newManifest = oldManifest + newPartOfManifestText; + + collectionBeforeUpload.setManifestText(newManifest); + return collectionsApiClient.update(collectionBeforeUpload); + } + + private List uploadToKeep(List files) { + File targetDir = config.getFileSplitDirectory(); + File combinedFile = new File(targetDir.getAbsolutePath() + Characters.SLASH + UUID.randomUUID()); + List chunks; + try { + FileMerge.merge(files, combinedFile); + chunks = FileSplit.split(combinedFile, targetDir, config.getFileSplitSize()); + } catch (IOException e) { + throw new ArvadosClientException("Cannot create file chunks for upload", e); + } + combinedFile.delete(); + + int copies = config.getNumberOfCopies(); + int numRetries = config.getNumberOfRetries(); + + List locators = Lists.newArrayList(); + for (File chunk : chunks) { + try { + locators.add(keepClient.put(chunk, copies, numRetries)); + } catch (ArvadosClientException e) { + log.error("Problem occurred while uploading chunk file {}", chunk.getName(), e); + throw e; + } + } + return locators.stream() + .filter(Objects::nonNull) + .collect(toList()); + } +} diff --git a/src/main/java/org/arvados/client/logic/keep/KeepClient.java b/src/main/java/org/arvados/client/logic/keep/KeepClient.java new file mode 100644 index 0000000000..9cc732d46d --- /dev/null +++ b/src/main/java/org/arvados/client/logic/keep/KeepClient.java @@ -0,0 +1,244 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.logic.keep; + +import com.google.common.collect.Lists; +import org.apache.commons.codec.digest.DigestUtils; +import org.apache.commons.io.FileUtils; +import org.arvados.client.api.client.KeepServicesApiClient; +import org.arvados.client.api.model.KeepService; +import org.arvados.client.api.model.KeepServiceList; +import org.arvados.client.common.Characters; +import org.arvados.client.common.Headers; +import org.arvados.client.config.ConfigProvider; +import org.arvados.client.exception.ArvadosApiException; +import org.arvados.client.exception.ArvadosClientException; +import org.slf4j.Logger; + +import java.io.File; +import java.io.IOException; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.concurrent.CompletableFuture; +import java.util.function.Function; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +public class KeepClient { + + private final KeepServicesApiClient keepServicesApiClient; + private final Logger log = org.slf4j.LoggerFactory.getLogger(KeepClient.class); + private List keepServices; + private List writableServices; + private Map gatewayServices; + private final String apiToken; + private Integer maxReplicasPerService; + private final ConfigProvider config; + + public KeepClient(ConfigProvider config) { + this.config = config; + keepServicesApiClient = new KeepServicesApiClient(config); + apiToken = config.getApiToken(); + } + + public byte[] getDataChunk(KeepLocator keepLocator) { + + Map headers = new HashMap<>(); + Map rootsMap = new HashMap<>(); + + List sortedRoots = mapNewServices(rootsMap, keepLocator, false, false, headers); + + byte[] dataChunk = sortedRoots + .stream() + .map(rootsMap::get) + .map(r -> r.get(keepLocator)) + .filter(Objects::nonNull) + .findFirst() + .orElse(null); + + if (dataChunk == null) { + throw new ArvadosClientException("No server responding. Unable to download data chunk."); + } + + return dataChunk; + } + + public String put(File data, int copies, int numRetries) { + + byte[] fileBytes; + try { + fileBytes = FileUtils.readFileToByteArray(data); + } catch (IOException e) { + throw new ArvadosClientException("An error occurred while reading data chunk", e); + } + + String dataHash = DigestUtils.md5Hex(fileBytes); + String locatorString = String.format("%s+%d", dataHash, data.length()); + + if (copies < 1) { + return locatorString; + } + KeepLocator locator = new KeepLocator(locatorString); + + // Tell the proxy how many copies we want it to store + Map headers = new HashMap<>(); + headers.put(Headers.X_KEEP_DESIRED_REPLICAS, String.valueOf(copies)); + + Map rootsMap = new HashMap<>(); + List sortedRoots = mapNewServices(rootsMap, locator, false, true, headers); + + int numThreads = 0; + if (maxReplicasPerService == null || maxReplicasPerService >= copies) { + numThreads = 1; + } else { + numThreads = ((Double) Math.ceil(1.0 * copies / maxReplicasPerService)).intValue(); + } + log.debug("Pool max threads is {}", numThreads); + + List> futures = Lists.newArrayList(); + for (int i = 0; i < numThreads; i++) { + String root = sortedRoots.get(i); + FileTransferHandler keepServiceLocal = rootsMap.get(root); + futures.add(CompletableFuture.supplyAsync(() -> keepServiceLocal.put(dataHash, data))); + } + + @SuppressWarnings("unchecked") + CompletableFuture[] array = futures.toArray(new CompletableFuture[0]); + + return Stream.of(array) + .map(CompletableFuture::join) + .reduce((a, b) -> b) + .orElse(null); + } + + private List mapNewServices(Map rootsMap, KeepLocator locator, + boolean forceRebuild, boolean needWritable, Map headers) { + + headers.putIfAbsent("Authorization", String.format("OAuth2 %s", apiToken)); + List localRoots = weightedServiceRoots(locator, forceRebuild, needWritable); + for (String root : localRoots) { + FileTransferHandler keepServiceLocal = new FileTransferHandler(root, headers, config); + rootsMap.putIfAbsent(root, keepServiceLocal); + } + return localRoots; + } + + /** + * Return an array of Keep service endpoints, in the order in which they should be probed when reading or writing + * data with the given hash+hints. + */ + private List weightedServiceRoots(KeepLocator locator, boolean forceRebuild, boolean needWritable) { + + buildServicesList(forceRebuild); + + List sortedRoots = new ArrayList<>(); + + // Use the services indicated by the given +K@... remote + // service hints, if any are present and can be resolved to a + // URI. + // + for (String hint : locator.getHints()) { + if (hint.startsWith("K@")) { + if (hint.length() == 7) { + sortedRoots.add(String.format("https://keep.%s.arvadosapi.com/", hint.substring(2))); + } else if (hint.length() == 29) { + KeepService svc = gatewayServices.get(hint.substring(2)); + if (svc != null) { + sortedRoots.add(svc.getServiceRoot()); + } + } + } + } + + // Sort the available local services by weight (heaviest first) + // for this locator, and return their service_roots (base URIs) + // in that order. + List useServices = keepServices; + if (needWritable) { + useServices = writableServices; + } + anyNonDiskServices(useServices); + + sortedRoots.addAll(useServices + .stream() + .sorted((ks1, ks2) -> serviceWeight(locator.getMd5sum(), ks2.getUuid()) + .compareTo(serviceWeight(locator.getMd5sum(), ks1.getUuid()))) + .map(KeepService::getServiceRoot) + .collect(Collectors.toList())); + + return sortedRoots; + } + + private void buildServicesList(boolean forceRebuild) { + if (keepServices != null && !forceRebuild) { + return; + } + KeepServiceList keepServiceList; + try { + keepServiceList = keepServicesApiClient.accessible(); + } catch (ArvadosApiException e) { + throw new ArvadosClientException("Cannot obtain list of accessible keep services"); + } + // Gateway services are only used when specified by UUID, + // so there's nothing to gain by filtering them by + // service_type. + gatewayServices = keepServiceList.getItems().stream().collect(Collectors.toMap(KeepService::getUuid, Function.identity())); + + if (gatewayServices.isEmpty()) { + throw new ArvadosClientException("No gateway services available!"); + } + + // Precompute the base URI for each service. + for (KeepService keepService : gatewayServices.values()) { + String serviceHost = keepService.getServiceHost(); + if (!serviceHost.startsWith("[") && serviceHost.contains(Characters.COLON)) { + // IPv6 URIs must be formatted like http://[::1]:80/... + serviceHost = String.format("[%s]", serviceHost); + } + + String protocol = keepService.getServiceSslFlag() ? "https" : "http"; + String serviceRoot = String.format("%s://%s:%d/", protocol, serviceHost, keepService.getServicePort()); + keepService.setServiceRoot(serviceRoot); + } + + keepServices = gatewayServices.values().stream().filter(ks -> !ks.getServiceType().startsWith("gateway:")).collect(Collectors.toList()); + writableServices = keepServices.stream().filter(ks -> !ks.getReadOnly()).collect(Collectors.toList()); + + // For disk type services, max_replicas_per_service is 1 + // It is unknown (unlimited) for other service types. + if (anyNonDiskServices(writableServices)) { + maxReplicasPerService = null; + } else { + maxReplicasPerService = 1; + } + } + + private Boolean anyNonDiskServices(List useServices) { + return useServices.stream().anyMatch(ks -> !ks.getServiceType().equals("disk")); + } + + /** + * Compute the weight of a Keep service endpoint for a data block with a known hash. + *

+ * The weight is md5(h + u) where u is the last 15 characters of the service endpoint's UUID. + */ + private static String serviceWeight(String dataHash, String serviceUuid) { + String shortenedUuid; + if (serviceUuid != null && serviceUuid.length() >= 15) { + int substringIndex = serviceUuid.length() - 15; + shortenedUuid = serviceUuid.substring(substringIndex); + } else { + shortenedUuid = (serviceUuid == null) ? "" : serviceUuid; + } + return DigestUtils.md5Hex(dataHash + shortenedUuid); + } + +} diff --git a/src/main/java/org/arvados/client/logic/keep/KeepLocator.java b/src/main/java/org/arvados/client/logic/keep/KeepLocator.java new file mode 100644 index 0000000000..4d3d42523c --- /dev/null +++ b/src/main/java/org/arvados/client/logic/keep/KeepLocator.java @@ -0,0 +1,86 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.logic.keep; + +import org.arvados.client.exception.ArvadosClientException; + +import java.time.Instant; +import java.time.LocalDateTime; +import java.time.ZoneOffset; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.LinkedList; +import java.util.List; +import java.util.Objects; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +import static org.arvados.client.common.Patterns.HINT_PATTERN; + +public class KeepLocator { + + private final List hints = new ArrayList<>(); + private String permSig; + private LocalDateTime permExpiry; + private final String md5sum; + private final Integer size; + + public KeepLocator(String locatorString) { + LinkedList pieces = new LinkedList<>(Arrays.asList(locatorString.split("\\+"))); + + md5sum = pieces.poll(); + size = Integer.valueOf(Objects.requireNonNull(pieces.poll())); + + for (String hint : pieces) { + if (!hint.matches(HINT_PATTERN)) { + throw new ArvadosClientException(String.format("invalid hint format: %s", hint)); + } else if (hint.startsWith("A")) { + parsePermissionHint(hint); + } else { + hints.add(hint); + } + } + } + + public List getHints() { + return hints; + } + + public String getMd5sum() { + return md5sum; + } + + @Override + public String toString() { + return Stream.concat(Stream.of(md5sum, size.toString(), permissionHint()), hints.stream()) + .filter(Objects::nonNull) + .collect(Collectors.joining("+")); + } + + public String stripped() { + return size != null ? String.format("%s+%d", md5sum, size) : md5sum; + } + + public String permissionHint() { + if (permSig == null || permExpiry == null) { + return null; + } + + long timestamp = permExpiry.toEpochSecond(ZoneOffset.UTC); + String signTimestamp = Long.toHexString(timestamp); + return String.format("A%s@%s", permSig, signTimestamp); + } + + private void parsePermissionHint(String hint) { + String[] hintSplit = hint.substring(1).split("@", 2); + permSig = hintSplit[0]; + + int permExpiryDecimal = Integer.parseInt(hintSplit[1], 16); + permExpiry = LocalDateTime.ofInstant(Instant.ofEpochSecond(permExpiryDecimal), ZoneOffset.UTC); + } +} diff --git a/src/main/java/org/arvados/client/logic/keep/exception/DownloadFolderAlreadyExistsException.java b/src/main/java/org/arvados/client/logic/keep/exception/DownloadFolderAlreadyExistsException.java new file mode 100644 index 0000000000..9968ff0daf --- /dev/null +++ b/src/main/java/org/arvados/client/logic/keep/exception/DownloadFolderAlreadyExistsException.java @@ -0,0 +1,24 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.logic.keep.exception; + +import org.arvados.client.exception.ArvadosClientException; + +/** + * Exception indicating that directory with given name was already created in specified location. + * + *

This exception will be thrown during an attempt to download all files from certain + * collection to a location that already contains folder named by this collection's UUID.

+ */ +public class DownloadFolderAlreadyExistsException extends ArvadosClientException { + + public DownloadFolderAlreadyExistsException(String message) { + super(message); + } + +} diff --git a/src/main/java/org/arvados/client/logic/keep/exception/FileAlreadyExistsException.java b/src/main/java/org/arvados/client/logic/keep/exception/FileAlreadyExistsException.java new file mode 100644 index 0000000000..ea02ffc84d --- /dev/null +++ b/src/main/java/org/arvados/client/logic/keep/exception/FileAlreadyExistsException.java @@ -0,0 +1,23 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.logic.keep.exception; + +import org.arvados.client.exception.ArvadosClientException; + +/** + * Signals that an attempt to download a file with given name has failed for a specified + * download location. + * + *

This exception will be thrown during an attempt to download single file to a location + * that already contains file with given name

+ */ +public class FileAlreadyExistsException extends ArvadosClientException { + + public FileAlreadyExistsException(String message) { super(message); } + +} diff --git a/src/main/java/org/arvados/client/utils/FileMerge.java b/src/main/java/org/arvados/client/utils/FileMerge.java new file mode 100644 index 0000000000..eaabbaaad6 --- /dev/null +++ b/src/main/java/org/arvados/client/utils/FileMerge.java @@ -0,0 +1,26 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.utils; + +import java.io.BufferedOutputStream; +import java.io.File; +import java.io.FileOutputStream; +import java.io.IOException; +import java.nio.file.Files; +import java.util.Collection; + +public class FileMerge { + + public static void merge(Collection files, File targetFile) throws IOException { + try (FileOutputStream fos = new FileOutputStream(targetFile); BufferedOutputStream mergingStream = new BufferedOutputStream(fos)) { + for (File file : files) { + Files.copy(file.toPath(), mergingStream); + } + } + } +} diff --git a/src/main/java/org/arvados/client/utils/FileSplit.java b/src/main/java/org/arvados/client/utils/FileSplit.java new file mode 100644 index 0000000000..e118edc026 --- /dev/null +++ b/src/main/java/org/arvados/client/utils/FileSplit.java @@ -0,0 +1,44 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.utils; + +import org.apache.commons.io.FileUtils; + +import java.io.*; +import java.util.ArrayList; +import java.util.List; + +/** + * Based on: + * {@link} https://stackoverflow.com/questions/10864317/how-to-break-a-file-into-pieces-using-java + */ +public class FileSplit { + + public static List split(File f, File dir, int splitSize) throws IOException { + int partCounter = 1; + + long sizeOfFiles = splitSize * FileUtils.ONE_MB; + byte[] buffer = new byte[(int) sizeOfFiles]; + + List files = new ArrayList<>(); + String fileName = f.getName(); + + try (FileInputStream fis = new FileInputStream(f); BufferedInputStream bis = new BufferedInputStream(fis)) { + int bytesAmount = 0; + while ((bytesAmount = bis.read(buffer)) > 0) { + String filePartName = String.format("%s.%03d", fileName, partCounter++); + File newFile = new File(dir, filePartName); + try (FileOutputStream out = new FileOutputStream(newFile)) { + out.write(buffer, 0, bytesAmount); + } + files.add(newFile); + } + } + return files; + } +} \ No newline at end of file diff --git a/src/main/resources/reference.conf b/src/main/resources/reference.conf new file mode 100644 index 0000000000..3ff2bb0a98 --- /dev/null +++ b/src/main/resources/reference.conf @@ -0,0 +1,23 @@ +# Arvados client default configuration +# +# Remarks: +# * While providing data remove apostrophes ("") from each line +# * See Arvados documentation for information how to obtain a token: +# https://doc.arvados.org/user/reference/api-tokens.html +# + +arvados { + api { + keepweb-host = localhost + keepweb-port = 8000 + host = localhost + port = 8000 + token = "" + protocol = https + host-insecure = false + } + split-size = 64 + temp-dir = /tmp/file-split + copies = 2 + retries = 0 +} diff --git a/src/test/java/org/arvados/client/api/client/BaseStandardApiClientTest.java b/src/test/java/org/arvados/client/api/client/BaseStandardApiClientTest.java new file mode 100644 index 0000000000..73b559afd4 --- /dev/null +++ b/src/test/java/org/arvados/client/api/client/BaseStandardApiClientTest.java @@ -0,0 +1,50 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.client; + +import okhttp3.HttpUrl; +import org.arvados.client.api.model.Item; +import org.arvados.client.api.model.ItemList; +import org.arvados.client.test.utils.ArvadosClientUnitTest; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.Spy; +import org.mockito.junit.MockitoJUnitRunner; + +import static org.assertj.core.api.Assertions.assertThat; + +@RunWith(MockitoJUnitRunner.class) +public class BaseStandardApiClientTest extends ArvadosClientUnitTest { + + @Spy + private BaseStandardApiClient client = new BaseStandardApiClient(CONFIG) { + @Override + String getResource() { + return "resource"; + } + + @Override + Class getType() { + return null; + } + + @Override + Class getListType() { + return null; + } + }; + + @Test + public void urlBuilderBuildsExpectedUrlFormat() { + // when + HttpUrl.Builder actual = client.getUrlBuilder(); + + // then + assertThat(actual.build().toString()).isEqualTo("http://localhost:9000/arvados/v1/resource"); + } +} diff --git a/src/test/java/org/arvados/client/api/client/CollectionsApiClientTest.java b/src/test/java/org/arvados/client/api/client/CollectionsApiClientTest.java new file mode 100644 index 0000000000..8da3bfbf51 --- /dev/null +++ b/src/test/java/org/arvados/client/api/client/CollectionsApiClientTest.java @@ -0,0 +1,112 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.client; + +import okhttp3.mockwebserver.RecordedRequest; +import org.arvados.client.api.model.Collection; +import org.arvados.client.api.model.CollectionList; +import org.arvados.client.test.utils.RequestMethod; +import org.arvados.client.test.utils.ArvadosClientMockedWebServerTest; +import org.junit.Test; + +import static org.arvados.client.test.utils.ApiClientTestUtils.*; +import static org.assertj.core.api.Assertions.assertThat; + +public class CollectionsApiClientTest extends ArvadosClientMockedWebServerTest { + + private static final String RESOURCE = "collections"; + + private CollectionsApiClient client = new CollectionsApiClient(CONFIG); + + @Test + public void listCollections() throws Exception { + + // given + server.enqueue(getResponse("collections-list")); + + // when + CollectionList actual = client.list(); + + // then + RecordedRequest request = server.takeRequest(); + assertAuthorizationHeader(request); + assertRequestPath(request, RESOURCE); + assertRequestMethod(request, RequestMethod.GET); + assertThat(actual.getItemsAvailable()).isEqualTo(41); + } + + @Test + public void getCollection() throws Exception { + + // given + server.enqueue(getResponse("collections-get")); + + String uuid = "112ci-4zz18-p51w7z3fpopo6sm"; + + // when + Collection actual = client.get(uuid); + + // then + RecordedRequest request = server.takeRequest(); + assertAuthorizationHeader(request); + assertRequestPath(request, RESOURCE + "/" + uuid); + assertRequestMethod(request, RequestMethod.GET); + assertThat(actual.getUuid()).isEqualTo(uuid); + assertThat(actual.getPortableDataHash()).isEqualTo("6c4106229b08fe25f48b3a7a8289dd46+143"); + } + + @Test + public void createCollection() throws Exception { + + // given + server.enqueue(getResponse("collections-create-simple")); + + String name = "Super Collection"; + + Collection collection = new Collection(); + collection.setName(name); + + // when + Collection actual = client.create(collection); + + // then + RecordedRequest request = server.takeRequest(); + assertAuthorizationHeader(request); + assertRequestPath(request, RESOURCE); + assertRequestMethod(request, RequestMethod.POST); + assertThat(actual.getName()).isEqualTo(name); + assertThat(actual.getPortableDataHash()).isEqualTo("d41d8cd98f00b204e9800998ecf8427e+0"); + assertThat(actual.getManifestText()).isEmpty(); + } + + @Test + public void createCollectionWithManifest() throws Exception { + + // given + server.enqueue(getResponse("collections-create-manifest")); + + String name = "Super Collection"; + String manifestText = ". 7df44272090cee6c0732382bba415ee9+70+Aa5ece4560e3329315165b36c239b8ab79c888f8a@5a1d5708 0:70:README.md\n"; + + Collection collection = new Collection(); + collection.setName(name); + collection.setManifestText(manifestText); + + // when + Collection actual = client.create(collection); + + // then + RecordedRequest request = server.takeRequest(); + assertAuthorizationHeader(request); + assertRequestPath(request, RESOURCE); + assertRequestMethod(request, RequestMethod.POST); + assertThat(actual.getName()).isEqualTo(name); + assertThat(actual.getPortableDataHash()).isEqualTo("d41d8cd98f00b204e9800998ecf8427e+0"); + assertThat(actual.getManifestText()).isEqualTo(manifestText); + } +} diff --git a/src/test/java/org/arvados/client/api/client/GroupsApiClientTest.java b/src/test/java/org/arvados/client/api/client/GroupsApiClientTest.java new file mode 100644 index 0000000000..6bb385a4ca --- /dev/null +++ b/src/test/java/org/arvados/client/api/client/GroupsApiClientTest.java @@ -0,0 +1,95 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.client; + +import com.google.common.collect.Lists; +import okhttp3.mockwebserver.RecordedRequest; +import org.arvados.client.api.model.Group; +import org.arvados.client.api.model.GroupList; +import org.arvados.client.api.model.argument.Filter; +import org.arvados.client.api.model.argument.ListArgument; +import org.arvados.client.test.utils.RequestMethod; +import org.arvados.client.test.utils.ArvadosClientMockedWebServerTest; +import org.junit.Test; + +import java.util.Arrays; + +import static org.arvados.client.test.utils.ApiClientTestUtils.*; +import static org.junit.Assert.assertEquals; + +public class GroupsApiClientTest extends ArvadosClientMockedWebServerTest { + private static final String RESOURCE = "groups"; + private GroupsApiClient client = new GroupsApiClient(CONFIG); + + @Test + public void listGroups() throws Exception { + + // given + server.enqueue(getResponse("groups-list")); + + // when + GroupList actual = client.list(); + + // then + RecordedRequest request = server.takeRequest(); + assertAuthorizationHeader(request); + assertRequestPath(request, RESOURCE); + assertRequestMethod(request, RequestMethod.GET); + assertEquals(20, actual.getItems().size()); + } + + @Test + public void listProjectsByOwner() throws Exception { + + // given + server.enqueue(getResponse("groups-list")); + String ownerUuid = "ardev-tpzed-n3kzq4fvoks3uw4"; + String filterSubPath = "?filters=[%20[%20%22owner_uuid%22,%20%22like%22,%20%22ardev-tpzed-n3kzq4fvoks3uw4%22%20],%20" + + "[%20%22group_class%22,%20%22in%22,%20[%20%22project%22,%20%22sub-project%22%20]%20]%20]"; + + // when + ListArgument listArgument = ListArgument.builder() + .filters(Arrays.asList( + Filter.of("owner_uuid", Filter.Operator.LIKE, ownerUuid), + Filter.of("group_class", Filter.Operator.IN, Lists.newArrayList("project", "sub-project") + ))) + .build(); + GroupList actual = client.list(listArgument); + + // then + RecordedRequest request = server.takeRequest(); + assertAuthorizationHeader(request); + assertRequestPath(request, RESOURCE + filterSubPath); + assertRequestMethod(request, RequestMethod.GET); + assertEquals(20, actual.getItems().size()); + } + + @Test + public void getGroup() throws Exception { + + // given + server.enqueue(getResponse("groups-get")); + + String uuid = "ardev-j7d0g-bmg3pfqtx3ivczp"; + + // when + Group actual = client.get(uuid); + + // then + RecordedRequest request = server.takeRequest(); + assertAuthorizationHeader(request); + assertRequestPath(request, RESOURCE + "/" + uuid); + assertRequestMethod(request, RequestMethod.GET); + assertEquals(uuid, actual.getUuid()); + assertEquals("3hw0vk4mbl0ofvia5k6x4dwrx", actual.getEtag()); + assertEquals("ardev-tpzed-n3kzq4fvoks3uw4", actual.getOwnerUuid()); + assertEquals("TestGroup1", actual.getName()); + assertEquals("project", actual.getGroupClass()); + + } +} diff --git a/src/test/java/org/arvados/client/api/client/KeepServerApiClientTest.java b/src/test/java/org/arvados/client/api/client/KeepServerApiClientTest.java new file mode 100644 index 0000000000..50a9cc1a6d --- /dev/null +++ b/src/test/java/org/arvados/client/api/client/KeepServerApiClientTest.java @@ -0,0 +1,70 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.client; + +import com.google.common.collect.Maps; +import okhttp3.mockwebserver.MockResponse; +import okhttp3.mockwebserver.RecordedRequest; +import okio.Buffer; +import org.apache.commons.io.FileUtils; +import org.arvados.client.common.Headers; +import org.arvados.client.test.utils.ArvadosClientMockedWebServerTest; +import org.junit.Test; + +import java.io.File; +import java.util.Map; + +import static org.arvados.client.test.utils.ApiClientTestUtils.assertAuthorizationHeader; +import static org.assertj.core.api.Assertions.assertThat; + +public class KeepServerApiClientTest extends ArvadosClientMockedWebServerTest { + + private KeepServerApiClient client = new KeepServerApiClient(CONFIG); + + @Test + public void uploadFileToServer() throws Exception { + + // given + String blockLocator = "7df44272090cee6c0732382bba415ee9"; + String signedBlockLocator = blockLocator + "+70+A189a93acda6e1fba18a9dffd42b6591cbd36d55d@5a1c17b6"; + server.enqueue(new MockResponse().setBody(signedBlockLocator)); + + String url = server.url(blockLocator).toString(); + File body = new File("README.md"); + Map headers = Maps.newHashMap(); + headers.put(Headers.X_KEEP_DESIRED_REPLICAS, "2"); + + // when + String actual = client.upload(url, headers, body); + + // then + RecordedRequest request = server.takeRequest(); + assertAuthorizationHeader(request); + assertThat(request.getPath()).isEqualTo("/" + blockLocator); + + assertThat(actual).isEqualTo(signedBlockLocator); + } + + @Test + public void downloadFileFromServer() throws Exception { + File data = new File("README.md"); + byte[] fileBytes = FileUtils.readFileToByteArray(data); + server.enqueue(new MockResponse().setBody(new Buffer().write(fileBytes))); + + String blockLocator = "7df44272090cee6c0732382bba415ee9"; + String signedBlockLocator = blockLocator + "+70+A189a93acda6e1fba18a9dffd42b6591cbd36d55d@5a1c17b6"; + + String url = server.url(signedBlockLocator).toString(); + + byte[] actual = client.download(url); + RecordedRequest request = server.takeRequest(); + assertThat(request.getPath()).isEqualTo("/" + signedBlockLocator); + assertThat(actual).isEqualTo(fileBytes); + + } +} diff --git a/src/test/java/org/arvados/client/api/client/KeepServicesApiClientTest.java b/src/test/java/org/arvados/client/api/client/KeepServicesApiClientTest.java new file mode 100644 index 0000000000..015f8328d8 --- /dev/null +++ b/src/test/java/org/arvados/client/api/client/KeepServicesApiClientTest.java @@ -0,0 +1,78 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.client; + +import okhttp3.mockwebserver.RecordedRequest; +import org.arvados.client.api.model.KeepService; +import org.arvados.client.api.model.KeepServiceList; +import org.arvados.client.test.utils.ArvadosClientMockedWebServerTest; +import org.junit.Test; + +import static org.arvados.client.test.utils.ApiClientTestUtils.*; +import static org.assertj.core.api.Assertions.assertThat; + +public class KeepServicesApiClientTest extends ArvadosClientMockedWebServerTest { + + private static final String RESOURCE = "keep_services"; + + private KeepServicesApiClient client = new KeepServicesApiClient(CONFIG); + + @Test + public void listKeepServices() throws Exception { + + // given + server.enqueue(getResponse("keep-services-list")); + + // when + KeepServiceList actual = client.list(); + + // then + RecordedRequest request = server.takeRequest(); + assertAuthorizationHeader(request); + assertRequestPath(request, RESOURCE); + + assertThat(actual.getItemsAvailable()).isEqualTo(3); + + } + + @Test + public void listAccessibleKeepServices() throws Exception { + + // given + server.enqueue(getResponse("keep-services-accessible")); + + // when + KeepServiceList actual = client.accessible(); + + // then + RecordedRequest request = server.takeRequest(); + assertAuthorizationHeader(request); + assertRequestPath(request, RESOURCE + "/accessible"); + assertThat(actual.getItemsAvailable()).isEqualTo(2); + } + + @Test + public void getKeepService() throws Exception { + + // given + server.enqueue(getResponse("keep-services-get")); + + String uuid = "112ci-bi6l4-hv02fg8sbti8ykk"; + + // whenFs + KeepService actual = client.get(uuid); + + // then + RecordedRequest request = server.takeRequest(); + assertAuthorizationHeader(request); + assertRequestPath(request, RESOURCE + "/" + uuid); + assertThat(actual.getUuid()).isEqualTo(uuid); + assertThat(actual.getServiceType()).isEqualTo("disk"); + } + +} diff --git a/src/test/java/org/arvados/client/api/client/UsersApiClientTest.java b/src/test/java/org/arvados/client/api/client/UsersApiClientTest.java new file mode 100644 index 0000000000..40f7bac080 --- /dev/null +++ b/src/test/java/org/arvados/client/api/client/UsersApiClientTest.java @@ -0,0 +1,126 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.client; + +import okhttp3.mockwebserver.RecordedRequest; +import org.arvados.client.api.model.User; +import org.arvados.client.api.model.UserList; +import org.arvados.client.test.utils.RequestMethod; +import org.arvados.client.test.utils.ArvadosClientMockedWebServerTest; +import org.junit.Test; + +import static org.arvados.client.common.Characters.SLASH; +import static org.arvados.client.test.utils.ApiClientTestUtils.*; +import static org.assertj.core.api.Assertions.assertThat; + +public class UsersApiClientTest extends ArvadosClientMockedWebServerTest { + + private static final String RESOURCE = "users"; + private static final String USER_UUID = "ardev-tpzed-q6dvn7sby55up1b"; + + private UsersApiClient client = new UsersApiClient(CONFIG); + + @Test + public void listUsers() throws Exception { + + // given + server.enqueue(getResponse("users-list")); + + // when + UserList actual = client.list(); + + // then + RecordedRequest request = server.takeRequest(); + assertAuthorizationHeader(request); + assertRequestPath(request, RESOURCE); + assertRequestMethod(request, RequestMethod.GET); + assertThat(actual.getItemsAvailable()).isEqualTo(13); + } + + @Test + public void getUser() throws Exception { + + // given + server.enqueue(getResponse("users-get")); + + // when + User actual = client.get(USER_UUID); + + // then + RecordedRequest request = server.takeRequest(); + assertAuthorizationHeader(request); + assertRequestPath(request, RESOURCE + SLASH + USER_UUID); + assertRequestMethod(request, RequestMethod.GET); + assertThat(actual.getUuid()).isEqualTo(USER_UUID); + } + + @Test + public void getCurrentUser() throws Exception { + + // given + server.enqueue(getResponse("users-get")); + + // when + User actual = client.current(); + + // then + RecordedRequest request = server.takeRequest(); + assertAuthorizationHeader(request); + assertRequestPath(request, RESOURCE + SLASH + "current"); + assertRequestMethod(request, RequestMethod.GET); + assertThat(actual.getUuid()).isEqualTo(USER_UUID); + } + + @Test + public void getSystemUser() throws Exception { + + // given + server.enqueue(getResponse("users-system")); + + // when + User actual = client.system(); + + // then + RecordedRequest request = server.takeRequest(); + assertAuthorizationHeader(request); + assertRequestPath(request, RESOURCE + SLASH + "system"); + assertRequestMethod(request, RequestMethod.GET); + assertThat(actual.getUuid()).isEqualTo("ardev-tpzed-000000000000000"); + } + + @Test + public void createUser() throws Exception { + + // given + server.enqueue(getResponse("users-create")); + + String firstName = "John"; + String lastName = "Wayne"; + String fullName = String.format("%s %s", firstName, lastName); + String username = String.format("%s%s", firstName, lastName).toLowerCase(); + + User user = new User(); + user.setFirstName(firstName); + user.setLastName(lastName); + user.setFullName(fullName); + user.setUsername(username); + + // when + User actual = client.create(user); + + // then + RecordedRequest request = server.takeRequest(); + assertAuthorizationHeader(request); + assertRequestPath(request, RESOURCE); + assertRequestMethod(request, RequestMethod.POST); + assertThat(actual.getFirstName()).isEqualTo(firstName); + assertThat(actual.getLastName()).isEqualTo(lastName); + assertThat(actual.getFullName()).isEqualTo(fullName); + assertThat(actual.getUsername()).isEqualTo(username); + } +} diff --git a/src/test/java/org/arvados/client/api/client/factory/OkHttpClientFactoryTest.java b/src/test/java/org/arvados/client/api/client/factory/OkHttpClientFactoryTest.java new file mode 100644 index 0000000000..f7e1813294 --- /dev/null +++ b/src/test/java/org/arvados/client/api/client/factory/OkHttpClientFactoryTest.java @@ -0,0 +1,96 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.api.client.factory; + +import okhttp3.OkHttpClient; +import okhttp3.Request; +import okhttp3.Response; +import okhttp3.mockwebserver.MockResponse; +import org.arvados.client.test.utils.ArvadosClientMockedWebServerTest; +import org.junit.Assert; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.junit.MockitoJUnitRunner; + +import javax.net.ssl.KeyManagerFactory; +import javax.net.ssl.SSLContext; +import javax.net.ssl.SSLSocketFactory; +import javax.net.ssl.TrustManagerFactory; +import java.io.FileInputStream; +import java.security.KeyStore; + + +@RunWith(MockitoJUnitRunner.class) +public class OkHttpClientFactoryTest extends ArvadosClientMockedWebServerTest { + + @Test(expected = javax.net.ssl.SSLHandshakeException.class) + public void secureOkHttpClientIsCreated() throws Exception { + + // given + OkHttpClientFactory factory = OkHttpClientFactory.builder().build(); + // * configure HTTPS server + SSLSocketFactory sf = getSSLSocketFactoryWithSelfSignedCertificate(); + server.useHttps(sf, false); + server.enqueue(new MockResponse().setBody("OK")); + // * prepare client HTTP request + Request request = new Request.Builder() + .url("https://localhost:9000/") + .build(); + + // when - then (SSL certificate is verified) + OkHttpClient actual = factory.create(false); + Response response = actual.newCall(request).execute(); + } + + @Test + public void insecureOkHttpClientIsCreated() throws Exception { + // given + OkHttpClientFactory factory = OkHttpClientFactory.builder().build(); + // * configure HTTPS server + SSLSocketFactory sf = getSSLSocketFactoryWithSelfSignedCertificate(); + server.useHttps(sf, false); + server.enqueue(new MockResponse().setBody("OK")); + // * prepare client HTTP request + Request request = new Request.Builder() + .url("https://localhost:9000/") + .build(); + + // when (SSL certificate is not verified) + OkHttpClient actual = factory.create(true); + Response response = actual.newCall(request).execute(); + + // then + Assert.assertEquals(response.body().string(),"OK"); + } + + + /* + This ugly boilerplate is needed to enable self signed certificate. + + It requires selfsigned.keystore.jks file. It was generated with: + keytool -genkey -v -keystore mystore.keystore.jks -alias alias_name -keyalg RSA -keysize 2048 -validity 10000 + */ + public SSLSocketFactory getSSLSocketFactoryWithSelfSignedCertificate() throws Exception { + + FileInputStream stream = new FileInputStream("src/test/resources/selfsigned.keystore.jks"); + char[] serverKeyStorePassword = "123456".toCharArray(); + KeyStore serverKeyStore = KeyStore.getInstance(KeyStore.getDefaultType()); + serverKeyStore.load(stream, serverKeyStorePassword); + + String kmfAlgorithm = KeyManagerFactory.getDefaultAlgorithm(); + KeyManagerFactory kmf = KeyManagerFactory.getInstance(kmfAlgorithm); + kmf.init(serverKeyStore, serverKeyStorePassword); + + TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(kmfAlgorithm); + trustManagerFactory.init(serverKeyStore); + + SSLContext sslContext = SSLContext.getInstance("SSL"); + sslContext.init(kmf.getKeyManagers(), trustManagerFactory.getTrustManagers(), null); + return sslContext.getSocketFactory(); + } +} diff --git a/src/test/java/org/arvados/client/facade/ArvadosFacadeIntegrationTest.java b/src/test/java/org/arvados/client/facade/ArvadosFacadeIntegrationTest.java new file mode 100644 index 0000000000..07269f7e7d --- /dev/null +++ b/src/test/java/org/arvados/client/facade/ArvadosFacadeIntegrationTest.java @@ -0,0 +1,258 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.facade; + +import org.apache.commons.io.FileUtils; +import org.arvados.client.api.model.Collection; +import org.arvados.client.common.Characters; +import org.arvados.client.config.ExternalConfigProvider; +import org.arvados.client.junit.categories.IntegrationTests; +import org.arvados.client.logic.collection.FileToken; +import org.arvados.client.test.utils.ArvadosClientIntegrationTest; +import org.arvados.client.test.utils.FileTestUtils; +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; +import org.junit.experimental.categories.Category; + +import java.io.File; +import java.util.Collections; +import java.util.List; +import java.util.UUID; + +import static org.arvados.client.test.utils.FileTestUtils.FILE_DOWNLOAD_TEST_DIR; +import static org.arvados.client.test.utils.FileTestUtils.FILE_SPLIT_TEST_DIR; +import static org.arvados.client.test.utils.FileTestUtils.TEST_FILE; +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +@Category(IntegrationTests.class) +public class ArvadosFacadeIntegrationTest extends ArvadosClientIntegrationTest { + + + private static final String COLLECTION_NAME = "Test collection " + UUID.randomUUID().toString(); + private String collectionUuid; + + @Before + public void setUp() throws Exception { + FileTestUtils.createDirectory(FILE_SPLIT_TEST_DIR); + FileTestUtils.createDirectory(FILE_DOWNLOAD_TEST_DIR); + } + + @Test + public void uploadOfFileIsPerformedSuccessfully() throws Exception { + // given + File file = FileTestUtils.generateFile(TEST_FILE, FileTestUtils.ONE_FOURTH_GB / 200); + + // when + Collection actual = FACADE.upload(Collections.singletonList(file), COLLECTION_NAME, PROJECT_UUID); + collectionUuid = actual.getUuid(); + + // then + assertThat(actual.getName()).contains("Test collection"); + assertThat(actual.getManifestText()).contains(file.length() + Characters.COLON + file.getName()); + } + + @Test + public void uploadOfFilesIsPerformedSuccessfully() throws Exception { + // given + List files = FileTestUtils.generatePredefinedFiles(); + files.addAll(FileTestUtils.generatePredefinedFiles()); + + // when + Collection actual = FACADE.upload(files, COLLECTION_NAME, PROJECT_UUID); + collectionUuid = actual.getUuid(); + + // then + assertThat(actual.getName()).contains("Test collection"); + files.forEach(f -> assertThat(actual.getManifestText()).contains(f.length() + Characters.COLON + f.getName().replace(" ", Characters.SPACE))); + } + + @Test + public void uploadToExistingCollectionIsPerformedSuccessfully() throws Exception { + // given + File file = FileTestUtils.generateFile(TEST_FILE, FileTestUtils.ONE_EIGTH_GB / 500); + Collection existing = createTestCollection(); + + // when + Collection actual = FACADE.uploadToExistingCollection(Collections.singletonList(file), collectionUuid); + + // then + assertEquals(collectionUuid, actual.getUuid()); + assertThat(actual.getManifestText()).contains(file.length() + Characters.COLON + file.getName()); + } + + @Test + public void uploadWithExternalConfigProviderWorksProperly() throws Exception { + //given + ArvadosFacade facade = new ArvadosFacade(buildExternalConfig()); + File file = FileTestUtils.generateFile(TEST_FILE, FileTestUtils.ONE_FOURTH_GB / 200); + + //when + Collection actual = facade.upload(Collections.singletonList(file), COLLECTION_NAME, PROJECT_UUID); + collectionUuid = actual.getUuid(); + + //then + assertThat(actual.getName()).contains("Test collection"); + assertThat(actual.getManifestText()).contains(file.length() + Characters.COLON + file.getName()); + } + + @Test + public void creationOfEmptyCollectionPerformedSuccesfully() { + // given + String collectionName = "Empty collection " + UUID.randomUUID().toString(); + + // when + Collection actual = FACADE.createEmptyCollection(collectionName, PROJECT_UUID); + collectionUuid = actual.getUuid(); + + // then + assertEquals(collectionName, actual.getName()); + assertEquals(PROJECT_UUID, actual.getOwnerUuid()); + } + + @Test + public void fileTokensAreListedFromCollection() throws Exception { + //given + List files = uploadTestFiles(); + + //when + List actual = FACADE.listFileInfoFromCollection(collectionUuid); + + //then + assertEquals(files.size(), actual.size()); + for (int i = 0; i < files.size(); i++) { + assertEquals(files.get(i).length(), actual.get(i).getFileSize()); + } + } + + @Test + public void downloadOfFilesPerformedSuccessfully() throws Exception { + //given + List files = uploadTestFiles(); + File destination = new File(FILE_DOWNLOAD_TEST_DIR + Characters.SLASH + collectionUuid); + + //when + List actual = FACADE.downloadCollectionFiles(collectionUuid, FILE_DOWNLOAD_TEST_DIR, false); + + //then + assertEquals(files.size(), actual.size()); + assertTrue(destination.exists()); + assertThat(actual).allMatch(File::exists); + for (int i = 0; i < files.size(); i++) { + assertEquals(files.get(i).length(), actual.get(i).length()); + } + } + + @Test + public void downloadOfFilesPerformedSuccessfullyUsingKeepWeb() throws Exception { + //given + List files = uploadTestFiles(); + File destination = new File(FILE_DOWNLOAD_TEST_DIR + Characters.SLASH + collectionUuid); + + //when + List actual = FACADE.downloadCollectionFiles(collectionUuid, FILE_DOWNLOAD_TEST_DIR, true); + + //then + assertEquals(files.size(), actual.size()); + assertTrue(destination.exists()); + assertThat(actual).allMatch(File::exists); + for (int i = 0; i < files.size(); i++) { + assertEquals(files.get(i).length(), actual.get(i).length()); + } + } + + @Test + public void singleFileIsDownloadedSuccessfullyUsingKeepWeb() throws Exception { + //given + File file = uploadSingleTestFile(false); + + //when + File actual = FACADE.downloadFile(file.getName(), collectionUuid, FILE_DOWNLOAD_TEST_DIR); + + //then + assertThat(actual).exists(); + assertThat(actual.length()).isEqualTo(file.length()); + } + + @Test + public void downloadOfOneFileSplittedToMultipleLocatorsPerformedSuccesfully() throws Exception { + //given + File file = uploadSingleTestFile(true); + + List actual = FACADE.downloadCollectionFiles(collectionUuid, FILE_DOWNLOAD_TEST_DIR, false); + + Assert.assertEquals(1, actual.size()); + assertThat(actual.get(0).length()).isEqualTo(file.length()); + } + + @Test + public void downloadWithExternalConfigProviderWorksProperly() throws Exception { + //given + ArvadosFacade facade = new ArvadosFacade(buildExternalConfig()); + List files = uploadTestFiles(); + //when + List actual = facade.downloadCollectionFiles(collectionUuid, FILE_DOWNLOAD_TEST_DIR, false); + + //then + assertEquals(files.size(), actual.size()); + assertThat(actual).allMatch(File::exists); + for (int i = 0; i < files.size(); i++) { + assertEquals(files.get(i).length(), actual.get(i).length()); + } + } + + private ExternalConfigProvider buildExternalConfig() { + return ExternalConfigProvider + .builder() + .apiHostInsecure(CONFIG.isApiHostInsecure()) + .keepWebHost(CONFIG.getKeepWebHost()) + .keepWebPort(CONFIG.getKeepWebPort()) + .apiHost(CONFIG.getApiHost()) + .apiPort(CONFIG.getApiPort()) + .apiToken(CONFIG.getApiToken()) + .apiProtocol(CONFIG.getApiProtocol()) + .fileSplitSize(CONFIG.getFileSplitSize()) + .fileSplitDirectory(CONFIG.getFileSplitDirectory()) + .numberOfCopies(CONFIG.getNumberOfCopies()) + .numberOfRetries(CONFIG.getNumberOfRetries()) + .build(); + } + + private Collection createTestCollection() { + Collection collection = FACADE.createEmptyCollection(COLLECTION_NAME, PROJECT_UUID); + collectionUuid = collection.getUuid(); + return collection; + } + + private List uploadTestFiles() throws Exception{ + createTestCollection(); + List files = FileTestUtils.generatePredefinedFiles(); + FACADE.uploadToExistingCollection(files, collectionUuid); + return files; + } + + private File uploadSingleTestFile(boolean bigFile) throws Exception{ + createTestCollection(); + Long fileSize = bigFile ? FileUtils.ONE_MB * 70 : FileTestUtils.ONE_EIGTH_GB / 100; + File file = FileTestUtils.generateFile(TEST_FILE, fileSize); + FACADE.uploadToExistingCollection(Collections.singletonList(file), collectionUuid); + return file; + } + + @After + public void tearDown() throws Exception { + FileTestUtils.cleanDirectory(FILE_SPLIT_TEST_DIR); + FileTestUtils.cleanDirectory(FILE_DOWNLOAD_TEST_DIR); + + if(collectionUuid != null) + FACADE.deleteCollection(collectionUuid); + } +} diff --git a/src/test/java/org/arvados/client/facade/ArvadosFacadeTest.java b/src/test/java/org/arvados/client/facade/ArvadosFacadeTest.java new file mode 100644 index 0000000000..a025011d79 --- /dev/null +++ b/src/test/java/org/arvados/client/facade/ArvadosFacadeTest.java @@ -0,0 +1,198 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.facade; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.databind.ObjectWriter; +import okhttp3.mockwebserver.MockResponse; +import okio.Buffer; +import org.apache.commons.io.FileUtils; +import org.arvados.client.api.model.Collection; +import org.arvados.client.api.model.KeepService; +import org.arvados.client.api.model.KeepServiceList; +import org.arvados.client.common.Characters; +import org.arvados.client.test.utils.ArvadosClientMockedWebServerTest; +import org.arvados.client.test.utils.FileTestUtils; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; + +import java.io.File; +import java.nio.charset.Charset; +import java.nio.file.Files; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.stream.Collectors; + +import static org.arvados.client.test.utils.ApiClientTestUtils.getResponse; +import static org.arvados.client.test.utils.FileTestUtils.*; +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +public class ArvadosFacadeTest extends ArvadosClientMockedWebServerTest { + + ArvadosFacade facade = new ArvadosFacade(CONFIG); + + @Before + public void setUp() throws Exception { + FileTestUtils.createDirectory(FILE_SPLIT_TEST_DIR); + FileTestUtils.createDirectory(FILE_DOWNLOAD_TEST_DIR); + } + + @Test + public void uploadIsPerformedSuccessfullyUsingDiskOnlyKeepServices() throws Exception { + + // given + String keepServicesAccessible = setMockedServerPortToKeepServices("keep-services-accessible-disk-only"); + server.enqueue(new MockResponse().setBody(keepServicesAccessible)); + + String blockLocator = "7df44272090cee6c0732382bba415ee9"; + String signedBlockLocator = blockLocator + "+70+A189a93acda6e1fba18a9dffd42b6591cbd36d55d@5a1c17b6"; + for (int i = 0; i < 8; i++) { + server.enqueue(new MockResponse().setBody(signedBlockLocator)); + } + server.enqueue(getResponse("users-get")); + server.enqueue(getResponse("collections-create-manifest")); + + FileTestUtils.generateFile(TEST_FILE, FileTestUtils.ONE_FOURTH_GB); + + // when + Collection actual = facade.upload(Arrays.asList(new File(TEST_FILE)), "Super Collection", null); + + // then + assertThat(actual.getName()).contains("Super Collection"); + } + + @Test + public void uploadIsPerformedSuccessfully() throws Exception { + + // given + String keepServicesAccessible = setMockedServerPortToKeepServices("keep-services-accessible"); + server.enqueue(new MockResponse().setBody(keepServicesAccessible)); + + String blockLocator = "7df44272090cee6c0732382bba415ee9"; + String signedBlockLocator = blockLocator + "+70+A189a93acda6e1fba18a9dffd42b6591cbd36d55d@5a1c17b6"; + for (int i = 0; i < 4; i++) { + server.enqueue(new MockResponse().setBody(signedBlockLocator)); + } + server.enqueue(getResponse("users-get")); + server.enqueue(getResponse("collections-create-manifest")); + + FileTestUtils.generateFile(TEST_FILE, FileTestUtils.ONE_FOURTH_GB); + + // when + Collection actual = facade.upload(Arrays.asList(new File(TEST_FILE)), "Super Collection", null); + + // then + assertThat(actual.getName()).contains("Super Collection"); + } + + @Test + public void downloadOfWholeCollectionIsPerformedSuccessfully() throws Exception { + + //given + String collectionUuid = "ardev-4zz18-jk5vo4uo9u5vj52"; + server.enqueue(getResponse("collections-download-file")); + + String keepServicesAccessible = setMockedServerPortToKeepServices("keep-services-accessible"); + server.enqueue(new MockResponse().setBody(keepServicesAccessible)); + File collectionDestination = new File(FILE_DOWNLOAD_TEST_DIR + Characters.SLASH + collectionUuid); + + List files = generatePredefinedFiles(); + List fileData = new ArrayList<>(); + for (File f : files) { + fileData.add(Files.readAllBytes(f.toPath())); + } + byte[] filesDataChunk = fileData.stream().reduce(new byte[0], this::addAll); + + server.enqueue(new MockResponse().setBody(new Buffer().write(filesDataChunk))); + + //when + List downloadedFiles = facade.downloadCollectionFiles(collectionUuid, FILE_DOWNLOAD_TEST_DIR, false); + + //then + assertEquals(3, downloadedFiles.size()); + assertTrue(collectionDestination.exists()); + assertThat(downloadedFiles).allMatch(File::exists); + assertEquals(files.stream().map(File::getName).collect(Collectors.toList()), downloadedFiles.stream().map(File::getName).collect(Collectors.toList())); + assertEquals(files.stream().map(File::length).collect(Collectors.toList()), downloadedFiles.stream().map(File::length).collect(Collectors.toList())); + } + + @Test + public void downloadOfWholeCollectionUsingKeepWebPerformedSuccessfully() throws Exception { + + //given + String collectionUuid = "ardev-4zz18-jk5vo4uo9u5vj52"; + server.enqueue(getResponse("collections-download-file")); + + List files = generatePredefinedFiles(); + for (File f : files) { + server.enqueue(new MockResponse().setBody(new Buffer().write(FileUtils.readFileToByteArray(f)))); + } + + //when + List downloadedFiles = facade.downloadCollectionFiles(collectionUuid, FILE_DOWNLOAD_TEST_DIR, true); + + //then + assertEquals(3, downloadedFiles.size()); + assertThat(downloadedFiles).allMatch(File::exists); + assertEquals(files.stream().map(File::getName).collect(Collectors.toList()), downloadedFiles.stream().map(File::getName).collect(Collectors.toList())); + assertTrue(downloadedFiles.stream().map(File::length).collect(Collectors.toList()).containsAll(files.stream().map(File::length).collect(Collectors.toList()))); + } + + @Test + public void downloadOfSingleFilePerformedSuccessfully() throws Exception { + + //given + String collectionUuid = "ardev-4zz18-jk5vo4uo9u5vj52"; + server.enqueue(getResponse("collections-download-file")); + + File file = generatePredefinedFiles().get(0); + byte[] fileData = FileUtils.readFileToByteArray(file); + server.enqueue(new MockResponse().setBody(new Buffer().write(fileData))); + + //when + File downloadedFile = facade.downloadFile(file.getName(), collectionUuid, FILE_DOWNLOAD_TEST_DIR); + + //then + assertTrue(downloadedFile.exists()); + assertEquals(file.getName(), downloadedFile.getName()); + assertEquals(file.length(), downloadedFile.length()); + } + + private String setMockedServerPortToKeepServices(String jsonPath) throws Exception { + + ObjectMapper mapper = new ObjectMapper().findAndRegisterModules(); + String filePath = String.format("src/test/resources/org/arvados/client/api/client/%s.json", jsonPath); + File jsonFile = new File(filePath); + String json = FileUtils.readFileToString(jsonFile, Charset.defaultCharset()); + KeepServiceList keepServiceList = mapper.readValue(json, KeepServiceList.class); + List items = keepServiceList.getItems(); + for (KeepService keepService : items) { + keepService.setServicePort(server.getPort()); + } + ObjectWriter writer = mapper.writer().withDefaultPrettyPrinter(); + return writer.writeValueAsString(keepServiceList); + } + + //Method to copy multiple byte[] arrays into one byte[] array + private byte[] addAll(byte[] array1, byte[] array2) { + byte[] joinedArray = new byte[array1.length + array2.length]; + System.arraycopy(array1, 0, joinedArray, 0, array1.length); + System.arraycopy(array2, 0, joinedArray, array1.length, array2.length); + return joinedArray; + } + + @After + public void tearDown() throws Exception { + FileTestUtils.cleanDirectory(FILE_SPLIT_TEST_DIR); + FileTestUtils.cleanDirectory(FILE_DOWNLOAD_TEST_DIR); + } +} diff --git a/src/test/java/org/arvados/client/junit/categories/IntegrationTests.java b/src/test/java/org/arvados/client/junit/categories/IntegrationTests.java new file mode 100644 index 0000000000..6a0e78d661 --- /dev/null +++ b/src/test/java/org/arvados/client/junit/categories/IntegrationTests.java @@ -0,0 +1,10 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.junit.categories; + +public interface IntegrationTests {} diff --git a/src/test/java/org/arvados/client/logic/collection/FileTokenTest.java b/src/test/java/org/arvados/client/logic/collection/FileTokenTest.java new file mode 100644 index 0000000000..13939852cb --- /dev/null +++ b/src/test/java/org/arvados/client/logic/collection/FileTokenTest.java @@ -0,0 +1,42 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.logic.collection; + +import org.arvados.client.common.Characters; +import org.junit.Assert; +import org.junit.Test; + +public class FileTokenTest { + + public static final String FILE_TOKEN_INFO = "0:1024:test-file1"; + public static final int FILE_POSITION = 0; + public static final int FILE_LENGTH = 1024; + public static final String FILE_NAME = "test-file1"; + public static final String FILE_PATH = "c" + Characters.SLASH; + + private static FileToken fileToken = new FileToken(FILE_TOKEN_INFO); + private static FileToken fileTokenWithPath = new FileToken(FILE_TOKEN_INFO, FILE_PATH); + + @Test + public void tokenInfoIsDividedCorrectly(){ + Assert.assertEquals(FILE_NAME, fileToken.getFileName()); + Assert.assertEquals(FILE_POSITION, fileToken.getFilePosition()); + Assert.assertEquals(FILE_LENGTH, fileToken.getFileSize()); + } + + @Test + public void toStringReturnsOriginalFileTokenInfo(){ + Assert.assertEquals(FILE_TOKEN_INFO, fileToken.toString()); + } + + @Test + public void fullPathIsReturnedProperly(){ + Assert.assertEquals(FILE_NAME, fileToken.getFullPath()); + Assert.assertEquals(FILE_PATH + FILE_NAME, fileTokenWithPath.getFullPath()); + } +} diff --git a/src/test/java/org/arvados/client/logic/collection/ManifestDecoderTest.java b/src/test/java/org/arvados/client/logic/collection/ManifestDecoderTest.java new file mode 100644 index 0000000000..c9464e03b6 --- /dev/null +++ b/src/test/java/org/arvados/client/logic/collection/ManifestDecoderTest.java @@ -0,0 +1,108 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.logic.collection; + +import org.arvados.client.exception.ArvadosClientException; +import org.junit.Assert; +import org.junit.Test; + +import java.util.List; + +import static junit.framework.TestCase.fail; + +public class ManifestDecoderTest { + + private ManifestDecoder manifestDecoder = new ManifestDecoder(); + + private static final String ONE_LINE_MANIFEST_TEXT = ". " + + "eff999f3b5158331eb44a9a93e3b36e1+67108864+Aad3839bea88bce22cbfe71cf4943de7dab3ea52a@5826180f " + + "db141bfd11f7da60dce9e5ee85a988b8+34038725+Ae8f48913fed782cbe463e0499ab37697ee06a2f8@5826180f " + + "0:101147589:rna.SRR948778.bam" + + "\\n"; + + private static final String MULTIPLE_LINES_MANIFEST_TEXT = ". " + + "930625b054ce894ac40596c3f5a0d947+33 " + + "0:0:a 0:0:b 0:33:output.txt\n" + + "./c d41d8cd98f00b204e9800998ecf8427e+0 0:0:d"; + + private static final String MANIFEST_TEXT_WITH_INVALID_FIRST_PATH_COMPONENT = "a" + ONE_LINE_MANIFEST_TEXT; + + + @Test + public void allLocatorsAndFileTokensAreExtractedFromSimpleManifest() { + + List actual = manifestDecoder.decode(ONE_LINE_MANIFEST_TEXT); + + // one manifest stream + Assert.assertEquals(1, actual.size()); + + ManifestStream manifest = actual.get(0); + // two locators + Assert.assertEquals(2, manifest.getKeepLocators().size()); + // one file token + Assert.assertEquals(1, manifest.getFileTokens().size()); + + } + + @Test + public void allLocatorsAndFileTokensAreExtractedFromComplexManifest() { + + List actual = manifestDecoder.decode(MULTIPLE_LINES_MANIFEST_TEXT); + + // two manifest streams + Assert.assertEquals(2, actual.size()); + + // first stream - 1 locator and 3 file tokens + ManifestStream firstManifestStream = actual.get(0); + Assert.assertEquals(1, firstManifestStream.getKeepLocators().size()); + Assert.assertEquals(3, firstManifestStream.getFileTokens().size()); + + // second stream - 1 locator and 1 file token + ManifestStream secondManifestStream = actual.get(1); + Assert.assertEquals(1, secondManifestStream.getKeepLocators().size()); + Assert.assertEquals(1, secondManifestStream.getFileTokens().size()); + } + + @Test + public void manifestTextWithInvalidStreamNameThrowsException() { + + try { + List actual = manifestDecoder.decode(MANIFEST_TEXT_WITH_INVALID_FIRST_PATH_COMPONENT); + fail(); + } catch (ArvadosClientException e) { + Assert.assertEquals("Invalid first path component (expecting \".\")", e.getMessage()); + } + + } + + @Test + public void emptyManifestTextThrowsException() { + String emptyManifestText = null; + + try { + List actual = manifestDecoder.decode(emptyManifestText); + fail(); + } catch (ArvadosClientException e) { + Assert.assertEquals("Manifest text cannot be empty.", e.getMessage()); + } + + emptyManifestText = ""; + try { + List actual = manifestDecoder.decode(emptyManifestText); + fail(); + } catch (ArvadosClientException e) { + Assert.assertEquals("Manifest text cannot be empty.", e.getMessage()); + } + + } + + + + + +} diff --git a/src/test/java/org/arvados/client/logic/collection/ManifestFactoryTest.java b/src/test/java/org/arvados/client/logic/collection/ManifestFactoryTest.java new file mode 100644 index 0000000000..06ed07d8ed --- /dev/null +++ b/src/test/java/org/arvados/client/logic/collection/ManifestFactoryTest.java @@ -0,0 +1,38 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.logic.collection; + +import org.arvados.client.test.utils.FileTestUtils; +import org.assertj.core.util.Lists; +import org.junit.Test; + +import java.io.File; +import java.util.List; + +import static org.assertj.core.api.Assertions.assertThat; + +public class ManifestFactoryTest { + + @Test + public void manifestIsCreatedAsExpected() throws Exception { + + // given + List files = FileTestUtils.generatePredefinedFiles(); + List locators = Lists.newArrayList("a", "b", "c"); + ManifestFactory factory = ManifestFactory.builder() + .files(files) + .locators(locators) + .build(); + + // when + String actual = factory.create(); + + // then + assertThat(actual).isEqualTo(". a b c 0:1024:test-file1 1024:20480:test-file2 21504:1048576:test-file\\0403\n"); + } +} diff --git a/src/test/java/org/arvados/client/logic/collection/ManifestStreamTest.java b/src/test/java/org/arvados/client/logic/collection/ManifestStreamTest.java new file mode 100644 index 0000000000..bc36889f71 --- /dev/null +++ b/src/test/java/org/arvados/client/logic/collection/ManifestStreamTest.java @@ -0,0 +1,27 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.logic.collection; + + +import org.junit.Assert; +import org.junit.Test; + +import java.util.List; + +public class ManifestStreamTest { + + private ManifestDecoder manifestDecoder = new ManifestDecoder(); + + @Test + public void toStringReturnsProperlyConnectedManifestStream() throws Exception{ + String encodedManifest = ". eff999f3b5158331eb44a9a93e3b36e1+67108864 db141bfd11f7da60dce9e5ee85a988b8+34038725 0:101147589:rna.SRR948778.bam\\n\""; + List manifestStreams = manifestDecoder.decode(encodedManifest); + Assert.assertEquals(encodedManifest, manifestStreams.get(0).toString()); + + } +} diff --git a/src/test/java/org/arvados/client/logic/keep/FileDownloaderTest.java b/src/test/java/org/arvados/client/logic/keep/FileDownloaderTest.java new file mode 100644 index 0000000000..0fb1f0206c --- /dev/null +++ b/src/test/java/org/arvados/client/logic/keep/FileDownloaderTest.java @@ -0,0 +1,145 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.logic.keep; + +import com.fasterxml.jackson.databind.ObjectMapper; +import org.arvados.client.api.client.CollectionsApiClient; +import org.arvados.client.api.client.KeepWebApiClient; +import org.arvados.client.api.model.Collection; +import org.arvados.client.common.Characters; +import org.arvados.client.logic.collection.FileToken; +import org.arvados.client.logic.collection.ManifestDecoder; +import org.arvados.client.logic.collection.ManifestStream; +import org.arvados.client.test.utils.FileTestUtils; +import org.arvados.client.utils.FileMerge; +import org.apache.commons.io.FileUtils; +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.InjectMocks; +import org.mockito.Mock; +import org.mockito.junit.MockitoJUnitRunner; + +import java.io.File; +import java.io.IOException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.UUID; + +import static org.arvados.client.test.utils.FileTestUtils.*; +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.Mockito.when; + +@RunWith(MockitoJUnitRunner.class) +public class FileDownloaderTest { + + static final ObjectMapper MAPPER = new ObjectMapper().findAndRegisterModules(); + private Collection collectionToDownload; + private ManifestStream manifestStream; + + @Mock + private CollectionsApiClient collectionsApiClient; + @Mock + private KeepClient keepClient; + @Mock + private KeepWebApiClient keepWebApiClient; + @Mock + private ManifestDecoder manifestDecoder; + @InjectMocks + private FileDownloader fileDownloader; + + @Before + public void setUp() throws Exception { + FileTestUtils.createDirectory(FILE_SPLIT_TEST_DIR); + FileTestUtils.createDirectory(FILE_DOWNLOAD_TEST_DIR); + + collectionToDownload = prepareCollection(); + manifestStream = prepareManifestStream(); + } + + @Test + public void downloadingAllFilesFromCollectionWorksProperly() throws Exception { + // given + List files = generatePredefinedFiles(); + byte[] dataChunk = prepareDataChunk(files); + + //having + when(collectionsApiClient.get(collectionToDownload.getUuid())).thenReturn(collectionToDownload); + when(manifestDecoder.decode(collectionToDownload.getManifestText())).thenReturn(Arrays.asList(manifestStream)); + when(keepClient.getDataChunk(manifestStream.getKeepLocators().get(0))).thenReturn(dataChunk); + + //when + List downloadedFiles = fileDownloader.downloadFilesFromCollection(collectionToDownload.getUuid(), FILE_DOWNLOAD_TEST_DIR); + + //then + Assert.assertEquals(3, downloadedFiles.size()); // 3 files downloaded + + File collectionDir = new File(FILE_DOWNLOAD_TEST_DIR + Characters.SLASH + collectionToDownload.getUuid()); + Assert.assertTrue(collectionDir.exists()); // collection directory created + + // 3 files correctly saved + assertThat(downloadedFiles).allMatch(File::exists); + + for(int i = 0; i < downloadedFiles.size(); i ++) { + File downloaded = new File(collectionDir + Characters.SLASH + files.get(i).getName()); + Assert.assertArrayEquals(FileUtils.readFileToByteArray(downloaded), FileUtils.readFileToByteArray(files.get(i))); + } + } + + @Test + public void downloadingSingleFileFromKeepWebWorksCorrectly() throws Exception{ + //given + File file = generatePredefinedFiles().get(0); + + //having + when(collectionsApiClient.get(collectionToDownload.getUuid())).thenReturn(collectionToDownload); + when(manifestDecoder.decode(collectionToDownload.getManifestText())).thenReturn(Arrays.asList(manifestStream)); + when(keepWebApiClient.download(collectionToDownload.getUuid(), file.getName())).thenReturn(FileUtils.readFileToByteArray(file)); + + //when + File downloadedFile = fileDownloader.downloadSingleFileUsingKeepWeb(file.getName(), collectionToDownload.getUuid(), FILE_DOWNLOAD_TEST_DIR); + + //then + Assert.assertTrue(downloadedFile.exists()); + Assert.assertEquals(file.getName(), downloadedFile.getName()); + Assert.assertArrayEquals(FileUtils.readFileToByteArray(downloadedFile), FileUtils.readFileToByteArray(file)); + } + + @After + public void tearDown() throws Exception { + FileTestUtils.cleanDirectory(FILE_SPLIT_TEST_DIR); + FileTestUtils.cleanDirectory(FILE_DOWNLOAD_TEST_DIR); + } + + private Collection prepareCollection() throws IOException { + // collection that will be returned by mocked collectionsApiClient + String filePath = "src/test/resources/org/arvados/client/api/client/collections-download-file.json"; + File jsonFile = new File(filePath); + return MAPPER.readValue(jsonFile, Collection.class); + } + + private ManifestStream prepareManifestStream() throws Exception { + // manifestStream that will be returned by mocked manifestDecoder + List fileTokens = new ArrayList<>(); + fileTokens.add(new FileToken("0:1024:test-file1")); + fileTokens.add(new FileToken("1024:20480:test-file2")); + fileTokens.add(new FileToken("21504:1048576:test-file\\0403")); + + KeepLocator keepLocator = new KeepLocator("163679d58edaadc28db769011728a72c+1070080+A3acf8c1fe582c265d2077702e4a7d74fcc03aba8@5aa4fdeb"); + return new ManifestStream(".", Arrays.asList(keepLocator), fileTokens); + } + + private byte[] prepareDataChunk(List files) throws IOException { + File combinedFile = new File(FILE_SPLIT_TEST_DIR + Characters.SLASH + UUID.randomUUID()); + FileMerge.merge(files, combinedFile); + return FileUtils.readFileToByteArray(combinedFile); + } +} diff --git a/src/test/java/org/arvados/client/logic/keep/KeepClientTest.java b/src/test/java/org/arvados/client/logic/keep/KeepClientTest.java new file mode 100644 index 0000000000..e4e7bf2720 --- /dev/null +++ b/src/test/java/org/arvados/client/logic/keep/KeepClientTest.java @@ -0,0 +1,118 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.logic.keep; + +import okhttp3.mockwebserver.MockResponse; +import okio.Buffer; +import org.apache.commons.io.FileUtils; +import org.arvados.client.config.FileConfigProvider; +import org.arvados.client.config.ConfigProvider; +import org.arvados.client.exception.ArvadosClientException; +import org.arvados.client.test.utils.ArvadosClientMockedWebServerTest; +import org.junit.Assert; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.InjectMocks; +import org.mockito.Mock; +import org.mockito.junit.MockitoJUnitRunner; + +import java.io.File; + +import static junit.framework.TestCase.fail; +import static org.arvados.client.test.utils.ApiClientTestUtils.getResponse; +import static org.assertj.core.api.Assertions.assertThat; + +@RunWith(MockitoJUnitRunner.class) +public class KeepClientTest extends ArvadosClientMockedWebServerTest { + + private ConfigProvider configProvider = new FileConfigProvider(); + private static final String TEST_FILE_PATH ="src/test/resources/org/arvados/client/api/client/keep-client-test-file.txt"; + + @InjectMocks + private KeepClient keepClient = new KeepClient(configProvider); + + @Mock + private KeepLocator keepLocator; + + @Test + public void uploadedFile() throws Exception { + // given + server.enqueue(getResponse("keep-services-accessible")); + server.enqueue(new MockResponse().setBody("0887c78c7d6c1a60ac0b3709a4302ee4")); + + // when + String actual = keepClient.put(new File(TEST_FILE_PATH), 1, 0); + + // then + assertThat(actual).isEqualTo("0887c78c7d6c1a60ac0b3709a4302ee4"); + } + + @Test + public void fileIsDownloaded() throws Exception { + //given + File data = new File(TEST_FILE_PATH); + byte[] fileBytes = FileUtils.readFileToByteArray(data); + + // when + server.enqueue(getResponse("keep-services-accessible")); + server.enqueue(new MockResponse().setBody(new Buffer().write(fileBytes))); + + byte[] actual = keepClient.getDataChunk(keepLocator); + + Assert.assertArrayEquals(fileBytes, actual); + } + + @Test + public void fileIsDownloadedWhenFirstServerDoesNotRespond() throws Exception { + // given + File data = new File(TEST_FILE_PATH); + byte[] fileBytes = FileUtils.readFileToByteArray(data); + server.enqueue(getResponse("keep-services-accessible")); // two servers accessible + server.enqueue(new MockResponse().setResponseCode(404)); // first one not responding + server.enqueue(new MockResponse().setBody(new Buffer().write(fileBytes))); // second one responding + + //when + byte[] actual = keepClient.getDataChunk(keepLocator); + + //then + Assert.assertArrayEquals(fileBytes, actual); + } + + @Test + public void exceptionIsThrownWhenNoServerResponds() throws Exception { + //given + File data = new File(TEST_FILE_PATH); + server.enqueue(getResponse("keep-services-accessible")); // two servers accessible + server.enqueue(new MockResponse().setResponseCode(404)); // first one not responding + server.enqueue(new MockResponse().setResponseCode(404)); // second one not responding + + try { + //when + keepClient.getDataChunk(keepLocator); + fail(); + } catch (ArvadosClientException e) { + //then + Assert.assertEquals("No server responding. Unable to download data chunk.", e.getMessage()); + } + } + + @Test + public void exceptionIsThrownWhenThereAreNoServersAccessible() throws Exception { + //given + server.enqueue(getResponse("keep-services-not-accessible")); // no servers accessible + + try { + //when + keepClient.getDataChunk(keepLocator); + fail(); + } catch (ArvadosClientException e) { + //then + Assert.assertEquals("No gateway services available!", e.getMessage()); + } + } +} diff --git a/src/test/java/org/arvados/client/logic/keep/KeepLocatorTest.java b/src/test/java/org/arvados/client/logic/keep/KeepLocatorTest.java new file mode 100644 index 0000000000..c4c48da7c5 --- /dev/null +++ b/src/test/java/org/arvados/client/logic/keep/KeepLocatorTest.java @@ -0,0 +1,57 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.logic.keep; + +import org.junit.Test; + +import static org.assertj.core.api.Assertions.assertThat; + +public class KeepLocatorTest { + + private KeepLocator locator; + + @Test + public void md5sumIsExtracted() throws Exception { + + // given + locator = new KeepLocator("7df44272090cee6c0732382bba415ee9+70"); + + // when + String actual = locator.getMd5sum(); + + // then + assertThat(actual).isEqualTo("7df44272090cee6c0732382bba415ee9"); + } + + @Test + public void locatorIsStrippedWithMd5sumAndSize() throws Exception { + + // given + locator = new KeepLocator("7df44272090cee6c0732382bba415ee9+70"); + + // when + String actual = locator.stripped(); + + // then + assertThat(actual).isEqualTo("7df44272090cee6c0732382bba415ee9+70"); + } + + + @Test + public void locatorToStringProperlyShowing() throws Exception { + + // given + locator = new KeepLocator("7df44272090cee6c0732382bba415ee9+70+Ae8f48913fed782cbe463e0499ab37697ee06a2f8@5826180f"); + + // when + String actual = locator.toString(); + + // then + assertThat(actual).isEqualTo("7df44272090cee6c0732382bba415ee9+70+Ae8f48913fed782cbe463e0499ab37697ee06a2f8@5826180f"); + } +} diff --git a/src/test/java/org/arvados/client/test/utils/ApiClientTestUtils.java b/src/test/java/org/arvados/client/test/utils/ApiClientTestUtils.java new file mode 100644 index 0000000000..ac7dd02795 --- /dev/null +++ b/src/test/java/org/arvados/client/test/utils/ApiClientTestUtils.java @@ -0,0 +1,45 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.test.utils; + +import org.arvados.client.config.FileConfigProvider; +import okhttp3.mockwebserver.MockResponse; +import okhttp3.mockwebserver.RecordedRequest; +import org.apache.commons.io.FileUtils; + +import java.io.File; +import java.io.IOException; +import java.nio.charset.Charset; + +import static org.assertj.core.api.Assertions.assertThat; + +public final class ApiClientTestUtils { + + static final String BASE_URL = "/arvados/v1/"; + + private ApiClientTestUtils() {} + + public static MockResponse getResponse(String filename) throws IOException { + String filePath = String.format("src/test/resources/org/arvados/client/api/client/%s.json", filename); + File jsonFile = new File(filePath); + String json = FileUtils.readFileToString(jsonFile, Charset.defaultCharset()); + return new MockResponse().setBody(json); + } + + public static void assertAuthorizationHeader(RecordedRequest request) { + assertThat(request.getHeader("authorization")).isEqualTo("OAuth2 " + new FileConfigProvider().getApiToken()); + } + + public static void assertRequestPath(RecordedRequest request, String subPath) { + assertThat(request.getPath()).isEqualTo(BASE_URL + subPath); + } + + public static void assertRequestMethod(RecordedRequest request, RequestMethod requestMethod) { + assertThat(request.getMethod()).isEqualTo(requestMethod.name()); + } +} diff --git a/src/test/java/org/arvados/client/test/utils/ArvadosClientIntegrationTest.java b/src/test/java/org/arvados/client/test/utils/ArvadosClientIntegrationTest.java new file mode 100644 index 0000000000..59bd446934 --- /dev/null +++ b/src/test/java/org/arvados/client/test/utils/ArvadosClientIntegrationTest.java @@ -0,0 +1,28 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.test.utils; + +import org.arvados.client.config.FileConfigProvider; +import org.arvados.client.facade.ArvadosFacade; +import org.junit.BeforeClass; + +import static org.junit.Assert.assertTrue; + +public class ArvadosClientIntegrationTest { + + protected static final FileConfigProvider CONFIG = new FileConfigProvider("integration-tests-application.conf"); + protected static final ArvadosFacade FACADE = new ArvadosFacade(CONFIG); + protected static final String PROJECT_UUID = CONFIG.getIntegrationTestProjectUuid(); + + @BeforeClass + public static void validateConfiguration(){ + String msg = " info must be provided in configuration"; + CONFIG.getConfig().entrySet() + .forEach(e -> assertTrue("Parameter " + e.getKey() + msg, !e.getValue().render().equals("\"\""))); + } +} diff --git a/src/test/java/org/arvados/client/test/utils/ArvadosClientMockedWebServerTest.java b/src/test/java/org/arvados/client/test/utils/ArvadosClientMockedWebServerTest.java new file mode 100644 index 0000000000..74324b60fc --- /dev/null +++ b/src/test/java/org/arvados/client/test/utils/ArvadosClientMockedWebServerTest.java @@ -0,0 +1,27 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.test.utils; + +import okhttp3.mockwebserver.MockWebServer; +import org.junit.After; +import org.junit.Before; + +public class ArvadosClientMockedWebServerTest extends ArvadosClientUnitTest { + private static final int PORT = CONFIG.getApiPort(); + protected MockWebServer server = new MockWebServer(); + + @Before + public void setUpServer() throws Exception { + server.start(PORT); + } + + @After + public void tearDownServer() throws Exception { + server.shutdown(); + } +} diff --git a/src/test/java/org/arvados/client/test/utils/ArvadosClientUnitTest.java b/src/test/java/org/arvados/client/test/utils/ArvadosClientUnitTest.java new file mode 100644 index 0000000000..67566b697b --- /dev/null +++ b/src/test/java/org/arvados/client/test/utils/ArvadosClientUnitTest.java @@ -0,0 +1,24 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.test.utils; + +import org.arvados.client.config.FileConfigProvider; +import org.junit.BeforeClass; + +import static org.junit.Assert.assertTrue; + +public class ArvadosClientUnitTest { + + protected static final FileConfigProvider CONFIG = new FileConfigProvider("application.conf"); + + @BeforeClass + public static void validateConfiguration(){ + String msg = " info must be provided in configuration"; + CONFIG.getConfig().entrySet().forEach(e -> assertTrue("Parameter " + e.getKey() + msg, !e.getValue().render().equals("\"\""))); + } +} diff --git a/src/test/java/org/arvados/client/test/utils/FileTestUtils.java b/src/test/java/org/arvados/client/test/utils/FileTestUtils.java new file mode 100644 index 0000000000..295345093b --- /dev/null +++ b/src/test/java/org/arvados/client/test/utils/FileTestUtils.java @@ -0,0 +1,50 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.test.utils; + +import org.apache.commons.io.FileUtils; +import org.assertj.core.util.Lists; + +import java.io.File; +import java.io.IOException; +import java.io.RandomAccessFile; +import java.util.List; + +public class FileTestUtils { + + public static final String FILE_SPLIT_TEST_DIR = "/tmp/file-split"; + public static final String FILE_DOWNLOAD_TEST_DIR = "/tmp/arvados-downloaded"; + public static final String TEST_FILE = FILE_SPLIT_TEST_DIR + "/test-file"; + public static long ONE_FOURTH_GB = FileUtils.ONE_GB / 4; + public static long ONE_EIGTH_GB = FileUtils.ONE_GB / 8; + public static long HALF_GB = FileUtils.ONE_GB / 2; + public static int FILE_SPLIT_SIZE = 64; + + public static void createDirectory(String path) throws Exception { + new File(path).mkdirs(); + } + + public static void cleanDirectory(String directory) throws Exception { + FileUtils.cleanDirectory(new File(directory)); + } + + public static File generateFile(String path, long length) throws IOException { + RandomAccessFile testFile = new RandomAccessFile(path, "rwd"); + testFile.setLength(length); + testFile.close(); + return new File(path); + } + + public static List generatePredefinedFiles() throws IOException { + return Lists.newArrayList( + generateFile(TEST_FILE + 1, FileUtils.ONE_KB), + generateFile(TEST_FILE + 2, FileUtils.ONE_KB * 20), + generateFile(TEST_FILE + " " + 3, FileUtils.ONE_MB) + ); + } +} diff --git a/src/test/java/org/arvados/client/test/utils/RequestMethod.java b/src/test/java/org/arvados/client/test/utils/RequestMethod.java new file mode 100644 index 0000000000..53249c9884 --- /dev/null +++ b/src/test/java/org/arvados/client/test/utils/RequestMethod.java @@ -0,0 +1,13 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.test.utils; + +public enum RequestMethod { + + GET, POST, PUT, DELETE +} diff --git a/src/test/java/org/arvados/client/utils/FileMergeTest.java b/src/test/java/org/arvados/client/utils/FileMergeTest.java new file mode 100644 index 0000000000..00ca0b21bb --- /dev/null +++ b/src/test/java/org/arvados/client/utils/FileMergeTest.java @@ -0,0 +1,48 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.utils; + +import org.arvados.client.test.utils.FileTestUtils; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; + +import java.io.File; +import java.util.List; + +import static org.arvados.client.test.utils.FileTestUtils.*; +import static org.assertj.core.api.Assertions.assertThat; + +public class FileMergeTest { + + @Before + public void setUp() throws Exception { + FileTestUtils.createDirectory(FILE_SPLIT_TEST_DIR); + } + + @Test + public void fileChunksAreMergedIntoOneFile() throws Exception { + + // given + FileTestUtils.generateFile(TEST_FILE, FileTestUtils.ONE_EIGTH_GB); + + List files = FileSplit.split(new File(TEST_FILE), new File(FILE_SPLIT_TEST_DIR), FILE_SPLIT_SIZE); + File targetFile = new File(TEST_FILE); + + // when + FileMerge.merge(files, targetFile); + + // then + assertThat(targetFile.length()).isEqualTo(FileTestUtils.ONE_EIGTH_GB); + } + + @After + public void tearDown() throws Exception { + FileTestUtils.cleanDirectory(FILE_SPLIT_TEST_DIR); + } +} diff --git a/src/test/java/org/arvados/client/utils/FileSplitTest.java b/src/test/java/org/arvados/client/utils/FileSplitTest.java new file mode 100644 index 0000000000..4cc523ce4e --- /dev/null +++ b/src/test/java/org/arvados/client/utils/FileSplitTest.java @@ -0,0 +1,48 @@ +/* + * Copyright (C) The Arvados Authors. All rights reserved. + * + * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0 + * + */ + +package org.arvados.client.utils; + +import org.arvados.client.test.utils.FileTestUtils; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; + +import java.io.File; +import java.util.List; + +import static org.arvados.client.test.utils.FileTestUtils.*; +import static org.assertj.core.api.Assertions.assertThat; + +public class FileSplitTest { + + @Before + public void setUp() throws Exception { + FileTestUtils.createDirectory(FILE_SPLIT_TEST_DIR); + } + + @Test + public void fileIsDividedIntoSmallerChunks() throws Exception { + + // given + int expectedSize = 2; + int expectedFileSizeInBytes = 67108864; + FileTestUtils.generateFile(TEST_FILE, FileTestUtils.ONE_EIGTH_GB); + + // when + List actual = FileSplit.split(new File(TEST_FILE), new File(FILE_SPLIT_TEST_DIR), FILE_SPLIT_SIZE); + + // then + assertThat(actual).hasSize(expectedSize); + assertThat(actual).allMatch(a -> a.length() == expectedFileSizeInBytes); + } + + @After + public void tearDown() throws Exception { + FileTestUtils.cleanDirectory(FILE_SPLIT_TEST_DIR); + } +} diff --git a/src/test/resources/application.conf b/src/test/resources/application.conf new file mode 100644 index 0000000000..f19f3dc9a8 --- /dev/null +++ b/src/test/resources/application.conf @@ -0,0 +1,10 @@ +# configuration for unit tests + +arvados { + api { + port = 9000 + keepweb-port = 9000 + token = 1m69yw9m2wanubzyfkb1e9icplqhtr2r969bu9rnzqbqhb7cnb + protocol = "http" + } +} \ No newline at end of file diff --git a/src/test/resources/integration-tests-application.conf b/src/test/resources/integration-tests-application.conf new file mode 100644 index 0000000000..2f934d4cdf --- /dev/null +++ b/src/test/resources/integration-tests-application.conf @@ -0,0 +1,23 @@ +# Configuration for integration tests +# +# Remarks: +# * For example see integration-tests-application.conf.example +# * While providing data remove apostrophes ("") from each line +# * See Arvados documentation for information how to obtain a token: +# https://doc.arvados.org/user/reference/api-tokens.html +# + +arvados { + api { + keepweb-host = "" + keepweb-port = 443 + host = "" + port = 443 + token = "" + protocol = https + host-insecure = false + } + integration-tests { + project-uuid = "" + } +} diff --git a/src/test/resources/integration-tests-application.conf.example b/src/test/resources/integration-tests-application.conf.example new file mode 100644 index 0000000000..e57991806a --- /dev/null +++ b/src/test/resources/integration-tests-application.conf.example @@ -0,0 +1,16 @@ +# example configuration for integration tests + +arvados { + api { + keepweb-host = collections.ardev.mycompany.com + keepweb-port = 443 + host = api.ardev.mycompany.com + port = 443 + token = mytoken + protocol = https + host-insecure = false + } + integration-tests { + project-uuid = ardev-j7d0g-aa123f81q6y7skk + } +} \ No newline at end of file diff --git a/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker b/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker new file mode 100644 index 0000000000..ca6ee9cea8 --- /dev/null +++ b/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker @@ -0,0 +1 @@ +mock-maker-inline \ No newline at end of file diff --git a/src/test/resources/org/arvados/client/api/client/collections-create-manifest.json b/src/test/resources/org/arvados/client/api/client/collections-create-manifest.json new file mode 100644 index 0000000000..68dce30206 --- /dev/null +++ b/src/test/resources/org/arvados/client/api/client/collections-create-manifest.json @@ -0,0 +1,22 @@ +{ + "href": "/collections/112ci-4zz18-12tncxzptzbec1p", + "kind": "arvados#collection", + "etag": "bqoujj7oybdx0jybwvtsebj7y", + "uuid": "112ci-4zz18-12tncxzptzbec1p", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-21T13:38:56.521853000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-21T13:38:56.521853000Z", + "name": "Super Collection", + "description": null, + "properties": {}, + "portable_data_hash": "d41d8cd98f00b204e9800998ecf8427e+0", + "manifest_text": ". 7df44272090cee6c0732382bba415ee9+70+Aa5ece4560e3329315165b36c239b8ab79c888f8a@5a1d5708 0:70:README.md\n", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false +} \ No newline at end of file diff --git a/src/test/resources/org/arvados/client/api/client/collections-create-simple.json b/src/test/resources/org/arvados/client/api/client/collections-create-simple.json new file mode 100644 index 0000000000..57a2ee5a5b --- /dev/null +++ b/src/test/resources/org/arvados/client/api/client/collections-create-simple.json @@ -0,0 +1,22 @@ +{ + "href": "/collections/112ci-4zz18-12tncxzptzbec1p", + "kind": "arvados#collection", + "etag": "bqoujj7oybdx0jybwvtsebj7y", + "uuid": "112ci-4zz18-12tncxzptzbec1p", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-21T13:38:56.521853000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-21T13:38:56.521853000Z", + "name": "Super Collection", + "description": null, + "properties": {}, + "portable_data_hash": "d41d8cd98f00b204e9800998ecf8427e+0", + "manifest_text": "", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false +} \ No newline at end of file diff --git a/src/test/resources/org/arvados/client/api/client/collections-download-file.json b/src/test/resources/org/arvados/client/api/client/collections-download-file.json new file mode 100644 index 0000000000..1fed3832b0 --- /dev/null +++ b/src/test/resources/org/arvados/client/api/client/collections-download-file.json @@ -0,0 +1,22 @@ +{ + "href": "/collections/ardev-4zz18-jk5vo4uo9u5vj52", + "kind": "arvados#collection", + "etag": "2vm76dxmzr23u9774iguuxsrg", + "uuid": "ardev-4zz18-jk5vo4uo9u5vj52", + "owner_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "created_at": "2018-02-19T11:00:00.852389000Z", + "modified_by_client_uuid": "ardev-ozdt8-97tzh5x96spqkay", + "modified_by_user_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "modified_at": "2018-02-19T11:00:00.852389000Z", + "name": "New Collection (2018-02-19 12:00:00.273)", + "description": null, + "properties": {}, + "portable_data_hash": "49581091dfad651945c12b08d4735d88+112", + "manifest_text": ". 163679d58edaadc28db769011728a72c+1070080+A3acf8c1fe582c265d2077702e4a7d74fcc03aba8@5aa4fdeb 0:1024:test-file1 1024:20480:test-file2 21504:1048576:test-file\\0403\n", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false +} \ No newline at end of file diff --git a/src/test/resources/org/arvados/client/api/client/collections-get.json b/src/test/resources/org/arvados/client/api/client/collections-get.json new file mode 100644 index 0000000000..e8fdd83e71 --- /dev/null +++ b/src/test/resources/org/arvados/client/api/client/collections-get.json @@ -0,0 +1,22 @@ +{ + "href": "/collections/112ci-4zz18-p51w7z3fpopo6sm", + "kind": "arvados#collection", + "etag": "52tk5yg024cwhkkcidu3zcmj2", + "uuid": "112ci-4zz18-p51w7z3fpopo6sm", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-15T10:36:03.554356000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-15T10:36:03.554356000Z", + "name": "Collection With Manifest #2", + "description": null, + "properties": {}, + "portable_data_hash": "6c4106229b08fe25f48b3a7a8289dd46+143", + "manifest_text": ". 66c9daa69630e092e9ce554b7aae8a20+524288+A4a15ffea58f259e09f68d3f7eea29942750a79d0@5a269ff6 435f38dd384b06c248feabee0cabca52+524288+A8a99e8148bd368c49901526098901bb7d7890c3b@5a269ff6 dc5b6c104aab35fff6d70a4dadc28d37+391727+Ab0662d549c422c983fccaad02b4ade7b48a8255b@5a269ff6 0:1440303:lombok.jar\n", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false +} \ No newline at end of file diff --git a/src/test/resources/org/arvados/client/api/client/collections-list.json b/src/test/resources/org/arvados/client/api/client/collections-list.json new file mode 100644 index 0000000000..86a3bdafbb --- /dev/null +++ b/src/test/resources/org/arvados/client/api/client/collections-list.json @@ -0,0 +1,871 @@ +{ + "kind": "arvados#collectionList", + "etag": "", + "self_link": "", + "offset": 0, + "limit": 100, + "items": [ + { + "href": "/collections/112ci-4zz18-x6xfmvz0chnkzgv", + "kind": "arvados#collection", + "etag": "8xyiwnih5b5vzmj5sa33348a7", + "uuid": "112ci-4zz18-x6xfmvz0chnkzgv", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-15T13:06:36.934337000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-15T13:06:36.934337000Z", + "name": "Collection With Manifest #3", + "description": null, + "properties": {}, + "portable_data_hash": "6c4106229b08fe25f48b3a7a8289dd46+143", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-p51w7z3fpopo6sm", + "kind": "arvados#collection", + "etag": "8cmhep8aixe4p42pxjoct5502", + "uuid": "112ci-4zz18-p51w7z3fpopo6sm", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-15T10:36:03.554356000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-15T10:36:03.554356000Z", + "name": "Collection With Manifest #2", + "description": null, + "properties": {}, + "portable_data_hash": "6c4106229b08fe25f48b3a7a8289dd46+143", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-xb6gf2yraln7cwa", + "kind": "arvados#collection", + "etag": "de2ol2dyvsba3mn46al760cyg", + "uuid": "112ci-4zz18-xb6gf2yraln7cwa", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-15T09:32:44.146172000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-15T09:32:44.146172000Z", + "name": "New collection", + "description": null, + "properties": {}, + "portable_data_hash": "d41d8cd98f00b204e9800998ecf8427e+0", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-r5jfktpn3a9o0ap", + "kind": "arvados#collection", + "etag": "dby68gd0vatvi090cu0axvtq3", + "uuid": "112ci-4zz18-r5jfktpn3a9o0ap", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-14T13:00:35.431046000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-14T13:00:35.431046000Z", + "name": "Collection With Manifest #1", + "description": null, + "properties": {}, + "portable_data_hash": "3c59518bf8e1100d420488d822682b4a+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-nqxk8xjn6mtskzt", + "kind": "arvados#collection", + "etag": "2b34uzau862w862a2rv36agv6", + "uuid": "112ci-4zz18-nqxk8xjn6mtskzt", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-14T12:59:34.767068000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-14T12:59:34.767068000Z", + "name": "Empty Collection #2", + "description": null, + "properties": {}, + "portable_data_hash": "d41d8cd98f00b204e9800998ecf8427e+0", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-rs9bcf5qnyfjrkm", + "kind": "arvados#collection", + "etag": "60aywazztwfspnasltufcjxpa", + "uuid": "112ci-4zz18-rs9bcf5qnyfjrkm", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-14T12:52:33.124452000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-14T12:52:33.124452000Z", + "name": "Empty Collection #1", + "description": null, + "properties": {}, + "portable_data_hash": "d41d8cd98f00b204e9800998ecf8427e+0", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-af656lee4kv7q2m", + "kind": "arvados#collection", + "etag": "1jward6snif3tsjzftxh8hvwh", + "uuid": "112ci-4zz18-af656lee4kv7q2m", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-14T12:09:05.319319000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-14T12:09:05.319319000Z", + "name": "create example", + "description": null, + "properties": {}, + "portable_data_hash": "d41d8cd98f00b204e9800998ecf8427e+0", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-y2zqix7k9an7nro", + "kind": "arvados#collection", + "etag": "zs2n4zliu6nb5yk3rw6h5ugw", + "uuid": "112ci-4zz18-y2zqix7k9an7nro", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-13T16:59:02.299257000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-13T16:59:02.299257000Z", + "name": "Saved at 2017-11-13 16:59:01 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "5ba3fc508718fabfa20d24390fe31856+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-wq77jfi62u5i4rv", + "kind": "arvados#collection", + "etag": "eijhemzgy44ofmu0dtrowl604", + "uuid": "112ci-4zz18-wq77jfi62u5i4rv", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-13T16:58:10.637548000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-13T16:58:10.637548000Z", + "name": "Saved at 2017-11-13 16:58:07 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "5ba3fc508718fabfa20d24390fe31856+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-unaeckkjgeg7ui0", + "kind": "arvados#collection", + "etag": "1oq7ye0gfbf3ih6y864w3n683", + "uuid": "112ci-4zz18-unaeckkjgeg7ui0", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-10T09:43:07.583862000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-10T09:43:07.583862000Z", + "name": "Saved at 2017-11-10 09:43:03 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "5ba3fc508718fabfa20d24390fe31856+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-5y6atonkxq55lms", + "kind": "arvados#collection", + "etag": "4qmqlro878yx8q7ikhilo8qwn", + "uuid": "112ci-4zz18-5y6atonkxq55lms", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T12:46:15.245770000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T12:46:15.245770000Z", + "name": "Saved at 2017-11-09 12:46:13 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "5ba3fc508718fabfa20d24390fe31856+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-b3fjqd01pxjvseo", + "kind": "arvados#collection", + "etag": "91v698hngoz241c38bbmh0ogc", + "uuid": "112ci-4zz18-b3fjqd01pxjvseo", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T11:54:07.259998000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T11:54:07.259998000Z", + "name": "Saved at 2017-11-09 11:54:04 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "5ba3fc508718fabfa20d24390fe31856+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-cwfxl8h41q18n65", + "kind": "arvados#collection", + "etag": "215t842ckrrgjpxrxr4j0gsui", + "uuid": "112ci-4zz18-cwfxl8h41q18n65", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T11:49:38.276888000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T11:49:38.276888000Z", + "name": "Saved at 2017-11-09 11:49:35 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "5ba3fc508718fabfa20d24390fe31856+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-uv4xu08739tn1vy", + "kind": "arvados#collection", + "etag": "90z6i3oqv197osng3wvjjir3t", + "uuid": "112ci-4zz18-uv4xu08739tn1vy", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T11:43:05.917513000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T11:43:05.917513000Z", + "name": "Saved at 2017-11-09 11:43:05 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "5ba3fc508718fabfa20d24390fe31856+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-pzisn8c5mefzczv", + "kind": "arvados#collection", + "etag": "5lcf6wvc3wypwobswdz22wen", + "uuid": "112ci-4zz18-pzisn8c5mefzczv", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T11:40:38.804718000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T11:40:38.804718000Z", + "name": "Saved at 2017-11-09 11:40:36 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "5ba3fc508718fabfa20d24390fe31856+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-mj24uwtnqqrno27", + "kind": "arvados#collection", + "etag": "98s08xew49avui1gy3mzit8je", + "uuid": "112ci-4zz18-mj24uwtnqqrno27", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T11:40:25.189869000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T11:40:25.189869000Z", + "name": "Saved at 2017-11-09 11:40:24 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "5ba3fc508718fabfa20d24390fe31856+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-oco162516upgqng", + "kind": "arvados#collection", + "etag": "a09wnvl4i51xqx7u9yf4qbi94", + "uuid": "112ci-4zz18-oco162516upgqng", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T11:39:04.148785000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T11:39:04.148785000Z", + "name": "Saved at 2017-11-09 11:39:03 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "5ba3fc508718fabfa20d24390fe31856+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-tlze7dgczsdwkep", + "kind": "arvados#collection", + "etag": "4ee2xudbc5rkr597drgu9tg10", + "uuid": "112ci-4zz18-tlze7dgczsdwkep", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T11:37:59.478975000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T11:37:59.478975000Z", + "name": "Saved at 2017-11-09 11:37:58 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "5ba3fc508718fabfa20d24390fe31856+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-nq0kxi9d7w64la1", + "kind": "arvados#collection", + "etag": "5aa3evnbceo3brnps2e1sq8ts", + "uuid": "112ci-4zz18-nq0kxi9d7w64la1", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T11:32:23.329259000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T11:32:23.329259000Z", + "name": "Saved at 2017-11-09 11:32:22 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "5ba3fc508718fabfa20d24390fe31856+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-fks9mewtw155pvx", + "kind": "arvados#collection", + "etag": "97vicgogv8bovmk4s2jymsdq", + "uuid": "112ci-4zz18-fks9mewtw155pvx", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T11:30:17.589462000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T11:30:17.589462000Z", + "name": "Saved at 2017-11-09 11:30:17 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "5ba3fc508718fabfa20d24390fe31856+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-kp356e0q2wdl2df", + "kind": "arvados#collection", + "etag": "btktwjclv063s1rd6duvk51v3", + "uuid": "112ci-4zz18-kp356e0q2wdl2df", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T11:29:26.820481000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T11:29:26.820481000Z", + "name": "Saved at 2017-11-09 11:29:25 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "5ba3fc508718fabfa20d24390fe31856+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-0ey8ob38xf7surq", + "kind": "arvados#collection", + "etag": "bob83na42pufqli1a5buxryvm", + "uuid": "112ci-4zz18-0ey8ob38xf7surq", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T11:08:53.781498000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T11:08:53.781498000Z", + "name": "Saved at 2017-11-09 11:08:52 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "5ba3fc508718fabfa20d24390fe31856+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-wu2n0fv3cewna1n", + "kind": "arvados#collection", + "etag": "7pl1x327eeutqtsjppdj284g8", + "uuid": "112ci-4zz18-wu2n0fv3cewna1n", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T11:08:33.423284000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T11:08:33.423284000Z", + "name": "Saved at 2017-11-09 11:08:33 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "5ba3fc508718fabfa20d24390fe31856+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-hyybo6yuvkx4hrm", + "kind": "arvados#collection", + "etag": "2wg1wn2o18ubrgbhbqwwsslhf", + "uuid": "112ci-4zz18-hyybo6yuvkx4hrm", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T10:44:53.096798000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T10:44:53.096798000Z", + "name": "Saved at 2017-11-09 10:44:51 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "5ba3fc508718fabfa20d24390fe31856+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-h3gjq7gzd4syanw", + "kind": "arvados#collection", + "etag": "8jk0at4e69cwjyjamvm4wz2oj", + "uuid": "112ci-4zz18-h3gjq7gzd4syanw", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T10:41:31.278281000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T10:41:31.278281000Z", + "name": "Saved at 2017-11-09 10:41:30 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "5ba3fc508718fabfa20d24390fe31856+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-jinwyyaeigjs1yg", + "kind": "arvados#collection", + "etag": "be57zhzufz2hp1tbdwidoro5j", + "uuid": "112ci-4zz18-jinwyyaeigjs1yg", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T10:41:07.083017000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T10:41:07.083017000Z", + "name": "Saved at 2017-11-09 10:41:06 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "5ba3fc508718fabfa20d24390fe31856+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-etf8aghyxlfxvo1", + "kind": "arvados#collection", + "etag": "29lj2roie4cygo5ffgrduflly", + "uuid": "112ci-4zz18-etf8aghyxlfxvo1", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T10:40:31.710865000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T10:40:31.710865000Z", + "name": "Saved at 2017-11-09 10:40:31 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "5ba3fc508718fabfa20d24390fe31856+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-jtbn4edpkkhbm9b", + "kind": "arvados#collection", + "etag": "6div78e1nhusii4x1xkp3rg2v", + "uuid": "112ci-4zz18-jtbn4edpkkhbm9b", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T10:39:36.999602000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T10:39:36.999602000Z", + "name": "Saved at 2017-11-09 10:39:36 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "5ba3fc508718fabfa20d24390fe31856+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-whdleimp34hiqp6", + "kind": "arvados#collection", + "etag": "12wlbsxlmy3sze4v2m0ua7ake", + "uuid": "112ci-4zz18-whdleimp34hiqp6", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T10:19:52.879907000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T10:19:52.879907000Z", + "name": "Saved at 2017-11-09 10:19:52 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "5ba3fc508718fabfa20d24390fe31856+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-kj8dz72zpo5kbtm", + "kind": "arvados#collection", + "etag": "9bv1bw9afb3w84gu55uzcgd6h", + "uuid": "112ci-4zz18-kj8dz72zpo5kbtm", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T10:16:31.558621000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T10:16:31.558621000Z", + "name": "Saved at 2017-11-09 10:16:30 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "5ba3fc508718fabfa20d24390fe31856+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-tr306nau9hrr437", + "kind": "arvados#collection", + "etag": "683d77tvlhe97etk9bk2bx8ds", + "uuid": "112ci-4zz18-tr306nau9hrr437", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T09:59:44.978811000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T09:59:44.978811000Z", + "name": "Saved at 2017-11-09 09:59:44 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "67cbebb9f739b6b06ca056d21115cf43+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-oxuk69569mxztp0", + "kind": "arvados#collection", + "etag": "1m34v9jbna2v7gv7auio54i8w", + "uuid": "112ci-4zz18-oxuk69569mxztp0", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T09:59:30.774888000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T09:59:30.774888000Z", + "name": "Saved at 2017-11-09 09:59:30 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "67cbebb9f739b6b06ca056d21115cf43+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-wf8sl6xbyfwjyer", + "kind": "arvados#collection", + "etag": "7l2a9fhqmxg7ghn7osx0s19v4", + "uuid": "112ci-4zz18-wf8sl6xbyfwjyer", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T09:58:21.496088000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T09:58:21.496088000Z", + "name": "Saved at 2017-11-09 09:58:20 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "67cbebb9f739b6b06ca056d21115cf43+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-drpia2es1hp9ydi", + "kind": "arvados#collection", + "etag": "33dw426fhs2vlb50b6301ukn0", + "uuid": "112ci-4zz18-drpia2es1hp9ydi", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T09:56:08.506505000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T09:56:08.506505000Z", + "name": "Saved at 2017-11-09 09:56:08 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "67cbebb9f739b6b06ca056d21115cf43+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-5b4px2i2dwyidfi", + "kind": "arvados#collection", + "etag": "2437tnhn2gmti52lpm8nfq9ct", + "uuid": "112ci-4zz18-5b4px2i2dwyidfi", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T09:54:06.651026000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T09:54:06.651026000Z", + "name": "Saved at 2017-11-09 09:54:06 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "67cbebb9f739b6b06ca056d21115cf43+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-94oslnwnxe1f9wp", + "kind": "arvados#collection", + "etag": "7e0k48zu93o57zudxjp1yrgjq", + "uuid": "112ci-4zz18-94oslnwnxe1f9wp", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T09:40:04.240297000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T09:40:04.240297000Z", + "name": "Saved at 2017-11-09 09:39:58 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "67cbebb9f739b6b06ca056d21115cf43+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-2fk0d5d4jjc1fmq", + "kind": "arvados#collection", + "etag": "cuirr803f54e89reakuq50oaq", + "uuid": "112ci-4zz18-2fk0d5d4jjc1fmq", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T09:36:14.952671000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T09:36:14.952671000Z", + "name": "Saved at 2017-11-09 09:36:08 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "67cbebb9f739b6b06ca056d21115cf43+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-xp9pu81xyc5h422", + "kind": "arvados#collection", + "etag": "3bi5xd8ezxrazk5266cwzn4s4", + "uuid": "112ci-4zz18-xp9pu81xyc5h422", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T09:35:29.552746000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T09:35:29.552746000Z", + "name": "Saved at 2017-11-09 09:35:29 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "67cbebb9f739b6b06ca056d21115cf43+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-znb4lo0if2as58c", + "kind": "arvados#collection", + "etag": "59uaoxy6uh82i6lrvr3ht8gz1", + "uuid": "112ci-4zz18-znb4lo0if2as58c", + "owner_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "created_at": "2017-11-09T09:31:08.109971000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-09T09:31:08.109971000Z", + "name": "Saved at 2017-11-09 09:31:06 UTC by VirtualBox", + "description": null, + "properties": {}, + "portable_data_hash": "67cbebb9f739b6b06ca056d21115cf43+53", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-6pvl5ea5u932qzi", + "kind": "arvados#collection", + "etag": "dksrh8jznxoaidl29i1vv5904", + "uuid": "112ci-4zz18-6pvl5ea5u932qzi", + "owner_uuid": "112ci-j7d0g-tw71k7mxii6fqgx", + "created_at": "2017-11-08T12:48:32.238698000Z", + "modified_by_client_uuid": "112ci-ozdt8-f4633qdjs6w8zcy", + "modified_by_user_uuid": "112ci-tpzed-nd84czdo4iea1mz", + "modified_at": "2017-11-08T12:50:23.946608000Z", + "name": "New collection", + "description": null, + "properties": {}, + "portable_data_hash": "18c037c51c3f74be53ea2b115afd0c5f+69", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/collections/112ci-4zz18-wq5pyrxfv1t9isu", + "kind": "arvados#collection", + "etag": "1w1rhhd6oql4ceb7h9t16sf0q", + "uuid": "112ci-4zz18-wq5pyrxfv1t9isu", + "owner_uuid": "112ci-j7d0g-anonymouspublic", + "created_at": "2017-11-03T10:03:20.364737000Z", + "modified_by_client_uuid": null, + "modified_by_user_uuid": "112ci-tpzed-000000000000000", + "modified_at": "2017-11-03T10:03:20.364737000Z", + "name": null, + "description": null, + "properties": {}, + "portable_data_hash": "d41d8cd98f00b204e9800998ecf8427e+0", + "replication_desired": null, + "replication_confirmed": null, + "replication_confirmed_at": null, + "delete_at": null, + "trash_at": null, + "is_trashed": false + } + ], + "items_available": 41 +} \ No newline at end of file diff --git a/src/test/resources/org/arvados/client/api/client/groups-get.json b/src/test/resources/org/arvados/client/api/client/groups-get.json new file mode 100644 index 0000000000..f1834e749c --- /dev/null +++ b/src/test/resources/org/arvados/client/api/client/groups-get.json @@ -0,0 +1,21 @@ +{ + "href": "/groups/ardev-j7d0g-bmg3pfqtx3ivczp", + "kind": "arvados#group", + "etag": "3hw0vk4mbl0ofvia5k6x4dwrx", + "uuid": "ardev-j7d0g-bmg3pfqtx3ivczp", + "owner_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "created_at": "2018-03-29T11:09:05.984597000Z", + "modified_by_client_uuid": "ardev-ozdt8-97tzh5x96spqkay", + "modified_by_user_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "modified_at": "2018-03-29T11:09:05.984597000Z", + "name": "TestGroup1", + "group_class": "project", + "description": null, + "writable_by": [ + "ardev-tpzed-n3kzq4fvoks3uw4", + "ardev-tpzed-n3kzq4fvoks3uw4" + ], + "delete_at": null, + "trash_at": null, + "is_trashed": false +} \ No newline at end of file diff --git a/src/test/resources/org/arvados/client/api/client/groups-list.json b/src/test/resources/org/arvados/client/api/client/groups-list.json new file mode 100644 index 0000000000..fa74e1cb53 --- /dev/null +++ b/src/test/resources/org/arvados/client/api/client/groups-list.json @@ -0,0 +1,430 @@ +{ + "kind": "arvados#groupList", + "etag": "", + "self_link": "", + "offset": 0, + "limit": 100, + "items": [ + { + "href": "/groups/ardev-j7d0g-ylx7wnu1moge2di", + "kind": "arvados#group", + "etag": "68vubv3iw7663763bozxebmyf", + "uuid": "ardev-j7d0g-ylx7wnu1moge2di", + "owner_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "created_at": "2018-04-18T09:09:21.126649000Z", + "modified_by_client_uuid": "ardev-ozdt8-97tzh5x96spqkay", + "modified_by_user_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "modified_at": "2018-04-18T09:09:21.126649000Z", + "name": "TestProject1", + "group_class": "project", + "description": null, + "writable_by": [ + "ardev-tpzed-n3kzq4fvoks3uw4", + "ardev-tpzed-n3kzq4fvoks3uw4" + ], + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/groups/ardev-j7d0g-mnzhga726itrbrq", + "kind": "arvados#group", + "etag": "68q7r8r37u9hckr2zsynvton3", + "uuid": "ardev-j7d0g-mnzhga726itrbrq", + "owner_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "created_at": "2018-04-17T12:11:24.389594000Z", + "modified_by_client_uuid": "ardev-ozdt8-97tzh5x96spqkay", + "modified_by_user_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "modified_at": "2018-04-17T12:11:24.389594000Z", + "name": "TestProject2", + "group_class": "project", + "description": null, + "writable_by": [ + "ardev-tpzed-n3kzq4fvoks3uw4", + "ardev-tpzed-n3kzq4fvoks3uw4" + ], + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/groups/ardev-j7d0g-0w9m1sz46ljtdnm", + "kind": "arvados#group", + "etag": "ef4vzx5gyudkrg9zml0zdv6qu", + "uuid": "ardev-j7d0g-0w9m1sz46ljtdnm", + "owner_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "created_at": "2018-04-17T12:08:39.066802000Z", + "modified_by_client_uuid": "ardev-ozdt8-97tzh5x96spqkay", + "modified_by_user_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "modified_at": "2018-04-17T12:08:39.066802000Z", + "name": "TestProject3", + "group_class": "project", + "description": null, + "writable_by": [ + "ardev-tpzed-n3kzq4fvoks3uw4", + "ardev-tpzed-n3kzq4fvoks3uw4" + ], + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/groups/ardev-j7d0g-r20iem5ou6h5wao", + "kind": "arvados#group", + "etag": "6h6h4ta6yyf9058delxk8fnqs", + "uuid": "ardev-j7d0g-r20iem5ou6h5wao", + "owner_uuid": "ardev-j7d0g-j7drd8yikkp6evd", + "created_at": "2018-04-17T12:03:39.647244000Z", + "modified_by_client_uuid": "ardev-ozdt8-97tzh5x96spqkay", + "modified_by_user_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "modified_at": "2018-04-17T12:03:39.647244000Z", + "name": "TestProject4", + "group_class": "project", + "description": null, + "writable_by": [ + "ardev-j7d0g-j7drd8yikkp6evd", + "ardev-tpzed-n3kzq4fvoks3uw4" + ], + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/groups/ardev-j7d0g-j7drd8yikkp6evd", + "kind": "arvados#group", + "etag": "6se2y8f9o7uu06pbopgq56xds", + "uuid": "ardev-j7d0g-j7drd8yikkp6evd", + "owner_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "created_at": "2018-04-17T11:58:31.339515000Z", + "modified_by_client_uuid": "ardev-ozdt8-97tzh5x96spqkay", + "modified_by_user_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "modified_at": "2018-04-17T11:58:31.339515000Z", + "name": "TestProject5", + "group_class": "project", + "description": null, + "writable_by": [ + "ardev-tpzed-n3kzq4fvoks3uw4", + "ardev-tpzed-n3kzq4fvoks3uw4" + ], + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/groups/ardev-j7d0g-kh1g7i5va870xt0", + "kind": "arvados#group", + "etag": "2si26vaig3vig9266pqkqh2gy", + "uuid": "ardev-j7d0g-kh1g7i5va870xt0", + "owner_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "created_at": "2018-04-17T10:56:54.391676000Z", + "modified_by_client_uuid": "ardev-ozdt8-97tzh5x96spqkay", + "modified_by_user_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "modified_at": "2018-04-17T10:56:54.391676000Z", + "name": "TestProject6", + "group_class": "project", + "description": null, + "writable_by": [ + "ardev-tpzed-n3kzq4fvoks3uw4", + "ardev-tpzed-n3kzq4fvoks3uw4" + ], + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/groups/ardev-j7d0g-sclkdyuwm4h2m78", + "kind": "arvados#group", + "etag": "edgnz6q0vt2u3o13ujtfohb75", + "uuid": "ardev-j7d0g-sclkdyuwm4h2m78", + "owner_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "created_at": "2018-04-17T10:27:15.914517000Z", + "modified_by_client_uuid": "ardev-ozdt8-97tzh5x96spqkay", + "modified_by_user_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "modified_at": "2018-04-17T10:27:15.914517000Z", + "name": "TestProject7", + "group_class": "project", + "description": null, + "writable_by": [ + "ardev-tpzed-n3kzq4fvoks3uw4", + "ardev-tpzed-n3kzq4fvoks3uw4" + ], + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/groups/ardev-j7d0g-593khc577zuyyhe", + "kind": "arvados#group", + "etag": "39ig9ttgec6lbe096uetn2cb9", + "uuid": "ardev-j7d0g-593khc577zuyyhe", + "owner_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "created_at": "2018-04-17T10:27:03.858203000Z", + "modified_by_client_uuid": "ardev-ozdt8-97tzh5x96spqkay", + "modified_by_user_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "modified_at": "2018-04-17T10:27:03.858203000Z", + "name": "TestProject8", + "group_class": "project", + "description": null, + "writable_by": [ + "ardev-tpzed-n3kzq4fvoks3uw4", + "ardev-tpzed-n3kzq4fvoks3uw4" + ], + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/groups/ardev-j7d0g-iotds0tm559dbz7", + "kind": "arvados#group", + "etag": "1dpr8v6tx6pta0fozq93eyeou", + "uuid": "ardev-j7d0g-iotds0tm559dbz7", + "owner_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "created_at": "2018-04-17T10:26:25.180623000Z", + "modified_by_client_uuid": "ardev-ozdt8-97tzh5x96spqkay", + "modified_by_user_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "modified_at": "2018-04-17T10:26:25.180623000Z", + "name": "TestProject9", + "group_class": "project", + "description": null, + "writable_by": [ + "ardev-tpzed-n3kzq4fvoks3uw4", + "ardev-tpzed-n3kzq4fvoks3uw4" + ], + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/groups/ardev-j7d0g-gbqay74778tonb8", + "kind": "arvados#group", + "etag": "dizbavs2opfe1wpx6thocfki0", + "uuid": "ardev-j7d0g-gbqay74778tonb8", + "owner_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "created_at": "2018-04-17T10:26:06.435961000Z", + "modified_by_client_uuid": "ardev-ozdt8-97tzh5x96spqkay", + "modified_by_user_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "modified_at": "2018-04-17T10:26:06.435961000Z", + "name": "TestProject10", + "group_class": "project", + "description": null, + "writable_by": [ + "ardev-tpzed-n3kzq4fvoks3uw4", + "ardev-tpzed-n3kzq4fvoks3uw4" + ], + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/groups/ardev-j7d0g-fmq1t0jlznehbdm", + "kind": "arvados#group", + "etag": "6xue8m3lx9qpptfvdf13val5t", + "uuid": "ardev-j7d0g-fmq1t0jlznehbdm", + "owner_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "created_at": "2018-04-17T10:25:55.546399000Z", + "modified_by_client_uuid": "ardev-ozdt8-97tzh5x96spqkay", + "modified_by_user_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "modified_at": "2018-04-17T10:25:55.546399000Z", + "name": "TestProject11", + "group_class": "project", + "description": null, + "writable_by": [ + "ardev-tpzed-n3kzq4fvoks3uw4", + "ardev-tpzed-n3kzq4fvoks3uw4" + ], + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/groups/ardev-j7d0g-vxju56ch64u51gq", + "kind": "arvados#group", + "etag": "2gqix9e4m023usi9exhrsjx6z", + "uuid": "ardev-j7d0g-vxju56ch64u51gq", + "owner_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "created_at": "2018-04-16T14:09:49.700566000Z", + "modified_by_client_uuid": "ardev-ozdt8-97tzh5x96spqkay", + "modified_by_user_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "modified_at": "2018-04-16T14:09:49.700566000Z", + "name": "TestProject12", + "group_class": "project", + "description": null, + "writable_by": [ + "ardev-tpzed-n3kzq4fvoks3uw4", + "ardev-tpzed-n3kzq4fvoks3uw4" + ], + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/groups/ardev-j7d0g-g8m4w0d22gv6fbj", + "kind": "arvados#group", + "etag": "73n8x82814o6ihld0kltf468d", + "uuid": "ardev-j7d0g-g8m4w0d22gv6fbj", + "owner_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "created_at": "2018-04-11T15:02:35.016850000Z", + "modified_by_client_uuid": "ardev-ozdt8-97tzh5x96spqkay", + "modified_by_user_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "modified_at": "2018-04-11T15:02:35.016850000Z", + "name": "TestProject13", + "group_class": "project", + "description": null, + "writable_by": [ + "ardev-tpzed-n3kzq4fvoks3uw4", + "ardev-tpzed-n3kzq4fvoks3uw4" + ], + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/groups/ardev-j7d0g-lstqed4y78khaqm", + "kind": "arvados#group", + "etag": "91f7uwq7pj3d3ez1u4smjg3ch", + "uuid": "ardev-j7d0g-lstqed4y78khaqm", + "owner_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "created_at": "2018-04-06T15:29:27.754408000Z", + "modified_by_client_uuid": "ardev-ozdt8-97tzh5x96spqkay", + "modified_by_user_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "modified_at": "2018-04-06T15:29:27.754408000Z", + "name": "TestProject14", + "group_class": "project", + "description": null, + "writable_by": [ + "ardev-tpzed-n3kzq4fvoks3uw4", + "ardev-tpzed-n3kzq4fvoks3uw4" + ], + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/groups/ardev-j7d0g-0jbezvnq8i07l7p", + "kind": "arvados#group", + "etag": "7dbxhvbcfaogwnvo8k4mtqthk", + "uuid": "ardev-j7d0g-0jbezvnq8i07l7p", + "owner_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "created_at": "2018-04-05T09:32:46.946417000Z", + "modified_by_client_uuid": "ardev-ozdt8-97tzh5x96spqkay", + "modified_by_user_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "modified_at": "2018-04-05T09:32:46.946417000Z", + "name": "TestProject15", + "group_class": "project", + "description": null, + "writable_by": [ + "ardev-tpzed-n3kzq4fvoks3uw4", + "ardev-tpzed-n3kzq4fvoks3uw4" + ], + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/groups/ardev-j7d0g-72dxer22g6iltqz", + "kind": "arvados#group", + "etag": "dhfu203rckzdzvx832wm7jv59", + "uuid": "ardev-j7d0g-72dxer22g6iltqz", + "owner_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "created_at": "2018-03-29T11:27:02.482218000Z", + "modified_by_client_uuid": "ardev-ozdt8-97tzh5x96spqkay", + "modified_by_user_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "modified_at": "2018-03-29T13:17:00.045606000Z", + "name": "TestProject16", + "group_class": "project", + "description": null, + "writable_by": [ + "ardev-tpzed-n3kzq4fvoks3uw4", + "ardev-tpzed-n3kzq4fvoks3uw4" + ], + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/groups/ardev-j7d0g-nebzwquxtq1v3o5", + "kind": "arvados#group", + "etag": "7l9oxbdf4e1m9ddnujokf7czz", + "uuid": "ardev-j7d0g-nebzwquxtq1v3o5", + "owner_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "created_at": "2018-03-29T11:11:26.235411000Z", + "modified_by_client_uuid": "ardev-ozdt8-97tzh5x96spqkay", + "modified_by_user_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "modified_at": "2018-03-29T11:11:26.235411000Z", + "name": "TestProject17", + "group_class": "project", + "description": null, + "writable_by": [ + "ardev-tpzed-n3kzq4fvoks3uw4", + "ardev-tpzed-n3kzq4fvoks3uw4" + ], + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/groups/ardev-j7d0g-5589c8dmxevecqh", + "kind": "arvados#group", + "etag": "83862x2o4453mja2rvypjl5gv", + "uuid": "ardev-j7d0g-5589c8dmxevecqh", + "owner_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "created_at": "2018-03-29T11:10:58.496482000Z", + "modified_by_client_uuid": "ardev-ozdt8-97tzh5x96spqkay", + "modified_by_user_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "modified_at": "2018-03-29T11:10:58.496482000Z", + "name": "TestProject18", + "group_class": "project", + "description": null, + "writable_by": [ + "ardev-tpzed-n3kzq4fvoks3uw4", + "ardev-tpzed-n3kzq4fvoks3uw4" + ], + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/groups/ardev-j7d0g-bmg3pfqtx3ivczp", + "kind": "arvados#group", + "etag": "3hw0vk4mbl0ofvia5k6x4dwrx", + "uuid": "ardev-j7d0g-bmg3pfqtx3ivczp", + "owner_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "created_at": "2018-03-29T11:09:05.984597000Z", + "modified_by_client_uuid": "ardev-ozdt8-97tzh5x96spqkay", + "modified_by_user_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "modified_at": "2018-03-29T11:09:05.984597000Z", + "name": "TestProject19", + "group_class": "project", + "description": null, + "writable_by": [ + "ardev-tpzed-n3kzq4fvoks3uw4", + "ardev-tpzed-n3kzq4fvoks3uw4" + ], + "delete_at": null, + "trash_at": null, + "is_trashed": false + }, + { + "href": "/groups/ardev-j7d0g-mfitz2oa4rpycou", + "kind": "arvados#group", + "etag": "6p9xbxpttj782mpqs537gfvc6", + "uuid": "ardev-j7d0g-mfitz2oa4rpycou", + "owner_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "created_at": "2018-03-29T11:00:19.809612000Z", + "modified_by_client_uuid": "ardev-ozdt8-97tzh5x96spqkay", + "modified_by_user_uuid": "ardev-tpzed-n3kzq4fvoks3uw4", + "modified_at": "2018-03-29T11:00:19.809612000Z", + "name": "TestProject20", + "group_class": "project", + "description": null, + "writable_by": [ + "ardev-tpzed-n3kzq4fvoks3uw4", + "ardev-tpzed-n3kzq4fvoks3uw4" + ], + "delete_at": null, + "trash_at": null, + "is_trashed": false + } + ], + "items_available": 20 +} \ No newline at end of file diff --git a/src/test/resources/org/arvados/client/api/client/keep-client-test-file.txt b/src/test/resources/org/arvados/client/api/client/keep-client-test-file.txt new file mode 100644 index 0000000000..5cbed85e5e --- /dev/null +++ b/src/test/resources/org/arvados/client/api/client/keep-client-test-file.txt @@ -0,0 +1 @@ +Sample text file to test keep client. \ No newline at end of file diff --git a/src/test/resources/org/arvados/client/api/client/keep-services-accessible-disk-only.json b/src/test/resources/org/arvados/client/api/client/keep-services-accessible-disk-only.json new file mode 100644 index 0000000000..d5bd0d83d1 --- /dev/null +++ b/src/test/resources/org/arvados/client/api/client/keep-services-accessible-disk-only.json @@ -0,0 +1,42 @@ +{ + "kind": "arvados#keepServiceList", + "etag": "", + "self_link": "", + "offset": null, + "limit": null, + "items": [ + { + "href": "/keep_services/112ci-bi6l4-hv02fg8sbti8ykk", + "kind": "arvados#keepService", + "etag": "bjzh7og2d9z949lbd38vnnslt", + "uuid": "112ci-bi6l4-hv02fg8sbti8ykk", + "owner_uuid": "112ci-tpzed-000000000000000", + "created_at": "2017-11-03T10:04:48.314229000Z", + "modified_by_client_uuid": "112ci-ozdt8-xxy0ipzwti8gnmt", + "modified_by_user_uuid": "112ci-tpzed-000000000000000", + "modified_at": "2017-11-03T10:04:48.314229000Z", + "service_host": "localhost", + "service_port": 9000, + "service_ssl_flag": false, + "service_type": "disk", + "read_only": false + }, + { + "href": "/keep_services/112ci-bi6l4-f0r03wrqymotwql", + "kind": "arvados#keepService", + "etag": "7m64l69kko4bytpsykf8cay7t", + "uuid": "112ci-bi6l4-f0r03wrqymotwql", + "owner_uuid": "112ci-tpzed-000000000000000", + "created_at": "2017-11-03T10:04:48.351577000Z", + "modified_by_client_uuid": "112ci-ozdt8-xxy0ipzwti8gnmt", + "modified_by_user_uuid": "112ci-tpzed-000000000000000", + "modified_at": "2017-11-03T10:04:48.351577000Z", + "service_host": "localhost", + "service_port": 9001, + "service_ssl_flag": false, + "service_type": "disk", + "read_only": false + } + ], + "items_available": 2 +} \ No newline at end of file diff --git a/src/test/resources/org/arvados/client/api/client/keep-services-accessible.json b/src/test/resources/org/arvados/client/api/client/keep-services-accessible.json new file mode 100644 index 0000000000..3d95cf932f --- /dev/null +++ b/src/test/resources/org/arvados/client/api/client/keep-services-accessible.json @@ -0,0 +1,42 @@ +{ + "kind": "arvados#keepServiceList", + "etag": "", + "self_link": "", + "offset": null, + "limit": null, + "items": [ + { + "href": "/keep_services/112ci-bi6l4-hv02fg8sbti8ykk", + "kind": "arvados#keepService", + "etag": "bjzh7og2d9z949lbd38vnnslt", + "uuid": "112ci-bi6l4-hv02fg8sbti8ykk", + "owner_uuid": "112ci-tpzed-000000000000000", + "created_at": "2017-11-03T10:04:48.314229000Z", + "modified_by_client_uuid": "112ci-ozdt8-xxy0ipzwti8gnmt", + "modified_by_user_uuid": "112ci-tpzed-000000000000000", + "modified_at": "2017-11-03T10:04:48.314229000Z", + "service_host": "localhost", + "service_port": 9000, + "service_ssl_flag": false, + "service_type": "disk", + "read_only": false + }, + { + "href": "/keep_services/112ci-bi6l4-f0r03wrqymotwql", + "kind": "arvados#keepService", + "etag": "7m64l69kko4bytpsykf8cay7t", + "uuid": "112ci-bi6l4-f0r03wrqymotwql", + "owner_uuid": "112ci-tpzed-000000000000000", + "created_at": "2017-11-03T10:04:48.351577000Z", + "modified_by_client_uuid": "112ci-ozdt8-xxy0ipzwti8gnmt", + "modified_by_user_uuid": "112ci-tpzed-000000000000000", + "modified_at": "2017-11-03T10:04:48.351577000Z", + "service_host": "localhost", + "service_port": 9000, + "service_ssl_flag": false, + "service_type": "gpfs", + "read_only": false + } + ], + "items_available": 2 +} \ No newline at end of file diff --git a/src/test/resources/org/arvados/client/api/client/keep-services-get.json b/src/test/resources/org/arvados/client/api/client/keep-services-get.json new file mode 100644 index 0000000000..f3c289497c --- /dev/null +++ b/src/test/resources/org/arvados/client/api/client/keep-services-get.json @@ -0,0 +1,16 @@ +{ + "href": "/keep_services/112ci-bi6l4-hv02fg8sbti8ykk", + "kind": "arvados#keepService", + "etag": "bjzh7og2d9z949lbd38vnnslt", + "uuid": "112ci-bi6l4-hv02fg8sbti8ykk", + "owner_uuid": "112ci-tpzed-000000000000000", + "created_at": "2017-11-03T10:04:48.314229000Z", + "modified_by_client_uuid": "112ci-ozdt8-xxy0ipzwti8gnmt", + "modified_by_user_uuid": "112ci-tpzed-000000000000000", + "modified_at": "2017-11-03T10:04:48.314229000Z", + "service_host": "10.0.2.15", + "service_port": 9000, + "service_ssl_flag": false, + "service_type": "disk", + "read_only": false +} \ No newline at end of file diff --git a/src/test/resources/org/arvados/client/api/client/keep-services-list.json b/src/test/resources/org/arvados/client/api/client/keep-services-list.json new file mode 100644 index 0000000000..90ba91631e --- /dev/null +++ b/src/test/resources/org/arvados/client/api/client/keep-services-list.json @@ -0,0 +1,58 @@ +{ + "kind": "arvados#keepServiceList", + "etag": "", + "self_link": "", + "offset": 0, + "limit": 100, + "items": [ + { + "href": "/keep_services/112ci-bi6l4-f0r03wrqymotwql", + "kind": "arvados#keepService", + "etag": "7m64l69kko4bytpsykf8cay7t", + "uuid": "112ci-bi6l4-f0r03wrqymotwql", + "owner_uuid": "112ci-tpzed-000000000000000", + "created_at": "2017-11-03T10:04:48.351577000Z", + "modified_by_client_uuid": "112ci-ozdt8-xxy0ipzwti8gnmt", + "modified_by_user_uuid": "112ci-tpzed-000000000000000", + "modified_at": "2017-11-03T10:04:48.351577000Z", + "service_host": "10.0.2.15", + "service_port": 9000, + "service_ssl_flag": false, + "service_type": "disk", + "read_only": false + }, + { + "href": "/keep_services/112ci-bi6l4-hv02fg8sbti8ykk", + "kind": "arvados#keepService", + "etag": "bjzh7og2d9z949lbd38vnnslt", + "uuid": "112ci-bi6l4-hv02fg8sbti8ykk", + "owner_uuid": "112ci-tpzed-000000000000000", + "created_at": "2017-11-03T10:04:48.314229000Z", + "modified_by_client_uuid": "112ci-ozdt8-xxy0ipzwti8gnmt", + "modified_by_user_uuid": "112ci-tpzed-000000000000000", + "modified_at": "2017-11-03T10:04:48.314229000Z", + "service_host": "10.0.2.15", + "service_port": 9001, + "service_ssl_flag": false, + "service_type": "disk", + "read_only": false + }, + { + "href": "/keep_services/112ci-bi6l4-ko27cfbsf2ssx2m", + "kind": "arvados#keepService", + "etag": "4be61qkpt6nzdfff4vj9nkpmj", + "uuid": "112ci-bi6l4-ko27cfbsf2ssx2m", + "owner_uuid": "112ci-tpzed-000000000000000", + "created_at": "2017-11-03T10:04:36.355045000Z", + "modified_by_client_uuid": "112ci-ozdt8-xxy0ipzwti8gnmt", + "modified_by_user_uuid": "112ci-tpzed-000000000000000", + "modified_at": "2017-11-03T10:04:36.355045000Z", + "service_host": "10.0.2.15", + "service_port": 9002, + "service_ssl_flag": false, + "service_type": "proxy", + "read_only": false + } + ], + "items_available": 3 +} \ No newline at end of file diff --git a/src/test/resources/org/arvados/client/api/client/keep-services-not-accessible.json b/src/test/resources/org/arvados/client/api/client/keep-services-not-accessible.json new file mode 100644 index 0000000000..c930ee2ce1 --- /dev/null +++ b/src/test/resources/org/arvados/client/api/client/keep-services-not-accessible.json @@ -0,0 +1,9 @@ +{ + "kind": "arvados#keepServiceList", + "etag": "", + "self_link": "", + "offset": null, + "limit": null, + "items": [], + "items_available": 0 +} \ No newline at end of file diff --git a/src/test/resources/org/arvados/client/api/client/users-create.json b/src/test/resources/org/arvados/client/api/client/users-create.json new file mode 100644 index 0000000000..87d09ab961 --- /dev/null +++ b/src/test/resources/org/arvados/client/api/client/users-create.json @@ -0,0 +1,26 @@ +{ + "href": "/users/ardev-tpzed-q6dvn7sby55up1b", + "kind": "arvados#user", + "etag": "b21emst9eu9u1wdpqcz6la583", + "uuid": "ardev-tpzed-q6dvn7sby55up1b", + "owner_uuid": "ardev-tpzed-000000000000000", + "created_at": "2017-10-30T19:42:43.324740000Z", + "modified_by_client_uuid": "ardev-ozdt8-97tzh5x96spqkay", + "modified_by_user_uuid": "ardev-tpzed-o3km4ug9jhs189j", + "modified_at": "2017-10-31T09:01:03.985749000Z", + "email": "example@email.com", + "username": "johnwayne", + "full_name": "John Wayne", + "first_name": "John", + "last_name": "Wayne", + "identity_url": "ardev-tpzed-r09t5ztf5qd3rlj", + "is_active": true, + "is_admin": null, + "is_invited": true, + "prefs": {}, + "writable_by": [ + "ardev-tpzed-000000000000000", + "ardev-tpzed-q6dvn7sby55up1b", + "ardev-j7d0g-000000000000000" + ] +} \ No newline at end of file diff --git a/src/test/resources/org/arvados/client/api/client/users-get.json b/src/test/resources/org/arvados/client/api/client/users-get.json new file mode 100644 index 0000000000..87d09ab961 --- /dev/null +++ b/src/test/resources/org/arvados/client/api/client/users-get.json @@ -0,0 +1,26 @@ +{ + "href": "/users/ardev-tpzed-q6dvn7sby55up1b", + "kind": "arvados#user", + "etag": "b21emst9eu9u1wdpqcz6la583", + "uuid": "ardev-tpzed-q6dvn7sby55up1b", + "owner_uuid": "ardev-tpzed-000000000000000", + "created_at": "2017-10-30T19:42:43.324740000Z", + "modified_by_client_uuid": "ardev-ozdt8-97tzh5x96spqkay", + "modified_by_user_uuid": "ardev-tpzed-o3km4ug9jhs189j", + "modified_at": "2017-10-31T09:01:03.985749000Z", + "email": "example@email.com", + "username": "johnwayne", + "full_name": "John Wayne", + "first_name": "John", + "last_name": "Wayne", + "identity_url": "ardev-tpzed-r09t5ztf5qd3rlj", + "is_active": true, + "is_admin": null, + "is_invited": true, + "prefs": {}, + "writable_by": [ + "ardev-tpzed-000000000000000", + "ardev-tpzed-q6dvn7sby55up1b", + "ardev-j7d0g-000000000000000" + ] +} \ No newline at end of file diff --git a/src/test/resources/org/arvados/client/api/client/users-list.json b/src/test/resources/org/arvados/client/api/client/users-list.json new file mode 100644 index 0000000000..2ff1ded00f --- /dev/null +++ b/src/test/resources/org/arvados/client/api/client/users-list.json @@ -0,0 +1,115 @@ +{ + "kind": "arvados#userList", + "etag": "", + "self_link": "", + "offset": 0, + "limit": 100, + "items": [ + { + "kind": "arvados#user", + "uuid": "ardev-tpzed-12389ux30402est", + "email": "test.user@email.com", + "first_name": "Test", + "last_name": "User", + "is_active": true + }, + { + "kind": "arvados#user", + "uuid": "ardev-tpzed-123vn7sby55up1b", + "email": "test.user1@email.com", + "first_name": "Test1", + "last_name": "User1", + "is_active": true + }, + { + "kind": "arvados#user", + "uuid": "ardev-tpzed-123g70lq1m3c6fz", + "email": "test.user2@email.com", + "first_name": "Test2", + "last_name": "User2", + "is_active": true + }, + { + "kind": "arvados#user", + "uuid": "ardev-tpzed-1233zsoudkgq92e", + "email": "test.user3@email.com", + "first_name": "Test3", + "last_name": "User3", + "is_active": true + }, + { + "kind": "arvados#user", + "uuid": "ardev-tpzed-1234xjvs0clppd3", + "email": "test.user4@email.com", + "first_name": "Test4", + "last_name": "User4", + "is_active": true + }, + { + "kind": "arvados#user", + "uuid": "ardev-tpzed-123bpggscmn6z8m", + "email": "test.user5@email.com", + "first_name": "Test5", + "last_name": "User5", + "is_active": true + }, + { + "kind": "arvados#user", + "uuid": "ardev-tpzed-1231uysivaz6ipi", + "email": "test.user6@email.com", + "first_name": "Test6", + "last_name": "User6", + "is_active": true + }, + { + "kind": "arvados#user", + "uuid": "ardev-tpzed-123b0a1wu0q6cm4", + "email": "test.user7@email.com", + "first_name": "Test7", + "last_name": "User7", + "is_active": true + }, + { + "kind": "arvados#user", + "uuid": "ardev-tpzed-123bz6n6si24t6v", + "email": "test.user8@email.com", + "first_name": "Test8", + "last_name": "User8", + "is_active": true + }, + { + "kind": "arvados#user", + "uuid": "ardev-tpzed-123lxhzifligheu", + "email": "test.user9@email.com", + "first_name": "Test9", + "last_name": "User9", + "is_active": true + }, + { + "kind": "arvados#user", + "uuid": "ardev-tpzed-123gaz31qbopewh", + "email": "test.user10@email.com", + "first_name": "Test10", + "last_name": "User10", + "is_active": true + }, + { + "kind": "arvados#user", + "uuid": "ardev-tpzed-123dmcf65z973uo", + "email": "test.user11@email.com", + "first_name": "Test11", + "last_name": "User11", + "is_active": true + }, + { + "kind": "arvados#user", + "uuid": "ardev-tpzed-1239y3lj7ybpyg8", + "email": "test.user12@email.com", + "first_name": "Test12", + "last_name": "User12", + "is_active": true + } + + ], + "items_available": 13 +} \ No newline at end of file diff --git a/src/test/resources/org/arvados/client/api/client/users-system.json b/src/test/resources/org/arvados/client/api/client/users-system.json new file mode 100644 index 0000000000..38441c588d --- /dev/null +++ b/src/test/resources/org/arvados/client/api/client/users-system.json @@ -0,0 +1,24 @@ +{ + "href": "/users/ardev-tpzed-000000000000000", + "kind": "arvados#user", + "etag": "2ehmra38iwfuexvz1cjno5xua", + "uuid": "ardev-tpzed-000000000000000", + "owner_uuid": "ardev-tpzed-000000000000000", + "created_at": "2016-10-19T07:48:04.838534000Z", + "modified_by_client_uuid": null, + "modified_by_user_uuid": "ardev-tpzed-000000000000000", + "modified_at": "2016-10-19T07:48:04.833164000Z", + "email": "root", + "username": null, + "full_name": "root", + "first_name": "root", + "last_name": "", + "identity_url": null, + "is_active": true, + "is_admin": true, + "is_invited": true, + "prefs": {}, + "writable_by": [ + "ardev-tpzed-000000000000000" + ] +} \ No newline at end of file diff --git a/src/test/resources/selfsigned.keystore.jks b/src/test/resources/selfsigned.keystore.jks new file mode 100644 index 0000000000..86b126af2e Binary files /dev/null and b/src/test/resources/selfsigned.keystore.jks differ