When running Workflows, it is very common to have steps that generate or consume artifacts. Often, the output artifacts of one task may be used as input artifacts to a subsequent task.
Onepanel's Workflows support s3, git and http artifacts.
You can also attach git repository (i.e Github) as an input artifact as follows.
inputs:
artifacts:
-git:
repo: https://github.com/onepanelio/Mask_RCNN.git
revision:"no-boto"
name: src
path: /mnt/src
Here, we specified Github repository along with branch (no-boto). path specifies where to mount this repository.
You can also use private repository with Workflows. For that, you first need to create personal access token from Github. Then, go to Settings and add following two environment variables. GITHUB_USERNAME and GITHUB_PASSWORD. For GITHUB_PASSWORD, you should be using your personal access token. Once this is done, you can modify above yaml section as follows to use private repository.
inputs:
artifacts:
-git:
repo: https://github.com/onepanelio/Mask_RCNN.git
revision:"no-boto"
usernameSecret:
name: onepanel-default-env
key: GITHUB_USERNAME
passwordSecret:
name: onepanel-default-env
key: GITHUB_PASSWORD
name: src
path: /mnt/src
note
The directories in path should exist before you can clone into them. For example, if you mount a volume at /mnt/src you cannot clone into /mnt/src/source.
The directories in path should exist before you can download into them. For example, if you mount a volume at /mnt/data you cannot download the file into /mnt/data/dataset.
You can also mark both input and output artifacts as optional by setting optional to true. In this case, if they don't exist, the Workflow will not throw an error:
...
inputs:
artifacts:
# Download files from S3 prefix random/input into local folder /tmp/input/
# # This downloads from the default S3 artifact repository for this namespace
-name: input
path: /tmp/input/
optional:true # Don't throw an error if the file doesn't exist in the S3 location