How to access object storage using OpenStack CLI on WEkEO
Cloud computing offers the ability to handle large chunks of data, directly on the remote server. OpenStack module Swift was created expressly to enable access to unstructured data that can grow without bounds, with the following design goals in mind :
durability,
scalability,
concurrency across the entire data set,
all while keeping the API simple.
Swift is installed as an independent module but on the syntax level, it is used through the parameters of openstack command.
What We Are Going To Cover
How to install Swift
How to connect Swift to OpenStack cloud
Basic openstack operations with containers
Basic openstack operations with objects
Prerequisites
No. 1 Hosting
You need a WEkEO hosting account, available at https://cf-admin.wekeo2.eu/login. If you want to follow up with articles about object storage on Horizon, you will this link too: https://horizon.cloudferro.com.
No. 2 Install or activate openstack command
To be able to connect to the cloud, openstack command must be operational. If not installed already, use article How to install OpenStackClient for Linux on WEkEO
No. 3 Authenticate to OpenStack using application credentials
Then you have to authenticate your account to the cloud. The usual way is to activate openstack command using an RC file for on- or two-factor authentication. That will not work in case of Swift module. It is authenticated with application credentials, as explained in article
How to generate or use Application Credentials via CLI on WEkEO.
No. 4 Familiarity with object storage on WEkEO OpenStack
This article is explaining the basics, using the Horizon interface:
How to use Object Storage on WEkEO.
Swift can be understood as the CLI tool for accessing object storage under OpenStack.
No. 5 Python installed
The following articles contain sections on how to install Python:
On Ubuntu, you can also use virtualenvwrapper: How to install Python virtualenv or virtualenvwrapper on WEkEO
How to install Swift
Here is how to install Swift on Ubuntu 22.04:
sudo apt-get install python3-pip # if not already installed, install pip on Ubuntu 22.04
pip install python-swiftclient # install python openstack client
You would then use Prerequisite No. 3 to authenticate Swift to WEkEO.
Basic openstack commands for object data
OpenStack allows large quantities of data to be placed into so-called
“containers”, which are similar to folders on a desktop computer. Containers can further contain
“files”, which are here called “objects”.
Therefore, the commands to work with data containers and objects look like this:
openstack container and
openstack object
Use --help option to show the parameters, for instance:
openstack container --help
Basic openstack CLI operations with containers
Container name cannot contain spaces and special characters.
Correct
Incorrect
backupvelero
backup velero
backup-velero
Create container
openstack container command can create, list and delete containers:
openstack container create backup09
openstack container list
openstack container delete backup09
Here is how you can
create a new container, called backup09,
list it and thus show that it has been created, then
delete it and again
use list,
this time to show that it has been deleted:
Basic openstack CLI operations with objects
Upload an Object
Similarly, it is possible to create, save, list and delete objects (which can think of as “files” in the usual sense of the word). This would be a general form of one such command:
openstack object create <container_name> <object_name>
For instance:
Let us list the contents of the container and verify that file test.txt is there:
To save file locally, add --file to denote the name of the local file:
openstack object save backup09 test.txt --file test32.txt
To delete, use:
openstack object delete <container_name> <object_name>
Set Metadata
Metadata can be used to tag and categorize objects so that later on, large amounts of data can be categorized. If you are creating some kind of media storage system, you can tag images, videos, pure text and PDF files differently. You can also add creation date as a meta tag.
This is a general command to add metadata to the object:
openstack object set --property <key=value> <container_name> <object_name>
And here is a more concrete example of
uploading an image to the container and
categorizing it as an image via a meta tag:
# First create a new container, solar_system, to contain images of solar system
openstack container create solar_system
# Upload an image and add metadata to categorize it
openstack object create solar_system image1.jpg
# Tag it as an image and with the date of creation
openstack object set \
--property type=image \
--property created=2024-06-12 solar_system image1.jpg
If you wanted to keep track of versions, you could tag the image like this:
openstack object set \
--property version=2 \
--property modified=2024-06-12 solar_system image1.jpg
What To Do Next
You can combine standard bash commands with basic object and container commands to
perform backups,
copy and/or migrate data to other containers,
search using metadata you entered earlier
and so on.
This type of bash programming is, however, out of scope of this article.
Also of interest: