Where are all Piccolo AI server data stored locally?

Technical Discussion
1

I am running a Piccolo AI server locally following the procedure explained in the README. When I run docker-compose up, the server seems to be operational. Where is all the data ingested by the server stored locally?

2

Before running docker-compose up, you can set the DOCKER_HOST_SML_DATA_DIR environment variable to the local destination where you want to store the server’s internal data.

All imported data, query and pipeline caches, and intermediate files related to different projects are stored in the designated folder, which will have the following structure:

server_data/
├── capture/
├── codegen/
├── custom_transforms/
├── datafile/
├── datafiles/
├── featurefile/
├── model_store/
├── pipelinecache/
└── querydata/

Here is how to set the DOCKER_HOST_SML_DATA_DIR environment variable on Linux, macOS, and Windows systems:

Linux and macOS

  1. Open a terminal.

  2. Use the export command to set the environment variable. Replace /path/to/your/data with your desired local destination.

    export DOCKER_HOST_SML_DATA_DIR=/path/to/your/data

  3. To make this change permanent, add the above command to your shell’s configuration file (e.g., ~/.bashrc or ~/.zshrc).

    echo 'export DOCKER_HOST_SML_DATA_DIR=/path/to/your/data' >> ~/.bashrc
# or for zsh
echo 'export DOCKER_HOST_SML_DATA_DIR=/path/to/your/data' >> ~/.zshrc

  4. Reload the shell configuration file to apply the changes immediately.

    source ~/.bashrc
# or for zsh
source ~/.zshrc

Windows

  1. Open a Command Prompt or PowerShell as Administrator.

  2. Use the set command to set the environment variable for the current session. Replace C:\path\to\your\data with your desired local destination.

    set DOCKER_HOST_SML_DATA_DIR=C:\path\to\your\data

  3. To set the environment variable permanently, use the setx command.

    setx DOCKER_HOST_SML_DATA_DIR C:\path\to\your\data

Alternatively, you can set the environment variable through the System Properties:

  1. Open the Start Menu and search for “Environment Variables.”
  2. Click on “Edit the system environment variables.”
  3. In the System Properties window, click on the “Environment Variables” button.
  4. In the Environment Variables window, click on “New” under the “System variables” section.
  5. Enter DOCKER_HOST_SML_DATA_DIR as the variable name and C:\path\to\your\data as the variable value.
  6. Click “OK” to save and apply the changes.

Running the Server

After setting the DOCKER_HOST_SML_DATA_DIR environment variable, you can run the server using the following command:

docker-compose up