How to configure the persistent storage path?

Support
1

I’m running the marqoai/marqo:1.5.0 and marqoai/marqo-os:0.0.3-arm containers on MacOS.

I would like to attach a docker volume and use the mount for marqo’s persistent storage. Is there anyway to change marqo’s storage path?

2

Hi @joe-maley ! This article outlines how Docker on MacOS handles persistent data.

In order to use a volume with a MacOS arm64 setup you can do the following:

  1. create the docker volume:
docker volume create marqo-os-data
  1. Run marqo-os for arm64, mounting the docker image:
docker rm -f marqo-os; docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" -v marqo-os-data:/usr/share/opensearch/data marqoai/marqo-os:0.0.3-arm
  1. Follow the regular arm64 MacOS docker run marqo instructions, found in step 2 of the README.
3

I am using docker-compose and want data persistency for marqo db. I have created a volume named “marqo_data” and mapped it to the path “/opt/vespa/” but each time my service re-starts data is lost. I am using it on ubuntu and not mac

marqo:
image: marqoai/marqo:1.5.1
ports:
- “8882:8882”
container_name: marqoDB
privileged: true
extra_hosts:
- “host.docker.internal:host-gateway”
environment:
MARQO_MAX_INDEX_FIELDS: 400
MARQO_MAX_DOC_BYTES: 200000
MARQO_MAX_RETRIEVABLE_DOCS: 600
MARQO_MAX_NUMBER_OF_REPLICAS: 2
MARQO_MODELS_TO_PRELOAD: ‘[“flax-sentence-embeddings/all_datasets_v4_mpnet-base”]’
volumes:
- ./marqo_data:/opt/vespa/

4

Hello Ankush, thanks for raising this issue with us. I noticed that you are using marqo version 1.5.1. Marqo v1.x is based on opensearch, that’s why your volume mapping to vespa does not work.

Marqo v2.x uses Vespa, our recommendation is to use the latest version of marqo for your testing. (you can find it here: https://hub.docker.com/r/marqoai/marqo/tags). You can follow this doc to map docker volumes: Transferring State - Marqo docs.

Please let me know if you encounter any other issues. Thanks.

2 Likes
5

I can’t update to 2.X as of now, is there a way in which I could have data persistency in 1.5.1

6

Hello @ankush-maherwal , could you please follow this instruction to see if it helps? Transferring State - Marqo docs. Thanks.