GCP RUN Celery
Celery Worker Service with GCP Cloud Run Jobs
Deprecated, because GCP Jobs is used as a scalable system instead.
Build the container image
Tag the container image
Push the container image to Google Container Registry
gcloud auth login
docker push us-west2-docker.pkg.dev/gcp-pathology-poc1/image-transform-svc/celery:1.0.0
Deploy to GCP Cloud Run
gcloud config set project gcp-pathology-poc1 && \ gcloud beta run jobs create imagetransformsvc-celery \ --image us-west2-docker.pkg.dev/gcp-pathology-poc1/image-transform-svc/celery:1.0.0 \ --region=us-west2 \ --memory=4Gi --cpu=2 \ --max-retries=3 --parallelism=10 \ --task-timeout=3600s \ --network default --subnet default \ --add-volume name=temp_files,type=cloud-storage,bucket=gcp-pathology-poc1-image-transformation-transformation-temp \ --add-volume-mount volume=temp_files,mount-path=/temp_files \ --set-env-vars="ITS_URL=https://dicom.poc1.gestaltcloud.com/public-proxy01/projects/gcp-pathology-poc1/locations/us-west2/datasets/dicom-pathology/dicomStores/slide-dicom-store-public/dicomWeb" \ --set-env-vars="ITS_HANDLER=Celery" \ --set-env-vars="ITS_KEEP_FILES_PATH=/temp_files" \ --set-env-vars="ITS_SFTPFileInputPath=/temp_files/Input" \ --set-env-vars="ITS_SFTPFileOutputPath=/temp_files/Output" \ --set-env-vars="ITS_MODE=default" \ --set-env-vars="ITS_CELERY_BROKER_URL=redis://10.158.57.91:6379/9" \ --set-env-vars="ITS_CELERY_RESULT_BACKEND=redis://10.158.57.91:6379/9" && \ gcloud beta run jobs execute imagetransformsvc-celery \ --region us-west2
Create a scheduler job that runs every minute
Note code below doesn't work; currently manually restarting the job gcloud scheduler jobs create http imagetransformsvc-celery-trigger \ --schedule="* * * * *" \ --location=us-west2 \ --uri="https://us-west2-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/gcp-pathology-poc1/jobs/imagetransformsvc-celery:run" \ --http-method=POST \ --oauth-service-account-email=transformation-k8s-service@gcp-pathology-poc1.iam.gserviceaccount.com