diff --git a/.gitignore b/.gitignore
index 3570d158e06625538be1695804ca4041226cdd61..1336c5cc2d0089ba4c0700b1bd960c570c9b405b 100644
--- a/.gitignore
+++ b/.gitignore
@@ -7,3 +7,6 @@ dockerfiles/commul-customization/init-dspace-dbs.sh
kubernetes/echo-server.yaml
kubernetes/maintenance-pod.yaml
kubernetes/test-postgres-deployment.yaml
+kubernetes/secrets*
+clarin-dev_eurac_edu.key
+clarin_eurac_edu.key
diff --git a/README.md b/README.md
index 494eb5b782bba0c865a74d6a838bffe59549f31f..9f63cd85d92fa62897f4c52ff904eeac393bc766 100644
--- a/README.md
+++ b/README.md
@@ -8,90 +8,60 @@ You need a Linux server (we tested on Ubuntu 16.04 and CentOS 7.3) with a recent
None of the files contain passwords, instead passwords and other confidential information are provided to the containers using environment variables. In Kubernetes you can use [secrets](https://kubernetes.io/docs/concepts/configuration/secret/) for this.
-# Exemplary workflow using Docker Compose
-## get Dockerfiles
-```
-git clone https://gitlab.inf.unibz.it/commul/docker/clarin-dspace/
-cd clarin-dspace
-```
+# Kubernetes
-## get my versions of dist files
-```
-cp password_mod.sh.dist password_mod.sh
-cp commul-customization/init-dspace-dbs.sh.dist commul-customization/init-dspace-dbs.sh
-cp commul-customization/local.properties.dist commul-customization/local.properties
-```
+## Prerequisites
-## change passwords
-```
-vi password_mod.sh
-chmod +x password_mod.sh
-./password_mod.sh
-```
+* Kubernetes cluster
+* Ceph filesystem
+* GitLab including docker registry
+* Kubernetes credentials
+* a working [kubectl] (https://kubernetes.io/docs/tasks/tools/install-kubectl/, make sure the kubectl and your Kubernetes cluster are compatible versions)
-## make sure the certificate and key are there
-```
-cp -r /tmp/certs ./commul-customization/
-```
+## Setup
-## build the images
-```
-docker-compose up -d --build
-```
+All sensitive information is stored in Kubernetes secrets. If you want to use the Kubernetes yaml files as is, make sure that you create those secrets with the same secret and key names.
-## enter the DSpace container
-```
-docker exec -ti clarindspace_dspace_1 bash
-```
-## deploy DSpace
-```
-make new_deploy
-```
+## Workflow
-## copy over modified aai_config.js
-```
-cp /tmp/aai_config.js /opt/lindat-dspace/installation/webapps/xmlui/themes/UFAL/lib/js/
-```
+### Building docker images
-## create dspace admin as tomcat8, so that the log files have the right owner
-```
-su -s /bin/sh tomcat8
-/opt/lindat-dspace/installation/bin/dspace create-administrator
-```
+Kubernetes is pulling images from a registry, so you need to first build the docker files locally and push them to a registry. We are using the registry feature of GitLab. Make sure to set up your local docker so that it can upload images into the registry.
+
+We try to keep the version numbers of all images in sync and use the script [release.sh](dockerfiles/release.sh) for this. You will need to edit this script and exchange the registry URL with your own. The script takes the new version number as its argument and builds all docker images and pushes them into our registry.
-## start the dspace webapp
```
-cd /opt/repository/sources/dspace/utilities/project_helpers/scripts
-/etc/init.d/tomcat8 start
+cd dockerfiles
+./release 1.2.3
```
-# Kubernetes
-
+By default this is pushing images into the staging branch of the registry. If you want to build a production image, you need to provide the extra argument `production`.
-## Prerequisites
-
-* Kubernetes cluster
-* Kubernetes credentials
-* a working [kubectl] (https://kubernetes.io/docs/tasks/tools/install-kubectl/)
-
-# Workflow
-
-## Building docker images
-Kubernetes is pulling images from a registry, so you need to first build the docker files. We try to keep the version numbers of all images in sync and use the script [release.sh](dockerfiles/release.sh) for this. It takes the new version number as its argument and builds all docker images and pushes them into our registry. You will need to edit this script and exchange the registry URL with your own.
+```
+cd dockerfiles
+./release 1.2.3 production
+```
-## deploying to Kubernetes
+### deploying to Kubernetes
-After pushing all images to your container registry you can ask Kubernetes to pull and deploy them. If necessary, you might need to create a [personal token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) for logging into the GitLab Registry and store it as a secret that you then reference in the Kubernetes yaml files.
+After pushing all images to your container registry you can ask Kubernetes to pull and deploy them. If necessary, you might need to create a [personal token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) for logging into the GitLab Registry and store it as a secret that you then reference in the Kubernetes yaml files. You need to edit the yaml files to suit your setup, especially you need to edit:
+* The image registry URL
+* The ceph setup
+* The name of your Kubernetes namespace
-There is a handy script that calls all necessary kubectl commands one after another called [start-kube.sh](kubernetes/start-kube.sh), for bringing it all down again use [stop-kube.sh](kubernetes/stop-kube.sh) instead. For this to work, make sure that you have the correct images referenced in the deployment yaml files.
+These edits you only have to do once, but for each new deploy you have to edit the version numbers of the images in the deployment yaml files. This can be done with a simple sed:
```
-image: gitlab.inf.unibz.it:4567/commul/docker/clarin-dspace/staging/dspace:1.2-RC1
+sed -i 's/1.2-RC1/1.2/' ../kubernetes/*deploy*
```
+There is a handy script that calls all necessary kubectl commands one after another called [start-kube.sh](kubernetes/start-kube.sh), for bringing it all down again use [stop-kube.sh](kubernetes/stop-kube.sh) instead. By default this doesn't touch the persistent volume claim. For a full redeploy that forgets all history you have to call the kubectl command manually:
+```
+kubectl delete -f pgdata-persistentvolumeclaim.yaml
+```
@@ -109,145 +79,61 @@ secrets
d
+# Exemplary workflow using Docker Compose (this might be outdated)
-### Use existing Docker image
-```console
-docker image ls -a gitlab.scientificnet.org:4567/commul/dspace/*
-docker pull gitlab.scientificnet.org:4567/commul/dspace/<my_image>:<my tag>
+## get Dockerfiles
```
-
-### Build images
-* build your docker image from `/clarin-dspace-docker` directory
-* push to `https://gitlab.scientificnet.org/commul/dspace/container_registry`
-
-```console
-docker build -t gitlab.scientificnet.org:4567/commul/dspace/<my_image>:<my tag> -f Dockerfile.XX .
-docker push gitlab.scientificnet.org:4567/commul/dspace/<my_image>:<my tag>
+git clone https://gitlab.inf.unibz.it/commul/docker/clarin-dspace/
+cd clarin-dspace
```
-# Installation
-This step in a "workspace preparation" and is not directly related with dspace.
-
-Provide gitlab & RBD credentials
-
-```console
-$ kubectl -n dspace-dev create -f gitlab-scientificnet-org-registry.yaml
+## get my versions of dist files
```
-```console
-$ kubectl -n dspace-dev create -f rbd-secret.yaml
+cp password_mod.sh.dist password_mod.sh
+cp commul-customization/init-dspace-dbs.sh.dist commul-customization/init-dspace-dbs.sh
+cp commul-customization/local.properties.dist commul-customization/local.properties
```
-## PostgreSQL
-To install PostgreSQL, we need some passwords, so we need Kubernetes secrets
-```console
-$ kubectl -n dspace-dev create -f postgresql-secret.yaml
-secret "dspace-postgresql" created
+## change passwords
```
-
-```console
-$ kubectl -n dspace-dev create -f dspace-secrets.yaml
-secret "dspace-secrets" created
+vi password_mod.sh
+chmod +x password_mod.sh
+./password_mod.sh
```
-```console
-$ kubectl -n dspace-dev get secrets
-NAME TYPE DATA AGE
-ceph-rbd-secret kubernetes.io/rbd 1 14m
-default-token-c1pd4 kubernetes.io/service-account-token 3 4d
-dspace-postgresql Opaque 1 1h
-dspace-secrets Opaque 1 1m
-gitlab-scientificnet-org-registry kubernetes.io/dockerconfigjson 1 46m
+## make sure the certificate and key are there
```
-
-
-
-Next, create a RBD Claim
-```console
-$ kubectl -n dspace-dev create -f postgresql-pvc.yaml
-persistentvolumeclaim "dspace-postgresql" created
-
-$ kubectl -n dspace-dev get pvc
-NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE
-dspace-postgresql Bound pvc-610c358d-a518-11e7-b21c-b60483de6a40 8Gi RWO auto 2s
+cp -r /tmp/certs ./commul-customization/
```
-Ok, now we are ready to deploy our PostgreSQL!
-```console
-$ kubectl -n dspace-dev create -f postgresql.yaml
-
-$ kubectl -n dspace-dev get po
-NAME READY STATUS RESTARTS AGE
-dspace-postgresql-1806947503-cj077 0/1 ContainerCreating 0 2s
-
-$ kubectl -n dspace-dev get po
-NAME READY STATUS RESTARTS AGE
-dspace-postgresql-1806947503-qw8q4 0/1 Running 0 9s
-
-$ kubectl -n dspace-dev get po
-NAME READY STATUS RESTARTS AGE
-dspace-postgresql-1806947503-qw8q4 1/1 Running 0 46s
+## build the images
+```
+docker-compose up -d --build
```
-We have now launched the pod but we have to wait until the pod is up before accessing it
-```console
-$ kubectl -n dspace-dev exec -it dspace-postgresql-1806947503-qw8q4 bash
-root@dspace-postgresql-1806947503-qw8q4:/#
+## enter the DSpace container
+```
+docker exec -ti clarindspace_dspace_1 bash
```
-Also, we can also test our PostgreSQL installation with a port forward
-```console
-$ kubectl -n dspace-dev port-forward dspace-postgresql-1806947503-qw8q4 5432:5432
+## deploy DSpace
+```
+make new_deploy
```
-now we are able to connect to localhost:5432
-
-
-```console
-$ kubectl -n dspace-dev create -f postgresql-service.yaml
-service "dspace-postgresql" created
+## copy over modified aai_config.js
```
-```console
-$ kubectl -n dspace-dev exec -it servlet env | grep POSTGRESQL
-DSPACE_POSTGRESQL_SERVICE_PORT_POSTGRESQL=5432
-DSPACE_POSTGRESQL_PORT_5432_TCP_ADDR=10.13.0.173
-DSPACE_POSTGRESQL_SERVICE_HOST=10.13.0.173
-DSPACE_POSTGRESQL_PORT_5432_TCP_PROTO=tcp
-DSPACE_POSTGRESQL_PORT_5432_TCP_PORT=5432
-DSPACE_POSTGRESQL_SERVICE_PORT=5432
-DSPACE_POSTGRESQL_PORT=tcp://10.13.0.173:5432
-DSPACE_POSTGRESQL_PORT_5432_TCP=tcp://10.13.0.173:5432
+cp /tmp/aai_config.js /opt/lindat-dspace/installation/webapps/xmlui/themes/UFAL/lib/js/
```
-## DSpace
-```console
-$ kubectl -n dspace-dev create -f servlet.yaml
-pod "servlet" created
+## create dspace admin as tomcat8, so that the log files have the right owner
```
-```console
-$ kubectl -n dspace-dev get po
-NAME READY STATUS RESTARTS AGE
-dspace-postgresql-1806947503-qw8q4 1/1 Running 0 4d
-servlet 1/1 Running 0 34s
+su -s /bin/sh tomcat8
+/opt/lindat-dspace/installation/bin/dspace create-administrator
```
-```console
-$ kubectl -n dspace-dev logs servlet
-[Wed Oct 4 12:24:25 UTC 2017] Starting
-[Wed Oct 4 12:24:25 UTC 2017] Cloning lindat-dspace repository...skip! /opt/repository/sources/dspace already exists
-[Wed Oct 4 12:24:25 UTC 2017] Creating workspace...skip! /opt/repository/workspace already exists
-[Wed Oct 4 12:24:25 UTC 2017] Copy makefile...skip! /opt/repository/workspace/config/variable.makefile already exists
-[Wed Oct 4 12:24:25 UTC 2017] Copy local.properties...skip! /opt/repository/workspace/sources/local.properties already exists
-[Wed Oct 4 12:24:25 UTC 2017] Compilation and Deployment of DSpace...already done
-already done
-already done
-[Wed Oct 4 12:24:25 UTC 2017] Copy aai_config.js...skip! /opt/lindat-dspace/installation/webapps/xmlui/themes/UFAL/lib/js/aai_config.js already exists
-[Wed Oct 4 12:24:25 UTC 2017] Create dspace admin...already done
-[Wed Oct 4 12:24:25 UTC 2017] Starting Tomcat...Starting Tomcat 8...
-Tomcat started.
+
+## start the dspace webapp
```
-```console
-$ kubectl -n dspace-dev port-forward servlet 8080:8080
-Forwarding from 127.0.0.1:8080 -> 8080
-Forwarding from [::1]:8080 -> 8080
+cd /opt/repository/sources/dspace/utilities/project_helpers/scripts
+/etc/init.d/tomcat8 start
```
-
-connect to http://localhost:8080/repository/xmlui/#
diff --git a/dockerfiles/Dockerfile.dspace b/dockerfiles/Dockerfile.dspace
index 8589dffc777b98777a5e0d9dc859e151a93e67b8..0167fcfe9f8f8aeb0fd5375094235f74f6da5543 100644
--- a/dockerfiles/Dockerfile.dspace
+++ b/dockerfiles/Dockerfile.dspace
@@ -68,6 +68,9 @@ RUN ./setup.sh /opt/repository/workspace
COPY commul-customization/variable.makefile /opt/repository/workspace/config/variable.makefile
# COPY commul-customization/dspace.cfg /opt/repository/sources/dspace/dspace/config/dspace.cfg
+# install python modules: magic dateutil, lxml
+RUN apt-get update && apt-get install -y python-magic python-dateutil python-lxml
+
# copy over configs and startup scripts
COPY commul-customization/start_stack.sh /opt/repository/workspace/scripts/
COPY commul-customization/stop_stack.sh /opt/repository/workspace/scripts/
@@ -83,7 +86,6 @@ RUN make new_deploy
# DEBUG: to be able to access tomcat manager through browser
# COPY commul-customization/tomcat-users.xml /opt/tomcat8/conf/
-RUN perl -pi -e 's/#xmlui\.session\.ipcheck = true/xmlui.session.ipcheck = false/' /opt/lindat-dspace/installation/config/dspace.cfg
RUN perl -pi -e 's/#useProxies = true/useProxies = true/' /opt/lindat-dspace/installation/config/dspace.cfg
# set up handle server
diff --git a/dockerfiles/commul-customization/default-ssl b/dockerfiles/commul-customization/default-ssl
index bfe9defb2f3516fe16f1206a5d9e3b482ff38058..25d34260315ee26fa1f618741d49017fc4592e64 100644
--- a/dockerfiles/commul-customization/default-ssl
+++ b/dockerfiles/commul-customization/default-ssl
@@ -78,9 +78,7 @@ server {
# }
location / {
- index index.php index.html index.htm;
-
-# try_files $uri $uri/ /index.php?$args;
+ index index.html index.htm;
if ($HTTP_X_SCHEME = "http") {
rewrite ^ https://$http_host$request_uri permanent;
@@ -148,7 +146,7 @@ server {
}
# remove the robots and favicon from the logs
-# location ~ /robots.txt$ { access_log off; log_not_found off; }
+ location ~ /robots.txt$ { access_log off; }
location ~ ^/favicon.ico$ { access_log off; log_not_found off; }
# this prevents hidden files (beginning with a period) from being served
diff --git a/dockerfiles/commul-customization/entrypoint.sh b/dockerfiles/commul-customization/entrypoint.sh
index e80223ccbb23b13f3d2ebba2c723219751c57886..c7890a495c37630858d37d45fc766d2d9890e2f1 100644
--- a/dockerfiles/commul-customization/entrypoint.sh
+++ b/dockerfiles/commul-customization/entrypoint.sh
@@ -19,7 +19,7 @@ make init_statistics
make update_discovery
make update_oai
/opt/lindat-dspace/installation/bin/dspace generate-sitemaps
-/opt/tomcat8/bin/shutdown.sh
+/opt/tomcat8/bin/shutdown.sh -force
# start tomcat
/opt/tomcat8/bin/catalina.sh run
diff --git a/dockerfiles/commul-customization/local.properties b/dockerfiles/commul-customization/local.properties
index a28a1d86552048ce7a41b40f526c2d555c69f859..37c34c187c5b260891a0d6d3ad10909872e9f80b 100644
--- a/dockerfiles/commul-customization/local.properties
+++ b/dockerfiles/commul-customization/local.properties
@@ -145,7 +145,7 @@ mail.registration.notify = ${mail.admin}
mail.admin = ${mail.feedback.recipient}
# Email where errors will be sent (from specific classes)
-lr.errors.email=
+lr.errors.email= appliedlinguisticsdevs@eurac.edu
# solr server address that your instance is using.
solr.server = http://localhost:8080/repository/solr
diff --git a/kubernetes/dspace-deployment.yaml b/kubernetes/dspace-deployment.yaml
index aa92bcc754f1d8806b826ee898a7c5cceaf9a33b..24ec000a7d0fa9bc6ea72fe24127ad53d20f177b 100644
--- a/kubernetes/dspace-deployment.yaml
+++ b/kubernetes/dspace-deployment.yaml
@@ -50,7 +50,7 @@ spec:
secretKeyRef:
key: mail.pass
name: dspace-secrets
- image: gitlab.inf.unibz.it:4567/commul/docker/clarin-dspace/staging/dspace:1.2
+ image: gitlab.inf.unibz.it:4567/commul/docker/clarin-dspace/staging/dspace:1.2.2b
name: clarin-dspace
imagePullPolicy: Always
ports:
diff --git a/kubernetes/dspace-ingress.yaml b/kubernetes/dspace-ingress.yaml
index 4fecf81f1b251decf1887f6a86185bfbadd2fa81..f89c630ecacb00b02597a298050100dbab50927b 100644
--- a/kubernetes/dspace-ingress.yaml
+++ b/kubernetes/dspace-ingress.yaml
@@ -2,20 +2,26 @@ apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
- ingress.kubernetes.io/add-base-url: "true"
- ingress.kubernetes.io/ssl-passthrough: "true"
+ # OAI-PMH uses HTTP transport, ie. we can't redirect *all* traffic to https.
+ # However, we need to make sure that 'our internal' nginx redirects http:// -> https://
+ # (except for /repository/oai).
ingress.kubernetes.io/ssl-redirect: "false"
+ # But just to be sure, we are /also/ redirecting here.
+ ingress.kubernetes.configuration-snippet: |
+ if ($scheme = "http") {
+ set $rewrite_https 1;
+ }
+ if ($request_uri ~ "^/repository/oai") {
+ set $rewrite_https 0;
+ }
+ if ($rewrite_https = 1) {
+ return 301 https://$server_name$request_uri;
+ }
+ ingress.kubernetes.io/secure-backends: "true"
name: dspace-ingress
namespace: dspace-dev
spec:
rules:
- - host: clarin-dev.eurac.edu
- http:
- paths:
- - backend:
- serviceName: nginx
- servicePort: 80
- path: /repository/oai
- host: clarin-dev.eurac.edu
http:
paths:
@@ -26,3 +32,4 @@ spec:
tls:
- hosts:
- clarin-dev.eurac.edu
+ secretName: host-tls-cert
diff --git a/kubernetes/nginx-deployment.yaml b/kubernetes/nginx-deployment.yaml
index ed7c50741c8981b77f9eb8439b3eb7c11e00beab..22feea2425f96b2bd0b3606326c5b74ce01b6740 100644
--- a/kubernetes/nginx-deployment.yaml
+++ b/kubernetes/nginx-deployment.yaml
@@ -20,7 +20,7 @@ spec:
io.kompose.service: nginx
spec:
containers:
- - image: gitlab.inf.unibz.it:4567/commul/docker/clarin-dspace/staging/nginx:1.2
+ - image: gitlab.inf.unibz.it:4567/commul/docker/clarin-dspace/staging/nginx:1.2.2b
imagePullPolicy: Always
name: clarin-nginx
ports:
diff --git a/kubernetes/postgres-deployment.yaml b/kubernetes/postgres-deployment.yaml
index ba880bf880170d8395f427cb86ef1a4d6f71352e..f65661bedb5c9292ef41aff8264290b63a24484d 100644
--- a/kubernetes/postgres-deployment.yaml
+++ b/kubernetes/postgres-deployment.yaml
@@ -48,7 +48,7 @@ spec:
secretKeyRef:
key: dspace.utils.name
name: dspace-secrets
- image: gitlab.inf.unibz.it:4567/commul/docker/clarin-dspace/staging/postgres:1.2
+ image: gitlab.inf.unibz.it:4567/commul/docker/clarin-dspace/staging/postgres:1.2.2b
imagePullPolicy: Always
name: clarin-postgres
resources: {}