CHANGELOG.md

+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Added
+### Changed
+### Removed
+### Fixed
+
+## [1.2] - 2018-08-27
+
+### Added
+
+- Added aa-statistics script to help aggregate statistics about IdP/SP Interoperability for LINDAT (see https://trac.clarin.eu/wiki/ServiceProviderFederation/LoginTest)
+- Added the shib testing page as required by CLARIN (#10)
+- Adapted item view to also show dc.relation.isbasedon if it is available (in [clarin-dspace repository](https://github.com/commul/clarin-dspace/commit/5f8f50457b48d966c169cd29268e7529dee4622f))
+
+### Changed
+
+- Made DSpace user configurable through environment variable (#45)
+- Docker Image naming convention has been changed
+- Upgraded tomcat to version 8.0.52
+- Changed links to licenses to https (in [clarin-dspace repository](https://github.com/commul/clarin-dspace/commit/b41edf205520afe373e602940a2263b394c858ab))
+
+### Removed
+
+- Removed link to Anonymous Privacy Policy from Shibboleth Metadata
+
+### Fixed
+
+- Shibboleth build script now uses fixed version numbers instead of latest
+- Super short session timeout will no longer happen (#44)
+
+## [1.1] - 2018-06-15
+
+### Added
+
+- Added a link to the issue tracker to FAQ
+
+### Changed
+
+- Mails are now sent with a no-reply reply-to address to make it more obvious that you cannot reply to them.
+- Exchanged example item in various documentation pages with PAISÁ.
+- Changed Dockerfile to pre-fetch Maven dependencies. This should reduce build time for most cases.
+
+### Fixed
+
+- Citation is now correctly showing authors and title in addition to the PID.
+
+
+## 1.0 - 2018-05-25
+
+This is the first release. It all starts here.
+
+[Unreleased]:https://gitlab.inf.unibz.it/commul/docker/clarin-dspace/compare/v1.1...HEAD
+[1.1]: https://gitlab.inf.unibz.it/commul/docker/clarin-dspace/compare/v1.0...v1.1
+[1.2]: https://gitlab.inf.unibz.it/commul/docker/clarin-dspace/compare/v1.1...v1.2

CONTRIBUTORS.md

+# CONTRIBUTORS
+
+* **[Alexander König](https://gitlab.inf.unibz.it/Alexander.Koenig)**
+
+* **[egon w. stemle](https://gitlab.inf.unibz.it/egon.stemle)**

README.md

 # dockerized CLARIN DSpace
 
-This is meant to provide an easy way to install [CLARIN DSpace](https://github.com/ufal/clarin-dspace) by providing a Docker Compose setup that automatizes the installation as far as possible. Additionally all files that need to be customized or added are also part of this repository.
+This is meant to provide an easy way to install [CLARIN DSpace](https://github.com/ufal/clarin-dspace) by providing a Docker setup that automatizes the installation as far as possible. We are running this within Kubernetes, but try to keep the Docker Compose setup also in working condition. All files that need to be customized or added are also part of this repository or in the sister projects https://github.com/commul/clarin-dspace and https://github.com/commul/lindat-common (both forked from the corresponding UFAL repositories).
 
 # How to use it
 
-You need a Linux server (we tested on Ubuntu 16.04 and CentOS 7.3) with a recent install of Docker (we tested with 17.05.0-ce). Clone this git repository onto the server. Then you need to look through all the files in commul-customization and adapt the configuration to your server (domain name, user names, etc). 
+You need a Linux server (we tested on Ubuntu 16.04 and CentOS 7.3) with a recent install of Docker (we tested with 17.05.0-ce) or a Kubernetes cluster (we are using version 1.8). Clone this git repository onto the server. Then you need to look through all the files in commul-customization and adapt the configuration to your server (domain name, user names, etc).
 
-None of the files contain passwords, but all files which need passwords in them are in the repository as .dist files. To insert the passwords you need to remove the dist ending and then run password_mod.sh (after you've modified it with your passwords).
+None of the files contain passwords, instead passwords and other confidential information are provided to the containers using environment variables. In Kubernetes you can use [secrets](https://kubernetes.io/docs/concepts/configuration/secret/) for this.
 
-# Exemplary workflow
+# Exemplary workflow using Docker Compose
 
 ## get Dockerfiles
 ```
-git clone https://github.com/commul/clarin-dspace-docker
-cd clarin-dspace-docker
+git clone https://gitlab.inf.unibz.it/commul/docker/clarin-dspace/
+cd clarin-dspace
 ```
 
 ## get my versions of dist files
@@ -42,7 +42,7 @@ docker-compose up -d --build
 
 ## enter the DSpace container
 ```
-docker exec -ti clarindspacedocker_dspace_1 bash
+docker exec -ti clarindspace_dspace_1 bash
 ```
 
 ## deploy DSpace
@@ -67,53 +67,49 @@ cd /opt/repository/sources/dspace/utilities/project_helpers/scripts
 /etc/init.d/tomcat8 start
 ```
 
-# DSpace directory structure
+# Kubernetes
 
-This is the directory structure 
+
+## Prerequisites
+
+* Kubernetes cluster
+* Kubernetes credentials
+* a working [kubectl] (https://kubernetes.io/docs/tasks/tools/install-kubectl/)
+
+# Workflow
+
+## Building docker images
+Kubernetes is pulling images from a registry, so you need to first build the docker files. We try to keep the version numbers of all images in sync and use the script [release.sh](dockerfiles/release.sh) for this. It takes the new version number as its argument and builds all docker images and pushes them into our registry. You will need to edit this script and exchange the registry URL with your own.
+
+## deploying to Kubernetes
+
+After pushing all images to your container registry you can ask Kubernetes to pull and deploy them. If necessary, you might need to create a [personal token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) for logging into the GitLab Registry and store it as a secret that you then reference in the Kubernetes yaml files.
+
+There is a handy script that calls all necessary kubectl commands one after another called [start-kube.sh](kubernetes/start-kube.sh), for bringing it all down again use [stop-kube.sh](kubernetes/stop-kube.sh) instead. For this to work, make sure that you have the correct images referenced in the deployment yaml files.
 
 ```
-├── clarin-dspace-docker
-│   └── commul-customization
-└── kubernetes
-    ├── helm
-    │   └── dspace
-    │       ├── charts
-    │       │   └── postgresql
-    │       │       └── templates
-    │       └── templates
-    └── yaml
+image: gitlab.inf.unibz.it:4567/commul/docker/clarin-dspace/staging/dspace:1.2-RC1
 ```
 
 
-* `/clarin-dspace-docker` - Docker files, copied from https://github.com/commul/clarin-dspace-docker @ 74d06d3 . **With this directory, you can build images**
-* `/kubernetes/helm` - Chart Directory. Refer to https://github.com/kubernetes/helm/blob/master/docs/charts.md#the-chart-file-structure :exclamation: **not working yet** :exclamation:
-* `/kubernetes/yaml` - Useful for tests, but helm is a more flexible approach **With this directory, you can deploy images in kubernetes cluster**
-# Introduction
-# Prerequisites
 
-* Kubernetes cluster
 
-## To interact with cluster
 
-* Kubernetes credentials
-* a working [kubectl] (https://kubernetes.io/docs/tasks/tools/install-kubectl/)
 
-## To interact with Helm server (tiller) -- Coming soon
 
-* Kubernetes credentials
-* a working [kubectl] (https://kubernetes.io/docs/tasks/tools/install-kubectl/)
-* a working [helm client] (https://docs.helm.sh/using_helm/#quickstart-guide)
+ceph
+
+secrets
+
+
+
+
+
+
+d
+
 
-# How to use it
-## Workflow
-First of all, check if a docker image already exists.
-If it exist, pull, otherwise, you need to build one.
-If necessary create a [personal token](https://gitlab.scientificnet.org/profile/personal_access_tokens) and login into GitLab Registry.
 
-```console
-docker login gitlab.scientificnet.org:4567
-```
- 
 ### Use existing Docker image
 ```console
 docker image ls -a gitlab.scientificnet.org:4567/commul/dspace/*
@@ -122,7 +118,7 @@ docker pull gitlab.scientificnet.org:4567/commul/dspace/<my_image>:<my tag>
 
 ### Build images
 * build your docker image from `/clarin-dspace-docker` directory
-* push to `https://gitlab.scientificnet.org/commul/dspace/container_registry` 
+* push to `https://gitlab.scientificnet.org/commul/dspace/container_registry`
 
 ```console
 docker build -t gitlab.scientificnet.org:4567/commul/dspace/<my_image>:<my tag> -f Dockerfile.XX .
@@ -144,12 +140,12 @@ $ kubectl -n dspace-dev create -f rbd-secret.yaml
 To install PostgreSQL, we need some passwords, so we need Kubernetes secrets
 
 ```console
-$ kubectl -n dspace-dev create -f postgresql-secret.yaml 
+$ kubectl -n dspace-dev create -f postgresql-secret.yaml
 secret "dspace-postgresql" created
 ```
 
 ```console
-$ kubectl -n dspace-dev create -f dspace-secrets.yaml 
+$ kubectl -n dspace-dev create -f dspace-secrets.yaml
 secret "dspace-secrets" created
 ```
 
@@ -167,7 +163,7 @@ gitlab-scientificnet-org-registry   kubernetes.io/dockerconfigjson        1
 
 Next, create a RBD Claim
 ```console
-$ kubectl -n dspace-dev create -f postgresql-pvc.yaml 
+$ kubectl -n dspace-dev create -f postgresql-pvc.yaml
 persistentvolumeclaim "dspace-postgresql" created
 
 $ kubectl -n dspace-dev get pvc
@@ -177,7 +173,7 @@ dspace-postgresql   Bound     pvc-610c358d-a518-11e7-b21c-b60483de6a40   8Gi
 
 Ok, now we are ready to deploy our PostgreSQL!
 ```console
-$ kubectl -n dspace-dev create -f postgresql.yaml 
+$ kubectl -n dspace-dev create -f postgresql.yaml
 
 $ kubectl -n dspace-dev get po
 NAME                                 READY     STATUS              RESTARTS   AGE
@@ -224,7 +220,7 @@ DSPACE_POSTGRESQL_PORT_5432_TCP=tcp://10.13.0.173:5432
 
 ## DSpace
 ```console
-$ kubectl -n dspace-dev create -f servlet.yaml 
+$ kubectl -n dspace-dev create -f servlet.yaml
 pod "servlet" created
 ```
 ```console
@@ -235,16 +231,16 @@ servlet                              1/1       Running   0          34s
 ```
 ```console
 $ kubectl -n dspace-dev logs servlet
-[Wed Oct  4 12:24:25 UTC 2017] Starting 
-[Wed Oct  4 12:24:25 UTC 2017] Cloning lindat-dspace repository...skip! /opt/repository/sources/dspace already exists 
-[Wed Oct  4 12:24:25 UTC 2017] Creating workspace...skip! /opt/repository/workspace already exists 
+[Wed Oct  4 12:24:25 UTC 2017] Starting
+[Wed Oct  4 12:24:25 UTC 2017] Cloning lindat-dspace repository...skip! /opt/repository/sources/dspace already exists
+[Wed Oct  4 12:24:25 UTC 2017] Creating workspace...skip! /opt/repository/workspace already exists
 [Wed Oct  4 12:24:25 UTC 2017] Copy makefile...skip! /opt/repository/workspace/config/variable.makefile already exists
 [Wed Oct  4 12:24:25 UTC 2017] Copy local.properties...skip! /opt/repository/workspace/sources/local.properties already exists
-[Wed Oct  4 12:24:25 UTC 2017] Compilation and Deployment of DSpace...already done 
-already done 
-already done 
+[Wed Oct  4 12:24:25 UTC 2017] Compilation and Deployment of DSpace...already done
+already done
+already done
 [Wed Oct  4 12:24:25 UTC 2017] Copy aai_config.js...skip! /opt/lindat-dspace/installation/webapps/xmlui/themes/UFAL/lib/js/aai_config.js already exists
-[Wed Oct  4 12:24:25 UTC 2017] Create dspace admin...already done 
+[Wed Oct  4 12:24:25 UTC 2017] Create dspace admin...already done
 [Wed Oct  4 12:24:25 UTC 2017] Starting Tomcat...Starting Tomcat 8...
 Tomcat started.
 ```

changelog.txt

-Releases
-========
-
-gitlab.scientificnet.org:4567/commul/dspace/dspace
-
-0.3
-added dspace-entrypoint.sh v0.2
-
-0.2
-Removed DSpace's installation from the Dockerfile.
-Moved commul-customization into /tmp , so is possible to mount external volumes into /opt/repository/ and then  copying files again from /tmp
-
-0.1
-Original build from https://github.com/commul/clarin-dspace-docker @ e5ad160
-
-
-gitlab.scientificnet.org:4567/commul/dspace/postgres
-
-0.3
-added env variables
-
-0.2
-
-0.1
-Original build from https://github.com/commul/clarin-dspace-docker @ e5ad160

clarin-dev2clarin.sh

@@ -7,6 +7,7 @@ perl -pi -e 's/clarin-dev/clarin/g' dockerfiles/commul-customization/default-ssl
 
 perl -pi -e 's/clarin-dev/clarin/g' dockerfiles/commul-customization/aai.js
 perl -pi -e 's/clarin-dev/clarin/g' dockerfiles/commul-customization/aai_config.js 
+perl -pi -e 's/clarin-dev/clarin/g' dockerfiles/commul-customization/aa-statistics.php
 perl -pi -e 's/clarin-dev/clarin/g' dockerfiles/commul-customization/clarin.eurac.edu.template.metadata.xml
 perl -pi -e 's/clarin-dev/clarin/g' dockerfiles/commul-customization/webpage/index.html 
 
@@ -20,6 +21,7 @@ perl -pi -e 's/_5d15d5a81bc952da1d34986ebc7713c0d5c9ea61/_5d15d5a81bc952da1d3498
 perl -pi -e 's/clarin-dev/clarin/g' dockerfiles/docker-compose.yml
 echo "Modified all files to point to clarin.eurac.edu instead of clarin-dev.eurac.edu."
 
-# adapt namespace in kubernetes yaml files
+# adapt namespace and image pull path in kubernetes yaml files
 sed -i 's/dspace-dev/dspace/' kubernetes/*.yaml 
 perl -pi -e 's/clarin-dev/clarin/g' kubernetes/dspace-ingress.yaml
+perl -pi -e 's#/staging/#/production/#g' kubernetes/*deploy*yaml

dockerfiles/Dockerfile.dspace

@@ -34,9 +34,9 @@ ENV LANG=en_US.UTF-8
 ENV LANGUAGE=en_US.UTF-8
 
 # build tomcat 8.0.35, version recommended by LINDAT
-RUN wget -O /tmp/apache-tomcat-8.0.35.tar.gz https://archive.apache.org/dist/tomcat/tomcat-8/v8.0.35/bin/apache-tomcat-8.0.35.tar.gz
+RUN wget -O /tmp/apache-tomcat-8.0.52.tar.gz https://archive.apache.org/dist/tomcat/tomcat-8/v8.0.52/bin/apache-tomcat-8.0.52.tar.gz
 RUN mkdir /opt/tomcat8
-RUN tar xvzf /tmp/apache-tomcat-8.0.35.tar.gz -C /opt/tomcat8/ --strip-components=1
+RUN tar xvzf /tmp/apache-tomcat-8.0.52.tar.gz -C /opt/tomcat8/ --strip-components=1
 RUN groupadd tomcat8
 RUN useradd -s /bin/false -g tomcat8 -d /opt/tomcat8/ tomcat8
 
@@ -83,6 +83,7 @@ RUN make new_deploy
 
 # DEBUG: to be able to access tomcat manager through browser
 # COPY commul-customization/tomcat-users.xml /opt/tomcat8/conf/
+RUN perl -pi -e 's/#xmlui\.session\.ipcheck = true/xmlui.session.ipcheck = false/' /opt/lindat-dspace/installation/config/dspace.cfg
 
 # set up handle server
 COPY commul-customization/hdl-setup.tmp /tmp/

dockerfiles/Dockerfile.nginx

@@ -49,12 +49,16 @@ RUN chmod a+x /etc/init.d/nginx
 RUN ln -s /opt/nginx/sbin/nginx /usr/sbin/nginx
 # copy over static html
 COPY commul-customization/webpage/ /opt/nginx/html/
+
 #RUN mkdir /opt/nginx/html/img
 #COPY commul-customization/index.html /opt/nginx/html/
 #COPY commul-customization/eurac.png /opt/nginx/html/img/
-# create the test secure folder
-RUN mkdir /opt/nginx/html/secure
-COPY commul-customization/shib_test.pl /opt/nginx/html/secure/
+
+# install php
+RUN apt-get update && apt-get install -y php-fpm php-xml
+# copy over aa-statistics script
+RUN mkdir /opt/nginx/html/php
+COPY commul-customization/aa-statistics.php /opt/nginx/html/php/
 
 # get aai project
 RUN apt-get install -y git
@@ -69,12 +73,26 @@ RUN /tmp/shibboleth_sp_with_fastcgi.sh
 RUN cp /opt/shibboleth-sp-fastcgi/etc/shibboleth/shibd-debian /etc/init.d/shibd
 RUN chmod a+x /etc/init.d/shibd
 
+# create the test secure folder and set up perl fastcgi
+RUN mkdir /opt/nginx/html/secure
+RUN apt-get update && apt-get install -y fcgiwrap
+RUN cpanm CGI URI XML::Twig LWP::Protocol::https
+COPY commul-customization/shib_test.pl /opt/nginx/html/secure/
+COPY commul-customization/shib_fastcgi_params /opt/nginx/conf/
+COPY commul-customization/attribute-map.xml /opt/shibboleth-sp-fastcgi/etc/shibboleth/
+RUN chown -R www-data:www-data /opt/nginx/html/secure
+RUN chmod a+x /opt/nginx/html/secure/shib_test.pl
 
 # install supervisor
 RUN apt-get install -y python-setuptools
 RUN easy_install supervisor
 COPY commul-customization/supervisord.conf /etc/
 RUN mkdir -p /var/log/supervisor
+# create dirs for php-fpm socket/pid and log files
+RUN mkdir -p /run/php
+RUN mkdir -p /var/log/php-fpm/
+COPY commul-customization/php-fpm.conf /etc/php/7.0/fpm/
+COPY commul-customization/php.ini /etc/php/7.0/fpm/
 
 # copy over config files
 COPY commul-customization/default-ssl /opt/nginx/conf/
@@ -104,4 +122,3 @@ RUN yui-compressor -o aai.min.js aai.js
 
 
 ENTRYPOINT ["/usr/local/bin/supervisord", "-c", "/etc/supervisord.conf"]
-

dockerfiles/adapt_utilities_sql.sh

@@ -3,6 +3,7 @@
 git clone https://github.com/commul/clarin-dspace.git -b commul-custom /tmp/dspace
 cp /tmp/dspace/utilities/utilities.sql /tmp/sql/
 cp /tmp/dspace/utilities/license_definition.txt /tmp/sql/
+chmod -R a+w /tmp/sql/
 perl -pi -e "s#afile :utildir '/license_definition.txt'#afile '/tmp/sql/license_definition.txt'#;" /tmp/sql/utilities.sql
-# perl -pi -e 's/\$DSPACE_USER/$ENV{DSPACE_USER}/;' /tmp/sql/utilities.sql
-# rm -rf /tmp/dspace
+#perl -pi -e 's/\$DSPACE_USER/$ENV{DSPACE_USER}/g;' /tmp/sql/utilities.sql
+rm -rf /tmp/dspace

dockerfiles/commul-customization/aa-statistics.php

+<?php
+
+// redirect to the correct destination
+header('Location: ' . $_GET['return'], true, 302);
+
+
+// This script finds exported header names from shibboleth and sends them to an aggregator.
+//
+// The main goal is to monitor attribute release and provide a central place where "bad" IdPs can be
+// easily blackmailed...
+//
+// See: https://github.com/ufal/clarin-sp-aaggregator
+// by lindat-dev team (ok, jm)
+//
+
+//====== CONFIGURATION ======
+// REST api of the aggregator
+$aggregator_url='https://clarin-aa.ms.mff.cuni.cz/aaggreg/v1/got';
+//your SP entity id
+$sp='https://clarin.eurac.edu/Shibboleth.sso/Metadata';
+//====== /CONFIGURATION ======
+
+
+/**
+ * Build the request url and execute curl on it.
+ * @todo test asynch...
+ */
+function send_info($idp, $sp, $ts, $attrs, $suspicious) 
+{
+    global $aggregator_url, $sp;
+    $attributes_encoded = "attributes[]=" . implode('&attributes[]=', array_map('urlencode', $attrs));
+    // -g does not allow curl to interpret []{}
+    $cmd = "curl -g '$aggregator_url?idp=$idp&sp=$sp&timestamp=$ts&$attributes_encoded&warn=$suspicious'";
+    $cmd .= " > /dev/null 2>&1 &";
+    echo '<pre>'.$cmd.'</pre>';
+
+    exec($cmd, $output, $exit);
+    return $exit == 0;
+}
+
+/**
+ * Array version of getenv - corner cases for 0, null etc.
+ */
+function getenvs($envarray, &$ret)
+{
+    foreach($envarray as $value)
+    {
+        $ret = getenv($value);
+        if ($ret) {
+            return true;
+        }
+    }
+    return false;
+}
+
+
+$idps = array();
+
+// we need the count so we can iterate over the exported assertions
+// - the env variable can be exposed under different names!
+//
+$assertion_count = 0;
+$assertion_count_name = "Shib-Assertion-Count";
+if (!getenvs(
+        array($assertion_count_name, str_replace('-','_',strtoupper("http_".$assertion_count_name))), 
+        $assertion_count
+    ))
+{
+    http_response_code(400);
+    echo("$assertion_count_name not found");
+    exit;
+}
+$assertion_count = (int)$assertion_count;
+
+
+// idp that was used to authenicate
+$idp = null;
+// timestamp
+$dt = new DateTime();
+$ts = $dt->format("Y-m-d\TH:i:s.z\Z");
+
+// obtain the assertions
+// - the env variable can be exposed under different names!
+//
+$assertion_link_attr_name = "Shib-Assertion-"; 
+$assertion_link_attr_name_upper = str_replace('-','_',strtoupper("http_". "Shib-Assertion-")); 
+for ($i=$assertion_count; 0 < $i; --$i) 
+{
+    // why would there be more than one assertion?
+    $n = str_pad($i, 2, "0", STR_PAD_LEFT);
+
+    //$n is 00, or 01
+    $assertion_link = "";
+    getenvs(array($assertion_link_attr_name . $n, $assertion_link_attr_name_upper . $n), $assertion_link);
+    
+    if(!empty($assertion_link))
+    {
+        $assertion_link = str_replace("https://" . getenv("SERVER_NAME"), "https://127.0.0.1", $assertion_link);
+        //TODO add a timeout
+		// need to ignore that certificate does not match 127.0.0.1
+       $arrContextOptions=array( 
+               "ssl"=>array( 
+                       "verify_peer"=>false, 
+                       "verify_peer_name"=>false, 
+               ), 
+       ); 
+       $assertion = file_get_contents($assertion_link, false, stream_context_create($arrContextOptions)); 
+        $xml = simplexml_load_string($assertion); 
+
+        // echo $xml->asXML()."=====\n\n";
+        try{
+            $idp = (string)$xml->xpath('//*[local-name()="Issuer"]')[0];
+        } catch(Exception $e){
+            echo 'Caught exception: ',  $e->getMessage(), "\nProbably no Issuer or xml.\n";
+            exit;
+        }
+        if (!array_key_exists($idp, $idps)) 
+        {
+            $idps[$idp] = array();
+        }
+        foreach ($xml->xpath('//*[local-name()="Attribute"]/@Name') as $name)
+        {
+            array_push($idps[$idp], (string)$name);
+        }
+        // AudienceRestriction/Audience and NameID/@SPNameQualifier usually
+        // have the value of spEntityId, it's not guaranteed though.
+        //$sp = (string)$xml->xpath('//@SPNameQualifier')[0];
+    }
+}
+
+// anything strange?
+$suspicious = "";
+if (1 < count($idps)) {
+    $suspicious = "more than 1 idp";
+}
+
+// aggregate the info
+send_info($idp, $sp, $ts, $idps[$idp], $suspicious);
+
+exit;

dockerfiles/commul-customization/attribute-map.xml

+<Attributes xmlns="urn:mace:shibboleth:2.0:attribute-map" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
+
+    <!--
+	SAML1 + SAML2 declarations : name = input, id is output name
+	-->
+
+    <!-- EduPersonPrincipalName -->
+	<Attribute name="urn:mace:dir:attribute-def:eduPersonPrincipalName" id="mace-eduPersonPrincipalName" >
+    		<AttributeDecoder xsi:type="ScopedAttributeDecoder"/>
+	</Attribute>
+	<Attribute name="urn:mace:dir:attribute-def:eduPersonPrincipalName" nameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified" id="mace-eduPersonPrincipalName" />
+	<Attribute name="urn:oid:1.3.6.1.4.1.5923.1.1.1.6" id="eppn" >
+    		<AttributeDecoder xsi:type="ScopedAttributeDecoder"/>
+	</Attribute>
+
+    <!-- eduPersonTargetedID -->
+    <Attribute name="urn:mace:dir:attribute-def:eduPersonTargetedID" id="eduPersonTargetedID" /> <!-- incorrect SAML 1.1 mapping, required for historical compatibility see https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPTargetedID -->
+    <Attribute name="urn:oid:1.3.6.1.4.1.5923.1.1.1.10" id="eduPersonTargetedID"> <!-- the usually recommended approach to passing an eduPersonTargetedID to SAML 2.0 SPs, including Shibboleth 2.x. https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPTargetedID -->
+           <AttributeDecoder xsi:type="NameIDAttributeDecoder" formatter="$Name!!$NameQualifier!!$SPNameQualifier"/>
+    </Attribute>
+    <Attribute name="urn:oid:1.3.6.1.4.1.5923.1.1.1.10" id="eduPersonTargetedID" nameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified">
+           <AttributeDecoder xsi:type="NameIDAttributeDecoder" formatter="$Name!!$NameQualifier!!$SPNameQualifier"/>
+    </Attribute>
+    <Attribute name="urn:oid:1.3.6.1.4.1.5923.1.1.1.10" id="eduPersonTargetedID" nameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri">
+           <AttributeDecoder xsi:type="NameIDAttributeDecoder" formatter="$Name!!$NameQualifier!!$SPNameQualifier"/>
+    </Attribute>
+    <!-- Fourth, the SAML 2.0 NameID Format: -->
+    <Attribute name="urn:oasis:names:tc:SAML:2.0:nameid-format:persistent" id="persistent-id">
+        <AttributeDecoder xsi:type="NameIDAttributeDecoder" formatter="$NameQualifier!$SPNameQualifier!$Name" defaultQualifiers="true"/>
+    </Attribute>
+
+    <!-- eduPersonScopedAffiliation -->
+    <Attribute name="urn:mace:dir:attribute-def:eduPersonScopedAffiliation" id="eduPersonScopedAffiliation"/>
+    <Attribute name="urn:oid:1.3.6.1.4.1.5923.1.1.1.9" id="eduPersonScopedAffiliation"/>
+
+    <!-- eduPersonEntitlement -->
+    <Attribute name="urn:mace:dir:attribute-def:eduPersonEntitlement" id="eduPersonEntitlement"/>
+    <Attribute name="urn:oid:1.3.6.1.4.1.5923.1.1.1.7" id="eduPersonEntitlement"/>
+
+    <!-- commonName -->
+    <Attribute name="urn:mace:dir:attribute-def:cn" id="cn" />
+    <Attribute name="urn:oid:2.5.4.3" id="cn" />
+
+    <!-- mail -->
+    <Attribute name="urn:mace:dir:attribute-def:mail" id="mail"/>
+    <Attribute name="urn:oid:0.9.2342.19200300.100.1.3" id="mail"/>
+
+    <!-- organization -->
+    <Attribute name="urn:mace:dir:attribute-def:organizationName" id="organizationName"/>
+    <Attribute name="urn:oid:2.5.4.10" id="organizationName"/>
+
+    <!-- displayName -->
+    <Attribute name="urn:mace:dir:attribute-def:displayName" id="displayName"/>
+    <Attribute name="urn:oid:2.16.840.1.113730.3.1.241" id="displayName"/>
+
+    <!--
+    <Attribute name="urn:mace:dir:attribute-def:uid" id="uid" />
+    <Attribute name="urn:oid:0.9.2342.19200300.100.1.1" id="uid" />
+    <Attribute name="urn:mace:dir:attribute-def:givenName" id="givenName"/>
+    <Attribute name="urn:mace:dir:attribute-def:surName" id="surName"/>
+    <Attribute name="urn:mace:dir:attribute-def:eduPersonAffiliation" id="eduPersonAffiliation"/>
+    <Attribute name="urn:mace:dir:attribute-def:damlrPersonAffiliation" id="eduPersonAffiliation" />
+    -->
+</Attributes>

dockerfiles/commul-customization/clarin.eurac.edu.template.metadata.xml

@@ -11,6 +11,10 @@
                 NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri">
                 <saml:AttributeValue>http://refeds.org/category/research-and-scholarship</saml:AttributeValue>
             </saml:Attribute>
+            <saml:Attribute 
+               xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" Name="http://macedir.org/entity-category" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri">
+                <saml:AttributeValue>http://clarin.eu/category/clarin-member</saml:AttributeValue>
+            </saml:Attribute>
         </mdattr:EntityAttributes>
     </md:Extensions>
     <md:SPSSODescriptor protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol urn:oasis:names:tc:SAML:1.1:protocol urn:oasis:names:tc:SAML:1.0:protocol">
@@ -26,9 +30,6 @@
                 <mdui:InformationURL xml:lang="de">https://clarin.eurac.edu/repository/xmlui/page/about</mdui:InformationURL>
                 <mdui:InformationURL xml:lang="it">https://clarin.eurac.edu/repository/xmlui/page/about</mdui:InformationURL>
 		<mdui:Logo height="85" width="180">https://clarin.eurac.edu/repository/xmlui/themes/UFAL/lib/lindat/public/img/eurac_research.svg</mdui:Logo>                    
-                <mdui:PrivacyStatementURL xml:lang="en">https://clarin.eurac.edu/repository/xmlui/page/privacypolicy</mdui:PrivacyStatementURL>
-                <mdui:PrivacyStatementURL xml:lang="de">https://clarin.eurac.edu/repository/xmlui/page/privacypolicy</mdui:PrivacyStatementURL>
-                <mdui:PrivacyStatementURL xml:lang="it">https://clarin.eurac.edu/repository/xmlui/page/privacypolicy</mdui:PrivacyStatementURL>
                 <mdui:PrivacyStatementURL xml:lang="en">https://clarin.eurac.edu/repository/xmlui/page/registration-privacypolicy</mdui:PrivacyStatementURL>
                 <mdui:PrivacyStatementURL xml:lang="de">https://clarin.eurac.edu/repository/xmlui/page/registration-privacypolicy</mdui:PrivacyStatementURL>
                 <mdui:PrivacyStatementURL xml:lang="it">https://clarin.eurac.edu/repository/xmlui/page/registration-privacypolicy</mdui:PrivacyStatementURL>

dockerfiles/commul-customization/default-ssl

@@ -82,11 +82,18 @@ server {
 
 #    try_files $uri $uri/ /index.php?$args;
 
-#    location ~ \.php$ {
-#      include process_php;
-#    }
   }
 
+  location /php/aa-statistics.php {
+      shib_request /shibauthorizer;
+	  shib_request_use_headers on;
+      fastcgi_pass unix:/run/php/php7.0-fpm.sock;
+      fastcgi_param  SCRIPT_FILENAME /opt/nginx/html$fastcgi_script_name;
+      fastcgi_param  PATH_INFO $fastcgi_script_name;
+      fastcgi_split_path_info ^(.+\.php)(/.+)$;
+      include fastcgi_params;
+   }
+
   location /repository {
 
     client_body_buffer_size 32K;
@@ -125,14 +132,14 @@ server {
     alias /opt/repository/sources/lindat-aai-discovery; }
 
   # add path your repository path that will be protected by shibboleth
-  location /repository/xmlui/shibboleth-login { 
-	include repository_auth; 
+  location /repository/xmlui/shibboleth-login {
+	include repository_auth;
     ajp_keep_conn on;
     ajp_pass tomcats;
 	}
 
-  location /xmlui/shibboleth-login            { 
-	include repository_auth; 
+  location /xmlui/shibboleth-login            {
+	include repository_auth;
 	ajp_keep_conn on;
     ajp_pass tomcats;
 	}
@@ -161,18 +168,19 @@ server {
   location /shibboleth-sp {
     alias /opt/shibboleth-sp-fastcgi/share/shibboleth/;
   }
- 
+
   location /secure {
-    alias /opt/repository/sources/secure;
-    include shib_clear_headers;
+         #include shib_clear_headers;
          #Add your attributes here. They get introduced as headers
          #by the FastCGI authorizer so we must prevent spoofing.
-         more_clear_input_headers 'displayName' 'mail' 'persistent-id';
+         #more_clear_input_headers 'displayName' 'mail' 'persistent-id';
          shib_request /shibauthorizer;
          shib_request_use_headers on;
-         proxy_pass http://localhost:8080;
+         gzip off;
+         include shib_fastcgi_params;
+         fastcgi_pass unix:/tmp/fcgiwrap.socket;
+         fastcgi_param SCRIPT_FILENAME /opt/nginx/html$fastcgi_script_name;
      }
 
   include /opt/nginx/conf/proxies-enabled/*;
 }
-

dockerfiles/commul-customization/fastcgi.conf

-
-fastcgi_param  SCRIPT_FILENAME    $document_root$fastcgi_script_name;
-fastcgi_param  QUERY_STRING       $query_string;
-fastcgi_param  REQUEST_METHOD     $request_method;
-fastcgi_param  CONTENT_TYPE       $content_type;
-fastcgi_param  CONTENT_LENGTH     $content_length;
-
-fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
-fastcgi_param  REQUEST_URI        $request_uri;
-fastcgi_param  DOCUMENT_URI       $document_uri;
-fastcgi_param  DOCUMENT_ROOT      $document_root;
-fastcgi_param  SERVER_PROTOCOL    $server_protocol;
-fastcgi_param  REQUEST_SCHEME     $scheme;
-fastcgi_param  HTTPS              $https if_not_empty;
-
-fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
-fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;
-
-fastcgi_param  REMOTE_ADDR        $remote_addr;
-fastcgi_param  REMOTE_PORT        $remote_port;
-fastcgi_param  SERVER_ADDR        $server_addr;
-fastcgi_param  SERVER_PORT        $server_port;
-fastcgi_param  SERVER_NAME        $server_name;
-
-# PHP only, required if PHP was built with --enable-force-cgi-redirect
-fastcgi_param  REDIRECT_STATUS    200;

dockerfiles/commul-customization/init-dspace-dbs.sh

 #!/bin/bash
 set -e
 
+# adapt utilities.sql with dspace username from env variable
+perl -pi -e 's/\$DSPACE_USER/$ENV{DSPACE_USER}/g;' /tmp/sql/utilities.sql
+
+# populate postgres databases
 psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" <<-EOSQL
     CREATE USER $DSPACE_USER WITH PASSWORD '$DSPACE_PASSWORD';
     CREATE DATABASE "$DSPACE_DBNAME" OWNER $DSPACE_USER ENCODING "UTF-8" TEMPLATE template0;
@@ -8,13 +12,3 @@ psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" <<-EOSQL
 EOSQL
 
 psql --username "$POSTGRES_USER" --set=utildir="/tmp/sql/" -p 5432 $DSPACE_UTILSDBNAME < /tmp/sql/utilities.sql
-
-# this is a better version using kubernetes environment variables
-##!/bin/bash
-#set -xe
-#
-#psql --username "$POSTGRES_USER" -c "CREATE USER $dspaceuser WITH PASSWORD '$dspacepassword'";
-#psql --username "$POSTGRES_USER" -c "CREATE DATABASE "$dspacedbname" OWNER $dspaceuser ENCODING \"UTF-8\" TEMPLATE template0;"
-#psql --username "$POSTGRES_USER" -c "CREATE DATABASE "$dspaceutilsdbname" OWNER $dspaceuser ENCODING \"UTF-8\" TEMPLATE template0;"
-#
-#psql --username "$POSTGRES_USER" --set=utildir="/tmp/sql/" -p 5432 $secretdspaceutilsdbname < /tmp/sql/utilities.sql

dockerfiles/commul-customization/local.properties

@@ -69,6 +69,7 @@ db.url = jdbc:postgresql://postgres:${lr.database.port}/${lr.database}
 db.driver = org.postgresql.Driver
 
 # Credentials for the database user. The user should have read/write permissions on your database.
+# note that the db user should not contain uppercase letters
 db.username = ${DSPACE_USER}
 db.password = ${DSPACE_PASSWORD}
 

dockerfiles/commul-customization/nginx_build.sh

@@ -36,6 +36,7 @@ cd nginx_src
    --add-module="$DIR/nginx-headers-more-module" \
    --add-module="$DIR/nginx-ajp-module" \
    --add-module="$DIR/nginx-shib-module" \
+   --with-http_realip_module \
    --with-http_ssl_module \
    --with-http_stub_status_module \
    --with-openssl="$DIR/openssl" \

dockerfiles/commul-customization/php-fpm.conf

+;;;;;;;;;;;;;;;;;;;;;
+; FPM Configuration ;
+;;;;;;;;;;;;;;;;;;;;;
+
+; All relative paths in this configuration file are relative to PHP's install
+; prefix (/usr). This prefix can be dynamically changed by using the
+; '-p' argument from the command line.
+
+;;;;;;;;;;;;;;;;;;
+; Global Options ;
+;;;;;;;;;;;;;;;;;;
+
+[global]
+; Pid file
+; Note: the default prefix is /var
+; Default Value: none
+pid = /run/php/php7.0-fpm.pid
+
+; Error log file
+; If it's set to "syslog", log is sent to syslogd instead of being written
+; in a local file.
+; Note: the default prefix is /var
+; Default Value: log/php-fpm.log
+error_log = /var/log/php7.0-fpm.log
+
+; syslog_facility is used to specify what type of program is logging the
+; message. This lets syslogd specify that messages from different facilities
+; will be handled differently.
+; See syslog(3) for possible values (ex daemon equiv LOG_DAEMON)
+; Default Value: daemon
+;syslog.facility = daemon
+
+; syslog_ident is prepended to every message. If you have multiple FPM
+; instances running on the same server, you can change the default value
+; which must suit common needs.
+; Default Value: php-fpm
+;syslog.ident = php-fpm
+
+; Log level
+; Possible Values: alert, error, warning, notice, debug
+; Default Value: notice
+;log_level = notice
+
+; If this number of child processes exit with SIGSEGV or SIGBUS within the time
+; interval set by emergency_restart_interval then FPM will restart. A value
+; of '0' means 'Off'.
+; Default Value: 0
+;emergency_restart_threshold = 0
+
+; Interval of time used by emergency_restart_interval to determine when
+; a graceful restart will be initiated.  This can be useful to work around
+; accidental corruptions in an accelerator's shared memory.
+; Available Units: s(econds), m(inutes), h(ours), or d(ays)
+; Default Unit: seconds
+; Default Value: 0
+;emergency_restart_interval = 0
+
+; Time limit for child processes to wait for a reaction on signals from master.
+; Available units: s(econds), m(inutes), h(ours), or d(ays)
+; Default Unit: seconds
+; Default Value: 0
+;process_control_timeout = 0
+
+; The maximum number of processes FPM will fork. This has been design to control
+; the global number of processes when using dynamic PM within a lot of pools.
+; Use it with caution.
+; Note: A value of 0 indicates no limit
+; Default Value: 0
+; process.max = 128
+
+; Specify the nice(2) priority to apply to the master process (only if set)
+; The value can vary from -19 (highest priority) to 20 (lower priority)
+; Note: - It will only work if the FPM master process is launched as root
+;       - The pool process will inherit the master process priority
+;         unless it specified otherwise
+; Default Value: no set
+; process.priority = -19
+
+; Send FPM to background. Set to 'no' to keep FPM in foreground for debugging.
+; Default Value: yes
+daemonize = no
+
+; Set open file descriptor rlimit for the master process.
+; Default Value: system defined value
+;rlimit_files = 1024
+
+; Set max core size rlimit for the master process.
+; Possible Values: 'unlimited' or an integer greater or equal to 0
+; Default Value: system defined value
+;rlimit_core = 0
+
+; Specify the event mechanism FPM will use. The following is available:
+; - select     (any POSIX os)
+; - poll       (any POSIX os)
+; - epoll      (linux >= 2.5.44)
+; - kqueue     (FreeBSD >= 4.1, OpenBSD >= 2.9, NetBSD >= 2.0)
+; - /dev/poll  (Solaris >= 7)
+; - port       (Solaris >= 10)
+; Default Value: not set (auto detection)
+;events.mechanism = epoll
+
+; When FPM is build with systemd integration, specify the interval,
+; in second, between health report notification to systemd.
+; Set to 0 to disable.
+; Available Units: s(econds), m(inutes), h(ours)
+; Default Unit: seconds
+; Default value: 10
+;systemd_interval = 10
+
+;;;;;;;;;;;;;;;;;;;;
+; Pool Definitions ;
+;;;;;;;;;;;;;;;;;;;;
+
+; Multiple pools of child processes may be started with different listening
+; ports and different management options.  The name of the pool will be
+; used in logs and stats. There is no limitation on the number of pools which
+; FPM can handle. Your system will tell you anyway :)
+
+; Include one or more files. If glob(3) exists, it is used to include a bunch of
+; files from a glob(3) pattern. This directive can be used everywhere in the
+; file.
+; Relative path can also be used. They will be prefixed by:
+;  - the global prefix if it's been set (-p argument)
+;  - /usr otherwise
+include=/etc/php/7.0/fpm/pool.d/*.conf

dockerfiles/commul-customization/shib_clear_headers

@@ -10,6 +10,8 @@
 more_clear_input_headers
     Auth-Type
     Shib-Application-Id
+    Shib-Assertion-Count
+    Shib-Assertion-01
     Shib-Authentication-Instant
     Shib-Authentication-Method
     Shib-Authncontext-Class
@@ -30,4 +32,3 @@ more_clear_input_headers
 #     DisplayName
 #     Email
 #     OrganizationName;
-