Image style: 300x300 Access Change?

mbc-image-processor is a Message Broker consumer requested by @aaronschachter that makes requests for report back images right after they're submitted to trigger generation for caching of the 300x300 image style.
A typical request would be https://www.dosomething.org/sites/default/files/styles/300x300/public/reportbacks/3501/uid_3139619-nid_3501-0.png. This requests produces a 403 "Access Denied/Forbidden" error.
The imageProcessingQueue (http://rabbit.dosomething.org/#/queues/dosomething/imageProcessingQueue) has entries waiting to be processed.
Have the access rights changed to prevent this request from within the DS network from mbc-php (10-100-25-161) to https://www.dosomething.org/sites/default/files/styles/300x300/public/reportbacks?
Reference:

300x300 Image Style:
https://www.dosomething.org/admin/config/media/image-styles/edit/300x300
mbc-image-processor: dosomething@ip-10-100-25-161:/opt/rabbit/MessageBroker-PHP/mbc-image-processor

Issue resolved, the itok=<random string> needs to be included in the request.


Unable to mount volumes for pod Learner

What happend:
Hi there, thanks a lot for your work. It's impressive, so I was trying to deploy it on local MINIKUBE and local DIND, but in fact none of them worked properly. I was stuck in an issue for few days, so I'd like to ask you guys for help. By chance I've found something similar to my issue from your docs but in the different condition, which means:

my local minikube encountered the issue which was recorded in the DIND-TRAING -- all pods worked as expected

alertmanager-7bd87d99cc-jhp2b                                     1/1       Running             0          6h
etcd0                                                             1/1       Running             0          6h
ffdl-lcm-8d555c7bf-dqqhg                                          1/1       Running             0          6h
ffdl-restapi-7f5c57c77d-k67pm                                     1/1       Running             0          6h
ffdl-trainer-6777dd5756-xkk65                                     1/1       Running             0          6h
ffdl-trainingdata-696b99ff5c-tvbtc                                1/1       Running             0          6h
ffdl-ui-95d6464c7-bv2sn                                           1/1       Running             0          6h
jobmonitor-0d296791-2adc-4336-4f01-b280090460c3-cbdb48cfd-qqsvz   1/1       Running             0          1h
learner-0d296791-2adc-4336-4f01-b280090460c3-0                    0/1       ContainerCreating   0          1h
lhelper-0d296791-2adc-4336-4f01-b280090460c3-54858658b-p7vfc      2/2       Running             0          1h
mongo-0                                                           1/1       Running             4          6h
prometheus-67fb854b59-c884p                                       2/2       Running             0          6h
pushgateway-5665768d5c-jdlnl                                      2/2       Running             0          6h
storage-0                                                         1/1       Running             0          6h

except the pod learner with eternal pending status because of the following warning.
Unable to mount volumes for pod "learner-d3a04eac-a64a-427e-56e5-8366cc84292f-0_default(33f78708-f963-11e8-aa08-0800275e57f0)": timeout expired waiting for volumes to attach or mount for pod "default"/"learner-d3a04eac-a64a-427e-56e5-8366cc84292f-0". list of unmounted volumes=[cosinputmount-d3a04eac-a64a-427e-56e5-8366cc84292f cosoutputmount-d3a04eac-a64a-427e-56e5-8366cc84292f]. list of unattached volumes=[cosinputmount-d3a04eac-a64a-427e-56e5-8366cc84292f cosoutputmount-d3a04eac-a64a-427e-56e5-8366cc84292f learner-entrypoint-files jobdata]

and here's the details of pod learner-x
Name:               learner-0d296791-2adc-4336-4f01-b280090460c3-0
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               minikube/<IP_ADDRESS>
Start Time:         Thu, 06 Dec 2018 17:05:52 +0100
Labels:             controller-revision-hash=learner-0d296791-2adc-4336-4f01-b280090460c3-999bf4986
                    service=dlaas-learner
                    statefulset.kubernetes.io/pod-name=learner-0d296791-2adc-4336-4f01-b280090460c3-0
                    training_id=training-bFEXXGPmR
                    user_id=test-user
Annotations:        scheduler.alpha.kubernetes.io/nvidiaGPU={ "AllocationPriority": "Dense" }
                    scheduler.alpha.kubernetes.io/tolerations=[ { "key": "dedicated", "operator": "Equal", "value": "gpu-task" } ]
Status:             Pending
IP:
Controlled By:      StatefulSet/learner-0d296791-2adc-4336-4f01-b280090460c3
Containers:
  learner:
    Container ID:
    Image:         tensorflow/tensorflow:1.5.0-py3
    Image ID:
    Ports:         22/TCP, 2222/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      bash
      -c
      export PATH=/usr/local/bin/:$PATH; cp /entrypoint-files/*.sh /usr/local/bin/; chmod +x /usr/local/bin/*.sh;
                        if [ ! -f /job/load-model.exit ]; then
                          while [ ! -f /job/load-model.start ]; do sleep 2; done ;
                          date "+%s%N" | cut -b1-13 > /job/load-model.start_time ;

                        echo "Starting Training $TRAINING_ID"
                        mkdir -p "$MODEL_DIR" ;
                        python -m zipfile -e $RESULT_DIR/_submitted_code/model.zip $MODEL_DIR  ;
                          echo $? > /job/load-model.exit ;
                        fi
                        echo "Done load-model" ;
                        if [ ! -f /job/learner.exit ]; then
                          while [ ! -f /job/learner.start ]; do sleep 2; done ;
                          date "+%s%N" | cut -b1-13 > /job/learner.start_time ;

                        for i in ${!ALERTMANAGER*} ${!DLAAS*} ${!ETCD*} ${!GRAFANA*} ${!HOSTNAME*} ${!KUBERNETES*} ${!MONGO*} ${!PUSHGATEWAY*}; do unset $i; done;
                        export LEARNER_ID=$((${DOWNWARD_API_POD_NAME##*-} + 1)) ;
                        mkdir -p $RESULT_DIR/learner-$LEARNER_ID ;
                        mkdir -p $CHECKPOINT_DIR ;bash -c 'train.sh >> $JOB_STATE_DIR/latest-log 2>&1 ; exit ${PIPESTATUS[0]}' ;
                          echo $? > /job/learner.exit ;
                        fi
                        echo "Done learner" ;
                        if [ ! -f /job/store-logs.exit ]; then
                          while [ ! -f /job/store-logs.start ]; do sleep 2; done ;
                          date "+%s%N" | cut -b1-13 > /job/store-logs.start_time ;

                        echo Calling copy logs.
                        mv -nf $LOG_DIR/* $RESULT_DIR/learner-$LEARNER_ID ;
                        ERROR_CODE=$? ;
                        echo $ERROR_CODE > $RESULT_DIR/learner-$LEARNER_ID/.log-copy-complete ;
                        bash -c 'exit $ERROR_CODE' ;
                          echo $? > /job/store-logs.exit ;
                        fi
                        echo "Done store-logs" ;
                      while true; do sleep 2; done ;
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:             500m
      memory:          1048576k
      nvidia.com/gpu:  0
    Requests:
      cpu:             500m
      memory:          1048576k
      nvidia.com/gpu:  0
    Environment:
      LOG_DIR:                     /job/logs
      GPU_COUNT:                   0.000000
      TRAINING_COMMAND:            python3 convolutional_network.py --trainImagesFile ${DATA_DIR}/train-images-idx3-ubyte.gz   --trainLabelsFile ${DATA_DIR}/train-labels-idx1-ubyte.gz --testImagesFile ${DATA_DIR}/t10k-images-idx3-ubyte.gz   --testLabelsFile ${DATA_DIR}/t10k-labels-idx1-ubyte.gz --learningRate 0.001   --trainingIters 2000
      TRAINING_ID:                 training-bFEXXGPmR
      DATA_DIR:                    /mnt/data/tf_training_data
      MODEL_DIR:                   /job/model-code
      RESULT_DIR:                  /mnt/results/tf_trained_model/training-bFEXXGPmR
      DOWNWARD_API_POD_NAME:       learner-0d296791-2adc-4336-4f01-b280090460c3-0 (v1:metadata.name)
      DOWNWARD_API_POD_NAMESPACE:  default (v1:metadata.namespace)
      LEARNER_NAME_PREFIX:         learner-0d296791-2adc-4336-4f01-b280090460c3
      TRAINING_ID:                 training-bFEXXGPmR
      NUM_LEARNERS:                1
      JOB_STATE_DIR:               /job
      CHECKPOINT_DIR:              /mnt/results/tf_trained_model/_wml_checkpoints
      RESULT_BUCKET_DIR:           /mnt/results/tf_trained_model
    Mounts:
      /entrypoint-files from learner-entrypoint-files (rw)
      /job from jobdata (rw)
      /mnt/data/tf_training_data from cosinputmount-0d296791-2adc-4336-4f01-b280090460c3 (rw)
      /mnt/results/tf_trained_model from cosoutputmount-0d296791-2adc-4336-4f01-b280090460c3 (rw)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  cosinputmount-0d296791-2adc-4336-4f01-b280090460c3:
    Type:       FlexVolume (a generic volume resource that is provisioned/attached using an exec based plugin)
    Driver:     ibm/ibmc-s3fs
    FSType:
    SecretRef:  &{cossecretdata-0d296791-2adc-4336-4f01-b280090460c3}
    ReadOnly:   false
    Options:    map[debug-level:warn endpoint:http://<IP_ADDRESS>:31172 tls-cipher-suite:DEFAULT cache-size-gb:0 chunk-size-mb:52 curl-debug:false kernel-cache:true multireq-max:20 bucket:tf_training_data ensure-disk-free:0 parallel-count:5 region:us-standard s3fs-fuse-retry-count:30 stat-cache-size:100000]
  cosoutputmount-0d296791-2adc-4336-4f01-b280090460c3:
    Type:       FlexVolume (a generic volume resource that is provisioned/attached using an exec based plugin)
    Driver:     ibm/ibmc-s3fs
    FSType:
    SecretRef:  &{cossecretresults-0d296791-2adc-4336-4f01-b280090460c3}
    ReadOnly:   false
    Options:    map[cache-size-gb:0 curl-debug:false endpoint:http://<IP_ADDRESS>:31172 parallel-count:2 bucket:tf_trained_model debug-level:warn s3fs-fuse-retry-count:30 stat-cache-size:100000 chunk-size-mb:52 kernel-cache:false ensure-disk-free:2048 region:us-standard tls-cipher-suite:DEFAULT multireq-max:20]
  learner-entrypoint-files:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      learner-entrypoint-files
    Optional:  false
  jobdata:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
QoS Class:       Guaranteed
Node-Selectors:  <none>
Tolerations:     dedicated=gpu-task:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age               From               Message
  ----     ------       ----              ----               -------
  Warning  FailedMount  1m (x40 over 1h)  kubelet, minikube  Unable to mount volumes for pod "learner-0d296791-2adc-4336-4f01-b280090460c3-0_default(ce612f9d-f970-11e8-aa08-0800275e57f0)": timeout expired waiting for volumes to attach or mount for pod "default"/"learner-0d296791-2adc-4336-4f01-b280090460c3-0". list of unmounted volumes=[cosinputmount-0d296791-2adc-4336-4f01-b280090460c3 cosoutputmount-0d296791-2adc-4336-4f01-b280090460c3]. list of unattached volumes=[cosinputmount-0d296791-2adc-4336-4f01-b280090460c3 cosoutputmount-0d296791-2adc-4336-4f01-b280090460c3 learner-entrypoint-files jobdata]

my local dind encountered the issue with non-hint FAILED ERROR while training. All the pods was running, but there're no pods jobmonitor, learner and lhelper.

Deploying model with manifest 'manifest_testrun.yml' and model files in '.'...
Handling connection for 31404
Handling connection for 31404
FAILED
Error 200: OK

What you expected to happen:
Make FfDL work as properly on either local DIND or MINIKUBE.
Environment:
OS: Darwin local 17.4.0 Darwin Kernel Version 17.4.0:
MINIKUBE:
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

How to reproduce it (as minimally and precisely as possible):
I was just following README.rd with several make instructions
make deploy-plugin
make quickstart-deploy
make test-push-data-s3
make test-job-submit

Anything else we need to know?:
In situation 2, I totally followed the above-mentioned steps;
In situation 1, because it popped out hints that nfs error at first, and I just remember one of the doc I've read about MINIKUBE as if to say that, for persistent volumes, it just supports hostpath type, so I created a PV and PVC, here's the details.
$ kubectl describe pv hostpathtest
Name:            hostpathtest
Labels:          <none>
Annotations:     pv.kubernetes.io/bound-by-controller=yes
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:
Status:          Bound
Claim:           default/static-volume-1
Reclaim Policy:  Retain
Access Modes:    RWO
Capacity:        20Gi
Node Affinity:   <none>
Message:
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /data/hostpath_test
    HostPathType:
Events:            <none>

$ kubectl describe pvc learner-1
Name:          learner-1
Namespace:     default
StorageClass:
Status:        Bound
Volume:        hostpathtest-learner
Labels:        type=dlaas-static-volume
Annotations:   kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{"volume.beta.kubernetes.io/storage-class":""},"labels":{"type":"dlaas-stat...
               pv.kubernetes.io/bind-completed=yes
               pv.kubernetes.io/bound-by-controller=yes
               volume.beta.kubernetes.io/storage-class=
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      20Gi
Access Modes:  RWO
Events:        <none>

Thanks in advance for all advices and have a good day

Hi, sorry to bother again, I recently deployed ffdl on Google cloud again, but one of those pod, ibmcloud-object-storage-deployer which runs in the kube-system namespace, can't work with following reason.
+ DRIVER_LOCATION=/host/usr/libexec/kubernetes/kubelet-plugins/volume/exec/ibm~ibmc-s3fs
+ KUBELET_SVC_CONFIG=/host/lib/systemd/system/kubelet.service
+ apt-get -y update
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [83.2 kB]
Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:5 http://security.ubuntu.com/ubuntu bionic-security/universe Sources [32.0 kB]
Get:6 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [133 kB]
Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/universe Sources [167 kB]
Get:8 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [1367 B]
Get:9 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [281 kB]
Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [900 kB]
Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [6931 B]
Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [599 kB]
Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [10.7 kB]
Get:14 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [3655 B]
Fetched 2381 kB in 1s (1754 kB/s)
Reading package lists...
+ apt-get -y install s3fs
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  ca-certificates file fuse libasn1-8-heimdal libcurl3-gnutls libfuse2
  libgssapi3-heimdal libhcrypto4-heimdal libheimbase1-heimdal
  libheimntlm0-heimdal libhx509-5-heimdal libicu60 libkrb5-26-heimdal
  libldap-2.4-2 libldap-common libmagic-mgc libmagic1 libnghttp2-14 libpsl5
  libroken18-heimdal librtmp1 libsasl2-2 libsasl2-modules libsasl2-modules-db
  libsqlite3-0 libssl1.1 libwind0-heimdal libxml2 mime-support openssl
  publicsuffix xz-utils
Suggested packages:
  libsasl2-modules-gssapi-mit | libsasl2-modules-gssapi-heimdal
  libsasl2-modules-ldap libsasl2-modules-otp libsasl2-modules-sql
The following NEW packages will be installed:
  ca-certificates file fuse libasn1-8-heimdal libcurl3-gnutls libfuse2
  libgssapi3-heimdal libhcrypto4-heimdal libheimbase1-heimdal
  libheimntlm0-heimdal libhx509-5-heimdal libicu60 libkrb5-26-heimdal
  libldap-2.4-2 libldap-common libmagic-mgc libmagic1 libnghttp2-14 libpsl5
  libroken18-heimdal librtmp1 libsasl2-2 libsasl2-modules libsasl2-modules-db
  libsqlite3-0 libssl1.1 libwind0-heimdal libxml2 mime-support openssl
  publicsuffix s3fs xz-utils
0 upgraded, 33 newly installed, 0 to remove and 33 not upgraded.
Need to get 13.3 MB of archives.
After this operation, 52.2 MB of additional disk space will be used.

(Reading database ...

Preparing to unpack .../00-libssl1.1_1.1.0g-2ubuntu4.3_amd64.deb ...
Unpacking libssl1.1:amd64 (1.1.0g-2ubuntu4.3) ...
Selecting previously unselected package openssl.
Preparing to unpack .../01-openssl_1.1.0g-2ubuntu4.3_amd64.deb ...
Unpacking openssl (1.1.0g-2ubuntu4.3) ...
Selecting previously unselected package ca-certificates.
Preparing to unpack .../02-ca-certificates_20180409_all.deb ...
Unpacking ca-certificates (20180409) ...
Selecting previously unselected package libmagic-mgc.
Preparing to unpack .../03-libmagic-mgc_1%3a5.32-2ubuntu0.1_amd64.deb ...
Unpacking libmagic-mgc (1:5.32-2ubuntu0.1) ...
Selecting previously unselected package libmagic1:amd64.
Preparing to unpack .../04-libmagic1_1%3a5.32-2ubuntu0.1_amd64.deb ...
Unpacking libmagic1:amd64 (1:5.32-2ubuntu0.1) ...
Selecting previously unselected package file.
Preparing to unpack .../05-file_1%3a5.32-2ubuntu0.1_amd64.deb ...
Unpacking file (1:5.32-2ubuntu0.1) ...
Selecting previously unselected package libicu60:amd64.
Preparing to unpack .../06-libicu60_60.2-3ubuntu3_amd64.deb ...
Unpacking libicu60:amd64 (60.2-3ubuntu3) ...
Selecting previously unselected package libsqlite3-0:amd64.
Preparing to unpack .../07-libsqlite3-0_3.22.0-1_amd64.deb ...
Unpacking libsqlite3-0:amd64 (3.22.0-1) ...
Selecting previously unselected package libxml2:amd64.
Preparing to unpack .../08-libxml2_2.9.4+dfsg1-6.1ubuntu1.2_amd64.deb ...
Unpacking libxml2:amd64 (2.9.4+dfsg1-6.1ubuntu1.2) ...
Selecting previously unselected package mime-support.
Preparing to unpack .../09-mime-support_3.60ubuntu1_all.deb ...
Unpacking mime-support (3.60ubuntu1) ...
Selecting previously unselected package xz-utils.
Preparing to unpack .../10-xz-utils_5.2.2-1.3_amd64.deb ...
Unpacking xz-utils (5.2.2-1.3) ...
Selecting previously unselected package libfuse2:amd64.
Preparing to unpack .../11-libfuse2_2.9.7-1ubuntu1_amd64.deb ...
Unpacking libfuse2:amd64 (2.9.7-1ubuntu1) ...
Selecting previously unselected package fuse.
Preparing to unpack .../12-fuse_2.9.7-1ubuntu1_amd64.deb ...
Unpacking fuse (2.9.7-1ubuntu1) ...
Selecting previously unselected package libpsl5:amd64.
Preparing to unpack .../13-libpsl5_0.19.1-5build1_amd64.deb ...
Unpacking libpsl5:amd64 (0.19.1-5build1) ...
Selecting previously unselected package publicsuffix.
Preparing to unpack .../14-publicsuffix_20180223.1310-1_all.deb ...
Unpacking publicsuffix (20180223.1310-1) ...
Selecting previously unselected package libroken18-heimdal:amd64.
Preparing to unpack .../15-libroken18-heimdal_7.5.0+dfsg-1_amd64.deb ...
Unpacking libroken18-heimdal:amd64 (7.5.0+dfsg-1) ...
Selecting previously unselected package libasn1-8-heimdal:amd64.
Preparing to unpack .../16-libasn1-8-heimdal_7.5.0+dfsg-1_amd64.deb ...
Unpacking libasn1-8-heimdal:amd64 (7.5.0+dfsg-1) ...
Selecting previously unselected package libheimbase1-heimdal:amd64.
Preparing to unpack .../17-libheimbase1-heimdal_7.5.0+dfsg-1_amd64.deb ...
Unpacking libheimbase1-heimdal:amd64 (7.5.0+dfsg-1) ...
Selecting previously unselected package libhcrypto4-heimdal:amd64.
Preparing to unpack .../18-libhcrypto4-heimdal_7.5.0+dfsg-1_amd64.deb ...
Unpacking libhcrypto4-heimdal:amd64 (7.5.0+dfsg-1) ...
Selecting previously unselected package libwind0-heimdal:amd64.
Preparing to unpack .../19-libwind0-heimdal_7.5.0+dfsg-1_amd64.deb ...
Unpacking libwind0-heimdal:amd64 (7.5.0+dfsg-1) ...
Selecting previously unselected package libhx509-5-heimdal:amd64.
Preparing to unpack .../20-libhx509-5-heimdal_7.5.0+dfsg-1_amd64.deb ...
Unpacking libhx509-5-heimdal:amd64 (7.5.0+dfsg-1) ...
Selecting previously unselected package libkrb5-26-heimdal:amd64.
Preparing to unpack .../21-libkrb5-26-heimdal_7.5.0+dfsg-1_amd64.deb ...
Unpacking libkrb5-26-heimdal:amd64 (7.5.0+dfsg-1) ...
Selecting previously unselected package libheimntlm0-heimdal:amd64.
Preparing to unpack .../22-libheimntlm0-heimdal_7.5.0+dfsg-1_amd64.deb ...
Unpacking libheimntlm0-heimdal:amd64 (7.5.0+dfsg-1) ...
Selecting previously unselected package libgssapi3-heimdal:amd64.
Preparing to unpack .../23-libgssapi3-heimdal_7.5.0+dfsg-1_amd64.deb ...
Unpacking libgssapi3-heimdal:amd64 (7.5.0+dfsg-1) ...
Selecting previously unselected package libsasl2-modules-db:amd64.
Preparing to unpack .../24-libsasl2-modules-db_2.1.27~101-g0780600+dfsg-3ubuntu2_amd64.deb ...
Unpacking libsasl2-modules-db:amd64 (2.1.27~101-g0780600+dfsg-3ubuntu2) ...
Selecting previously unselected package libsasl2-2:amd64.
Preparing to unpack .../25-libsasl2-2_2.1.27~101-g0780600+dfsg-3ubuntu2_amd64.deb ...
Unpacking libsasl2-2:amd64 (2.1.27~101-g0780600+dfsg-3ubuntu2) ...
Selecting previously unselected package libldap-common.
Preparing to unpack .../26-libldap-common_2.4.45+dfsg-1ubuntu1.1_all.deb ...
Unpacking libldap-common (2.4.45+dfsg-1ubuntu1.1) ...
Selecting previously unselected package libldap-2.4-2:amd64.
Preparing to unpack .../27-libldap-2.4-2_2.4.45+dfsg-1ubuntu1.1_amd64.deb ...
Unpacking libldap-2.4-2:amd64 (2.4.45+dfsg-1ubuntu1.1) ...
Selecting previously unselected package libnghttp2-14:amd64.
Preparing to unpack .../28-libnghttp2-14_1.30.0-1ubuntu1_amd64.deb ...
Unpacking libnghttp2-14:amd64 (1.30.0-1ubuntu1) ...
Selecting previously unselected package librtmp1:amd64.
Preparing to unpack .../29-librtmp1_2.4+20151223.gitfa8646d.1-1_amd64.deb ...
Unpacking librtmp1:amd64 (2.4+20151223.gitfa8646d.1-1) ...
Selecting previously unselected package libcurl3-gnutls:amd64.
Preparing to unpack .../30-libcurl3-gnutls_7.58.0-2ubuntu3.5_amd64.deb ...
Unpacking libcurl3-gnutls:amd64 (7.58.0-2ubuntu3.5) ...
Selecting previously unselected package libsasl2-modules:amd64.
Preparing to unpack .../31-libsasl2-modules_2.1.27~101-g0780600+dfsg-3ubuntu2_amd64.deb ...
Unpacking libsasl2-modules:amd64 (2.1.27~101-g0780600+dfsg-3ubuntu2) ...
Selecting previously unselected package s3fs.
Preparing to unpack .../32-s3fs_1.82-1_amd64.deb ...
Unpacking s3fs (1.82-1) ...
Setting up libicu60:amd64 (60.2-3ubuntu3) ...
Setting up libnghttp2-14:amd64 (1.30.0-1ubuntu1) ...
Setting up mime-support (3.60ubuntu1) ...
Setting up libldap-common (2.4.45+dfsg-1ubuntu1.1) ...
Setting up libpsl5:amd64 (0.19.1-5build1) ...
Setting up libfuse2:amd64 (2.9.7-1ubuntu1) ...
Setting up libsasl2-modules-db:amd64 (2.1.27~101-g0780600+dfsg-3ubuntu2) ...
Setting up libsasl2-2:amd64 (2.1.27~101-g0780600+dfsg-3ubuntu2) ...
Setting up libroken18-heimdal:amd64 (7.5.0+dfsg-1) ...
Setting up librtmp1:amd64 (2.4+20151223.gitfa8646d.1-1) ...
Setting up libxml2:amd64 (2.9.4+dfsg1-6.1ubuntu1.2) ...
Setting up libmagic-mgc (1:5.32-2ubuntu0.1) ...
Setting up libmagic1:amd64 (1:5.32-2ubuntu0.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Setting up publicsuffix (20180223.1310-1) ...
Setting up libssl1.1:amd64 (1.1.0g-2ubuntu4.3) ...
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.26.1 /usr/local/share/perl/5.26.1 /usr/lib/x86_64-linux-gnu/perl5/5.26 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.26 /usr/share/perl/5.26 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)
debconf: falling back to frontend: Teletype
Setting up xz-utils (5.2.2-1.3) ...
update-alternatives: using /usr/bin/xz to provide /usr/bin/lzma (lzma) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/lzma.1.gz because associated file /usr/share/man/man1/xz.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/unlzma.1.gz because associated file /usr/share/man/man1/unxz.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/lzcat.1.gz because associated file /usr/share/man/man1/xzcat.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/lzmore.1.gz because associated file /usr/share/man/man1/xzmore.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/lzless.1.gz because associated file /usr/share/man/man1/xzless.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/lzdiff.1.gz because associated file /usr/share/man/man1/xzdiff.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/lzcmp.1.gz because associated file /usr/share/man/man1/xzcmp.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/lzgrep.1.gz because associated file /usr/share/man/man1/xzgrep.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/lzegrep.1.gz because associated file /usr/share/man/man1/xzegrep.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/lzfgrep.1.gz because associated file /usr/share/man/man1/xzfgrep.1.gz (of link group lzma) doesn't exist
Setting up libheimbase1-heimdal:amd64 (7.5.0+dfsg-1) ...
Setting up openssl (1.1.0g-2ubuntu4.3) ...
Setting up libsqlite3-0:amd64 (3.22.0-1) ...
Setting up libsasl2-modules:amd64 (2.1.27~101-g0780600+dfsg-3ubuntu2) ...
Setting up ca-certificates (20180409) ...
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.26.1 /usr/local/share/perl/5.26.1 /usr/lib/x86_64-linux-gnu/perl5/5.26 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.26 /usr/share/perl/5.26 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)
debconf: falling back to frontend: Teletype
Updating certificates in /etc/ssl/certs...
133 added, 0 removed; done.
Setting up fuse (2.9.7-1ubuntu1) ...
Setting up libwind0-heimdal:amd64 (7.5.0+dfsg-1) ...
Setting up libasn1-8-heimdal:amd64 (7.5.0+dfsg-1) ...
Setting up libhcrypto4-heimdal:amd64 (7.5.0+dfsg-1) ...
Setting up file (1:5.32-2ubuntu0.1) ...
Setting up libhx509-5-heimdal:amd64 (7.5.0+dfsg-1) ...
Setting up libkrb5-26-heimdal:amd64 (7.5.0+dfsg-1) ...
Setting up libheimntlm0-heimdal:amd64 (7.5.0+dfsg-1) ...
Setting up libgssapi3-heimdal:amd64 (7.5.0+dfsg-1) ...
Setting up libldap-2.4-2:amd64 (2.4.45+dfsg-1ubuntu1.1) ...
Setting up libcurl3-gnutls:amd64 (7.58.0-2ubuntu3.5) ...
Setting up s3fs (1.82-1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Processing triggers for ca-certificates (20180409) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
+ cp /root/bin/s3fs /host/usr/local/bin/
cp: cannot create regular file '/host/usr/local/bin/': Not a directory

I was taking a look at ffdl/ibmcloud-object-storage-deployer:v0.1 on dockerhub, but no dockerfile there. Thanks in advance for any ideas.

@sboagibm @fplk Please look into this

GKE should use Container-Optimized OS underneath, cmp. https://cloud.google.com/container-optimized-os/ and it is possible the open source driver will not work without modification on that. If you want to deploy to GKE, you would have to first make sure https://github.com/IBM/ibmcloud-object-storage-plugin works. Since I don't have access to GKE, I cannot test or fix this for you.
Regarding the general FfDL setup, it should cleanly deploy against IBM Cloud. Unfortunately, I have two hard deadlines at the end of the week, so I cannot look into deployment on Minikube and DIND right now. DIND 1.10 worked a while ago, I briefly tried to deploy against 1.12 not too long ago and also ran into problems.

Thanks for the advice of locating the issue, still on trying.

@JunFugithub For minikube could you look for the statefulset that is created and do a kubectl get ss/xxxxx -o yaml and send results?  And for DIND do the same, but also do a kubectl describe for the failed pod?

I apologize for the delay due to the holidays. I think I can reproduce the error you encountered and have been able to get it working. A couple of things:
a) The scripts in https://github.com/IBM/FfDL/tree/master/bin/dind_scripts should largely work with the exception that you need to update DIND in launch_kubernetes.sh (RawGit is deprecated and 1.9 is old, so I successfully used https://github.com/kubernetes-sigs/kubeadm-dind-cluster/releases/download/v0.1.0/dind-cluster-v1.13.sh - just replace all occurrences in the file accordingly)
b) Here are my steps:
ssh<EMAIL_ADDRESS>apt install -y git software-properties-common
mkdir -p /home/ffdlr/go/src/github.com/IBM/ && cd $_ && git clone https://github.com/IBM/FfDL.git && cd FfDL
# Replace DIND version as explained in (a)
cd bin/dind_scripts/
chmod +x create_user.sh
. create_user.sh
# Enter new password and get kicked out

ssh<EMAIL_ADDRESS>cd /home/ffdlr/go/src/github.com/IBM/FfDL/bin/dind_scripts/
sudo chmod +x experimental_master.sh
. experimental_master.sh

Build own manifest with:
name: tf_convolutional_network_tutorial
description: Convolutional network model using tensorflow
version: "1.0"
gpus: 0
cpus: 0.5
memory: 1Gb
learners: 1

# Object stores that allow the system to retrieve training data.
data_stores:
  - id: sl-internal-os
    type: mount_cos
    training_data:
      container: REPLACE_INPUT_BUCKET
    training_results:
      container: REPLACE_OUTPUT_BUCKET
    connection:
      auth_url: REPLACE_ENDPOINT
      user_name: REPLACE_KEY_ID
      password: REPLACE_KEY

framework:
  name: tensorflow
  version: "1.5.0-py3"
  command: >
    python3 convolutional_network.py --trainImagesFile ${DATA_DIR}/train-images-idx3-ubyte.gz
      --trainLabelsFile ${DATA_DIR}/train-labels-idx1-ubyte.gz --testImagesFile ${DATA_DIR}/t10k-images-idx3-ubyte.gz
      --testLabelsFile ${DATA_DIR}/t10k-labels-idx1-ubyte.gz --learningRate 0.001
      --trainingIters 2000
  # Change trainingIters to 20000 if you want your model to have over 80% Accuracy rate.

evaluation_metrics:
  type: tensorboard
  in: "$JOB_STATE_DIR/logs/tb"
  # (Eventual) Available event types: 'images', 'distributions', 'histograms', 'images'
  # 'audio', 'scalars', 'tensors', 'graph', 'meta_graph', 'run_metadata'
  #  event_types: [scalars]

Run with DLAAS_URL=http://<IP_ADDRESS>:31826 DLAAS_USERNAME=test-user DLAAS_PASSWORD=test /home/ffdlr/go/src/github.com/IBM/FfDL/cli/bin/ffdl-linux train mymanifest.yml . from within /home/ffdlr/go/src/github.com/IBM/FfDL/etc/examples/tf-model
c) This should work, but is currently not the most user-friendly way of setting things up. I can try to push some changes to improve usability - it looks from the outside like the manifest creation gets garbled up somewhere, but until then this should get you running.
d) Minikube is a suboptimal environment due to unfixed storage bugs on their side. FfDL should be able to run against GKE in principle, but I'm not sure the open source S3 driver will work against that. I think the storage team supports DIND and IBM Cloud and their architecture should work against any cloud provider, but you would have to test the driver manually and PR minor changes if it does not work against your target provider out of the box. Or use a different driver.

OK, with https://github.com/IBM/FfDL/pull/158 I can deploy FfDL with the typical 4 commands from the README against DIND on macOS and Linux as well as IBM Cloud. Please let me know if it helps.


error test

I'm sorry about that. I will upload the test code for the comparison methods within a week.

I'm sorry about that. I will upload the test code for the comparison methods within a week.
if possible can you guide me how to run pretrain mode solar? .I tried to test solar model and got the same error as this token. thanks

I just update the network part and the test code, you can try it now. You need to modify the model and load the corresponding model weights.


[SPARK-41532][CONNECT][CLIENT] Add check for operations that involve multiple data frames

What changes were proposed in this pull request?
Add check for operations that involve multiple data frames,  because spark do not support joining for example two data frames from different Spark Connect Sessions.

Why are the changes needed?
Spark do not support joining for example two data frames from different Spark Connect Sessions. To avoid exceptions, the client should clearly fail when it tries to construct such a composition.

Does this PR introduce any user-facing change?
No

How was this patch tested?
Add new test

LGTM

@HyukjinKwon @amaliujia @hvanhovell Hi, I add support for python. PTAL again. Thanks!

@HyukjinKwon @hvanhovell Hi, kindly ask, can merge this PR now?

kindly ping @HyukjinKwon

I am fine. but I would defer to @hvanhovell to merge.


fix(learn): remove white space for quick fix

Checklist:

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[x] All the files I changed are in the same world language, for example: only English changes, or only Chinese changes, etc.

Related to: #39925 (does not close)

Sorry for pushing this branch to the wrong repo. Should I close it and delete?

I think it should be okay. 🤔


When "Unsuppressed" and "Suppressed" control receives the keyboard focus, the color contrast ratio of border with respect to background is less than 3:1.

Actual:
When "Unsuppressed" and "Suppressed" control receives the keyboard focus, the color contrast ratio of border with respect to background is 1.627:1 which is less than 3:1.
Expected:
When "Unsuppressed" and "Suppressed" control receives the keyboard focus, the color contrast ratio of border with respect to background should be greater than or equal to 3:1.

I see a dark blue border with contrast ratio of 4.53:1.

Note: I am not referring to the decorative light blue "glow".

@jeffersonking
When we expand this "Suppression" control using enter two list items appears and when we move the keyboard focus on those two list items "Unsuppressed" and "Suppressed" then the color contrast ratio of keyboard focus border on "Unsuppressed" and "Suppressed" control with respect to background is less than 3:1.

Fixed in v0.5 (2413b17b8cf2f5a11cf4238df5418f00eddd43a1). Thanks for reporting!

Fixed in v0.5 (2413b17b8cf2f5a11cf4238df5418f00eddd43a1). Thanks for reporting!


result.get_ran_qasm() not working

What is the current behavior?
Result.get_ran_qasm() seem to be broken due to PR #501. That's because the Result datastructure no longer has a copy of the qobj json inside it. We should see how/if this can be fixed with the recent introduction of Qobj class.
Steps to reproduce the problem
from qiskit import *
q = QuantumRegister(3)
c = ClassicalRegister(3)
circ = QuantumCircuit(q, c)
circ.ccx(q[0], q[1], q[2])
print(circ.qasm())

result = execute(circ, 'local_qasm_simulator').result()
result.get_ran_qasm(circ.name)  # does not return an answer

When we move to qobj why do we still want this option. This should be the output of the transpiler before qobj. Or are you thinking we make a qobj to qasm file

Well, while implementing I realize two cases:
If you use the wrapper shortcut execute(), you receive a BaseJob instance which does not expose the compiled QASM. The solution would be exposing the Qobj as a read-only property. Thoughts?
Is it guaranteed that the backend will always execute the QASM sent in the Qobj? If not, you need the backend to return the actual QASM it ran.

This seems related to #680, where running compile returns a Qobj with an incorrect qasm string, and then fails to run on the device.

@delapuente I raised this bug mainly because the method is exposed in our API, but not working correctly at the moment.
Going forward, QASM will not be included in Qobj at all (it is only included now because devices run QASM, and can't run Qobj directly).
But even after we work with pure Qobj, it will still be very useful to see what Qobj was run after the run finished. For this, I think there should be a method BaseJob#get_ran_qobj() which returns the Qobj that was run in that Job.
Then we need two converter methods for converting from Qobj -> QASM and Qobj -> QuantumCircuit.
Does that make sense?

Agree. I would simplify the method name, say get_qobj() or a qobj property but that's all.


Guide to Publish Bot with Node on Azure?

On the page: http://docs.botframework.com/connector/getstarted/#publishing-your-bot-application-to-microsoft-azure
I can find the doc to publish bot using c#, I don't think there is a doc detailing how to publish app in Node.

I personnaly used a private git repo and enabled continuous deployment from this repository on an azure website. Azure then automatically install modules and dependencies if it founds a "package.json" file. Need a detailed explanation ?

@danmarshall did you have a sample for publishing node to azure?

here is the guid for deploying to azure.

@msft-shahins Can we add this link to the link I added in the description?


Docker Registry 2.0.1 and NGI Proxy with LDAP Authentication Issue on GET /v1/users

Hello,
I'm facing some problems to setup a docker registry with an active directory.
I've setup with success the authentication with active directory, but after login to registry this fails with these messages:
On docker side:
registry_1 | time="2015-08-05T14:08:52.073707203Z" level=info msg="endpoint local-8082 disabled, skipping" environment=development instance.id=2952161b-4a43-41ae-8799-7e8f7dd55e48 service=registry version=v2.0.1
registry_1 | time="2015-08-05T14:08:52.073774138Z" level=info msg="endpoint local-8083 disabled, skipping" environment=development instance.id=2952161b-4a43-41ae-8799-7e8f7dd55e48 service=registry version=v2.0.1
registry_1 | time="2015-08-05T14:08:52.073821358Z" level=info msg="using inmemory layerinfo cache" environment=development instance.id=2952161b-4a43-41ae-8799-7e8f7dd55e48 service=registry version=v2.0.1
registry_1 | time="2015-08-05T14:08:52.073843722Z" level=info msg="listening on :5000" environment=development instance.id=2952161b-4a43-41ae-8799-7e8f7dd55e48 service=registry version=v2.0.1
registry_1 | time="2015-08-05T14:08:52.073924198Z" level=info msg="debug server listening localhost:5001"
nginx_1    | nginx: [emerg] http_auth_ldap: parse_require in /etc/nginx/nginx.conf:22
nginx_1    | nginx: [emerg] http_auth_ldap: Setting satisfy all in /etc/nginx/nginx.conf:23
registry_1 | time="2015-08-05T14:09:06.38493673Z" level=info msg="response completed" environment=development http.request.host=localhost http.request.id=88e7ecdf-1306-4b7c-be5c-c2d8aa64c0ac http.request.method=GET http.request.remoteaddr=<IP_ADDRESS> http.request.uri="/v1/_ping" http.request.useragent="docker/1.7.1 go/go1.4.2 git-commit/786b29d kernel/3.19.0-26-generic os/linux arch/amd64" http.response.contenttype="text/plain; charset=utf-8" http.response.duration="115.795µs" http.response.status=404 http.response.written=19 instance.id=2952161b-4a43-41ae-8799-7e8f7dd55e48 service=registry version=v2.0.1
registry_1 | <IP_ADDRESS> - - [05/Aug/2015:14:09:06 +0000] "GET /v1/_ping HTTP/1.0" 404 19 "" "docker/1.7.1 go/go1.4.2 git-commit/786b29d kernel/3.19.0-26-generic os/linux arch/amd64"
registry_1 | time="2015-08-05T14:09:06.402938569Z" level=info msg="response completed" environment=development http.request.host=localhost http.request.id=a7ae38ac-bb89-43e5-ac2e-588e6594254e http.request.method=GET http.request.remoteaddr=<IP_ADDRESS> http.request.uri="/v1/users/" http.request.useragent="docker/1.7.1 go/go1.4.2 git-commit/786b29d kernel/3.19.0-26-generic os/linux arch/amd64" http.response.contenttype="text/plain; charset=utf-8" http.response.duration="112.529µs" http.response.status=404 http.response.written=19 instance.id=2952161b-4a43-41ae-8799-7e8f7dd55e48 service=registry version=v2.0.1
registry_1 | <IP_ADDRESS> - - [05/Aug/2015:14:09:06 +0000] "GET /v1/users/ HTTP/1.0" 404 19 "" "docker/1.7.1 go/go1.4.2 git-commit/786b29d kernel/3.19.0-26-generic os/linux arch/amd64"
On User side after a docker login:
Username: test
Password:
Email<EMAIL_ADDRESS>Error response from daemon: Login: 404 page not found
(Code: 404; Headers: map[Server:[nginx/1.7.12] Date:[Wed, 05 Aug 2015 14:09:06 GMT] Content-Type:[text/plain; charset=utf-8] Content-Length:[19] Connection:[keep-alive] Docker-Distribution-Api-Version:[registry/2.0 registry/2.0.1]])
Then I cannot access to registry.
My nginx.conf is:
###########################
worker_processes  1;
events {
worker_connections  1024;
}
http {
upstream docker-registry {
    server docker-registry:5000;
}

ldap_server ldapserver {
url ldap://**********:389/OU=Users,OU=*******,DC=*******,DC=it?samaccountname?sub?(objectClass=user);
    #binddn "cn=admin,dc=example,dc=com";
    binddn ************;
    binddn_passwd **********
    group_attribute uniquemember;
    group_attribute_is_dn on;
    #require group 'cn=docker,ou=groups,dc=example,dc=com';
    require valid_user;
    satisfy all;
}

server {

    listen 443;
    server_name *************;

    error_log /var/log/nginx/error.log debug;
    access_log /var/log/nginx/access.log;

    ssl on;
    ssl_certificate /etc/ssl/docker/docker-registry.crt;
    ssl_certificate_key /etc/ssl/docker/docker-registry.key;

    client_max_body_size 0;

    chunked_transfer_encoding on;

add_header Docker-Distribution-Api-Version registry/2.0.1 always;

    location / {
        auth_ldap "Forbidden";
        auth_ldap_servers ldapserver;
        include docker-registry.conf;
    }

    location ~* /v1/repositories/(?<namespace>([^/]*))/ {
        auth_ldap "Forbidden";
        auth_ldap_servers ldapserver;
        set $deny_write_request "";
        if ($request_method = PUT) {
            set $deny_write_request "W";
        }
        if ($request_method = DELETE) {
            set $deny_write_request "W";
        }
        if ($remote_user != $namespace) {
            set $deny_write_request "${deny_write_request}A";
        }
        if ($deny_write_request = 'WA') {
            return 401;
        }
        include docker-registry.conf;
    }

location /v2/ {
    	# Do not allow connections from docker 1.5 and earlier
    	# docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
    	if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
          		return 404;
    	}

        	auth_ldap "Forbidden";
        	auth_ldap_servers ldapserver;
	#auth_basic off;
        	include docker-registry.conf;
}

    location /_ping {
        auth_basic off;
        include docker-registry.conf;
    }

    location /v1/_ping {
        auth_basic off;
        include docker-registry.conf;
    }

    location /v1/search {
        auth_basic off;
        include docker-registry.conf;
    }

}

}
#######################
docker-registry.conf  is
#####################################
proxy_pass                       http://docker-registry;
proxy_set_header  Docker-Distribution-Api-Version registry/2.0.1;
proxy_set_header  Host           $http_host;   # required for docker client's sake
proxy_set_header  X-Real-IP      $remote_addr; # pass on real client's IP
proxy_set_header  Authorization  ""; # see https://github.com/dotcloud/docker-registry/issues/170
proxy_read_timeout               900;
#####################################
Any suggestions?

Hi,
I strongly recommend you move away from registry:1 - or at least first try to make things work for registry:2 only.
Try to follow the instructions in the official documentation, specifically: https://github.com/docker/distribution/blob/master/docs/authentication.md
and see if you can setup basic auth successfully.
Then when you are done, add ldap.

Hello,
How I can move away from registry:1 ? I'm already using image registry:2.0.1.
Here my docker-compose file:
################################
nginx:
image: h3nrik/registry-ldap-auth
ports:
- "443:443"
links:
- registry:docker-registry
volumes:
- $PWD:/etc/ssl/docker:ro
- $PWD:/etc/nginx:ro
restart: always
registry:
image: registry:2.0.1
hostname: "docker-registry"
volumes:
- /docker_images:/tmp/registry-dev
ports:
- "5000"
restart: always
#############################
Moreover your link in your response is not valid.

Hi @deviscalio
The documentation page was moved.
Have a look here: https://github.com/docker/distribution/blob/master/docs/nginx.md
And if you don't want to use v1, please remove the v1 sections from your configuration (or start from the exemple in the documentation).

Many thanks Oliver.
When you say "... please remove the v1 sections from your configuration", do you mean remove from nginx configuration or from registry configuration? In the second case which is this file?

@deviscalio I suggest you start from scratch, using the documentation I linked.
Once you get it working, you move ahead and add your LDAP snippets.

Hello Olivier,
I've followed the guide and I've had a successful installation of registry.
Then, as you suggested, I've added LDAP configuration, but of course the nginx image is wrong because  I've an error : "unknown directive "ldap_server".
I've tried with h3nrik/registry-ldap-auth but now I've this error:
Error response from daemon: no successful auth challenge for https://localhost:5043/v2/ - errors: [basic auth attempt to https://localhost:5043/v2/ realm "Forbidden" failed with status: 401 Unauthorized]
I've substituted in registry.conf:
auth_basic "Registry realm";
auth_basic_user_file /etc/nginx/conf.d/htpasswd;
with
auth_ldap "Forbidden";
auth_ldap_servers ldapserver;

Hello Oliver,
I've resolved!
I put here the ngix.conf used, hoping it helps someone:
###########################################
worker_processes  1;
events {
worker_connections  1024;
}
http {
upstream docker-registry {
    server docker-registry:5000;
}

ldap_server ldapserver {
url ldap://xxxxxxxxx:389/OU=Users,DC=xxxxx,DC=it?samaccountname?sub?(objectClass=user);
    binddn<EMAIL_ADDRESS>    binddn_passwd xxxxxxxxxxxx
    group_attribute uniquemember;
    group_attribute_is_dn on;
}

server {

    listen 443;
    server_name xxxxxxxxxxxxxxxxxx;

    error_log /var/log/nginx/error.log debug;
    access_log /var/log/nginx/access.log;

    ssl on;
    ssl_certificate /etc/ssl/docker/docker-registry.crt;
    ssl_certificate_key /etc/ssl/docker/docker-registry.key;

    client_max_body_size 0;

    chunked_transfer_encoding on;

location /v2/ {
    	# Do not allow connections from docker 1.5 and earlier
    	# docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
    	if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
          		return 404;
    	}

            auth_ldap "Forbidden";
        	auth_ldap_servers ldapserver;
            add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;

	proxy_pass                          http://docker-registry;
    	proxy_set_header  Host              $http_host;   # required for docker client's sake
    	proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
    	proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
    	proxy_set_header  X-Forwarded-Proto $scheme;
    	proxy_read_timeout                  900;
}
}

}
###########################################
And docker-compose.yml used:
###########################################
nginx:
image: h3nrik/registry-ldap-auth
ports:
- "443:443"
links:
- registry:docker-registry
volumes:
- $PWD/certs:/etc/ssl/docker:ro
- $PWD/conf:/etc/nginx:ro
restart: always
registry:
image: registry:2.0
hostname: "docker-registry"
volumes:
- /docker_images:/tmp/registry-dev
ports:
- "5000"
restart: always
###########################################

@deviscalio : Could you please help me with the link you followed.
I am trying to setup private registry authentication with SSL self signed certificates + LDAP(not a container, existing server ).
Basic authentication using SSL is working fine. But unable to figure out configurations with LDAP.

Hi @deviscalio
i just have got this error. Could you tell me how to resolve this problem?
Thank you.

https://github.com/tierratelematics/existing-ldap-docker-registry

https://github.com/tierratelematics/existing-ldap-docker-registry

Hi @deviscalio
I'm trying to use Your image, but I got that error:
[root@D-L-TOOLS docker]# docker-compose up -d
Recreating b06951df6d83_b06951df6d83_b06951df6d83_docker_registry_1
ERROR: for registry  No such image: sha256:34bccec547938a300eb4a3dc8161ff71ec40ccddc8341961ddcf57fcbf8a73f5
Traceback (most recent call last):
File "", line 3, in
File "compose/cli/main.py", line 63, in main
AttributeError: 'ProjectError' object has no attribute 'msg'
docker-compose returned -1
Somebody erased some of images? How can I obey it? If I check docker images I have h3nrik/registry-ldap-auth and registry 2.3.1 downloaded. The only thing which I have changed in Your files were the paths that You mentioned here:
https://github.com/tierratelematics/existing-ldap-docker-registry
###############################################################################
My docker-compose.yml file looks like this:
nginx:
image: h3nrik/registry-ldap-auth
ports:
- "443:443"
links:
- registry:docker-registry
volumes:
- $PWD/certs:/etc/ssl/docker:ro
- $PWD/conf:/etc/nginx:ro
restart: always
registry:
image: registry:2.3.1
hostname: "docker-registry"
environment:
- REGISTRY_DELETE_ENABLED=true
volumes:
- /data/docker_registry/docker:/var/lib/registry
ports:
- "5000"
restart: always
###############################################################################
Nginx conf looks like:
worker_processes  1;
events {
worker_connections  1024;
}
http {
upstream docker-registry {
    server docker-registry:5000;
}

ldap_server ldapserver {
    url "ldaps://xxxxxxxxxxxxxxx:636/OU=xxxx,DC=xxxx,DC=local?sAMAccountName?sub?(&(memberOf:1.2.840.1135<IP_ADDRESS>1:=cn=xxxxxx,OU=User_Groups,OU=Groups,OU=GLOB000,OU=Global,OU=ROOT,DC=xxxx,DC=local)(objectClass=person))" SSL;
    binddn "CN=xxxxxxxxxxx,OU=Service_Accounts,OU=Admins,OU=GLOB000,OU=Global,OU=xxxx,DC=xxxxxx,DC=local";
    binddn_passwd xxxxxxxxxxxxxx
    group_attribute uniquemember;
    group_attribute_is_dn on;
}

server {

    listen 443;
    server_name <my-server-name>;

    error_log /var/log/nginx/error.log debug;
    access_log /var/log/nginx/access.log;

    ssl on;
    ssl_certificate /etc/ssl/docker/docker-registry.crt;
    ssl_certificate_key /etc/ssl/docker/docker-registry.key;

    client_max_body_size 0;

    chunked_transfer_encoding on;

    location / {
            return 301 https://<my-server-name>/v2;
    }

    location /v2/ {
            # Do not allow connections from docker 1.5 and earlier
            # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
            if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
                    return 404;
            }

            auth_ldap "Forbidden";
            auth_ldap_servers ldapserver;
            add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;

            proxy_pass                          http://docker-registry;
            proxy_set_header  Host              $http_host;   # required for docker client's sake
            proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
            proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
            proxy_set_header  X-Forwarded-Proto $scheme;
            proxy_read_timeout                  900;
    }
}

}
###############################################################################
I have also overwritten my certs to Your certs catalog. It should work, but some container could mess things up or missing? I have removed with docker rmi -f all unused images of registry and nginx servers...
Please help.
Bartosz

I found the problem (I should remove all the containers that were in memory), but another one came out. During the last part - when I try to log in to server I have:
docker login https://<my_address>.local
Username: <my_user>
Password:
Error response from daemon: Missing client certificate domain.cert for key domain.key
Seems like my self-signed certs are not visible for environment. But the paths are ok:
From Nginx conf:
ssl_certificate /etc/ssl/docker/docker-registry.crt;
ssl_certificate_key /etc/ssl/docker/docker-registry.key;
From docker-compose.yml:
volumes:
- /data/docker_registry/docker/certs:/etc/ssl/docker:ro
- /data/docker_registry/docker/conf:/etc/nginx:ro
[root@D-L-TOOLS certs]# pwd
/data/docker_registry/docker/certs
[root@D-L-TOOLS certs]# ll
total 8
-rwx------. 1 root root 2187 Jun  1 09:56 docker-registry.crt
-rwx------. 1 root root 3272 Jun  1 09:56 docker-registry.key
Still there is something wrong:/

Try to connect to nginx container:
docker exec -it <container_id> /bin/bash
and check if file are present.

Hi,
Here is an output from nginx container certificates path:
root@3e2e82d415bd:/etc/ssl/docker# ls -la
total 12
drwx------. 2 root root 4096 Jun  1 06:51 .
drwxr-xr-x. 5 root root   63 Jun  1 12:50 ..
-rwx------. 1 root root 2187 Jun  1 06:56 docker-registry.crt
-rwx------. 1 root root 3272 Jun  1 06:56 docker-registry.key
root@3e2e82d415bd:/etc/ssl/docker# pwd
/etc/ssl/docker
Checked the docker-registry.crt file and it seems ok. The cert is the same as in the /data/docker_registry/docker/certs on my server.
I saw also that I got that error:
Error response from daemon: Missing client certificate domain.cert for key domain.key
when I login with correct and wrong credentials...

Got the same when I'm trying to push image to my registry:
docker push d-l-xxxxx.xxxxx.local:5000/ubuntu
The push refers to a repository [d-l-xxxxxx.xxxxx.local:5000/ubuntu]
Get https://d-l-tools.ocnet.local:5000/v1/_ping: dial tcp <IP_ADDRESS>:5000: getsockopt: connection refused
docker push d-l-xxxxx.xxxxxx.local/ubuntu
Error response from daemon: Missing client certificate domain.cert for key domain.key

Hi Again,
I have added to configuration two lines (where ldap.pem is CAcertificate from LDAP server):
ssl_check_cert off;
ssl_ca_file /etc/ssl/docker/ldap.pem;

And now I have something like this:
docker-compose up
Starting docker_registry_1
Starting docker_nginx_1
Attaching to docker_registry_1, docker_nginx_1
registry_1  | time="2016-06-02T09:21:26Z" level=warning msg="Ignoring unrecognized environment variable REGISTRY_DELETE_ENABLED"
registry_1  | time="2016-06-02T09:21:26Z" level=warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable." go.version=go1.5.3 instance.id=033e45be-ee04-46c1-acc3-f813ddde18fb version=v2.3.1
registry_1  | time="2016-06-02T09:21:26Z" level=info msg="redis not configured" go.version=go1.5.3 instance.id=033e45be-ee04-46c1-acc3-f813ddde18fb version=v2.3.1
registry_1  | time="2016-06-02T09:21:26Z" level=info msg="using inmemory blob descriptor cache" go.version=go1.5.3 instance.id=033e45be-ee04-46c1-acc3-f813ddde18fb version=v2.3.1
registry_1  | time="2016-06-02T09:21:26Z" level=info msg="listening on [::]:5000" go.version=go1.5.3 instance.id=033e45be-ee04-46c1-acc3-f813ddde18fb version=v2.3.1
registry_1  | time="2016-06-02T09:21:26Z" level=info msg="Starting upload purge in 18m0s" go.version=go1.5.3 instance.id=033e45be-ee04-46c1-acc3-f813ddde18fb version=v2.3.1
nginx_1     | 2016/06/02 09:18:49 [debug] 13#0: epoll add event: fd:8 op:1 ev:00002001
nginx_1     | 2016/06/02 09:21:27 [debug] 13#0: epoll add event: fd:8 op:1 ev:00002001

But got the same issue: Error response from daemon: Missing client certificate domain.cert for key domain.key
When I use it as that:
ssl_check_cert on;
ssl_ca_file /etc/ssl/docker/ldap.pem;

I have:
nginx_1     | nginx: [emerg] http_auth_ldap: 'ssl_cert_check': cannot verify remote certificate's domain name because your version of OpenSSL is too old. Please install OpenSSL >= 1.02 and recompile nginx. in /etc/nginx/nginx.conf:20
nginx_1     | 2016/06/02 09:25:23 [emerg] 10#0: http_auth_ldap: 'ssl_cert_check': cannot verify remote certificate's domain name because your version of OpenSSL is too old. Please install OpenSSL >= 1.02 and recompile nginx. in /etc/nginx/nginx.conf:20
nginx_1     | 2016/06/02 09:25:23 [debug] 13#0: epoll add event: fd:9 op:1 ev:00002001

But I have downloaded and installed new openssl version:
openssl version
OpenSSL 1.0.2g  1 Mar 2016

Do I need to tell nginx to look for openssl? It's configured to be in a default path:
which openssl
/bin/openssl
I think I'm near the end, as I don't have ldap errors any more...

I wanted to workaround that issue with openssl and missing certs by using:
 ldap_server ldapserver {
         url ldap://xxxxxx.xxxxx.local:389/OU=xxxx,DC=xxxxx,DC=local?sAMAccountName?sub?(&(memberOf:1.2.840.1135<IP_ADDRESS>1:=cn=xxxxxxx,OU=xxxxxx,OU=xxxxxx,OU=xxxxx,OU=xxxxx,OU=xxxxx,DC=xxxxxx,DC=local)(objectClass=person));
        binddn "CN=xxxxxxxx,OU=xxxxxxxx,OU=Admins,OU=xxxxxxx,OU=xxxxxx,OU=xxxxx,DC=xxxxxx,DC=local";
        binddn_passwd xxxxxxxxxxx
        group_attribute uniquemember;
        group_attribute_is_dn on;
        #ssl_check_cert on;
        #ssl_ca_file /etc/ssl/docker/ldap.pem;
        #ssl_ca_dir /etc/ssl/docker/;

I have tested ldap connection with ldapserarch - it's running. Simple docker registry (without unsecure registry option) is working with that self-signed cert (also from other servers)...
But at the end I have been asked for certificates, no matter if I'm login with good or bad credentials:
docker login https://d-l-tools.ocnet.local
Username: xxxxxx
Password:
Error response from daemon: Missing client certificate domain.cert for key domain.key

It seems like docker needs to have address with port on the end, because https:// is not enough. I've managed to login and push images after typing:
docker login https://<my_domain_address>:443
Thank You very much for Your help. My docker works! How can I make docker ask for login everytime somebody wants to push image? On my side there is too long timeout before logout...


help me, I can't make it work, I'm inexperienced.

hi I recently installed MagicMirror and I would like to use your app (or module). I followed what you described and I also looked into various problems. I can't solve. I can say that for example when I try the command from the shell "wget http: // localhost: 8080 / mmm-googlebirthdaysprovider" I get out of connection refused. I checked very well if I had written bad birthdays .. can you help me? I am a newbie in these things and from the day before yesterday I am trying to learn by self-learning. I hope for an answer. thank you so much

Connection refused sounds to me like there is no server listening. Did you start the magicmirror with npm start?
Also, there should always be some output on the terminal where you ran npm start. Please create a config.js where MMM-GoogleBirthdaysProvider is the only loaded module, so we don't get spammed with log entries from other modules. (Either create a new config file and backup your current one, or just comment out all other modules). Then run npm start again and paste the output here. There should be something that will help us :)
Also, self-learning by experimenting with a project like magic mirror is a great way to get started with Linux, command line, programming and all that stuff. In fact, most of the knowledge and skills I have now I gained this way. It will take time however, so stay at it and it will be worth it ;-)

closed due to inactivity


My Contribution?

Hi, Madalin. Would you like me to do anything for your project? I'd like to practice my skills.

Hello, Mark! Thank you!
Stick around. Today I shall make a roadmap so you may know how to contribute.

You have it now.


can't delete error dats

blocked by https://github.com/datproject/multidrive/pull/13

Maybe this is the same issue:
If I delete a dat from the desktop app, and also delete the folder via Finder, then refresh the app, the deleted app comes back. And the folder is created as well.

I think those are different issues. This is about adding a delete button to errored dats and fixing the issue in multidrive.
That's an interesting bug however! I'm opening a separate issue.

pull request here: https://github.com/datproject/dat-desktop/pull/405


Ensure command name suggestions give the closest match

Resolves #210
When obtaining the list of command names from the logic class, it will return the sorted list of commands instead of a set.

Works for all except listing add and listing delete. It will suggest listing but not listing add or listing delete.

I second this. I've noticed this has been an issue for awhile now too.


Update a protocol.html error

Removed two words that indicated that the primitive types of the protocol would be listed here, which is not yet the case.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

FAILURE
7959 tests run, 5 skipped, 1 failed.
--none--

FAILURE
8067 tests run, 5 skipped, 2 failed.
--none--

FAILURE
7959 tests run, 5 skipped, 1 failed.
--none--


Clean/target data are reverberant speech and not clean speech

Hi,
At the specifications of the challenge it is written:

EEE ICASSP 2023 Deep Noise Suppression (DNS) grand challenge is the 5th edition of Microsoft DNS challenges with focus on deep speech enhancement achieved by suppressing background noise, reverberation and neighboring talkers and enhancing the signal quality

Thus, this means that the clean/target speech signals must be without reverberation. Though, at the data creation file, noisyspeech_synthesizer_singleprocess.py and at line 192, we can see that the clean signal is processed and made reverberant.
Can you please elaborate on this? Since the script for data creation comes from the challenge organisers, does the above processing of the clean data (i.e. making the clean speech signals reverberant) mean that the target speech signals are reverberant signals? If yes, then what is meant by

EEE ICASSP 2023 Deep Noise Suppression (DNS) grand challenge is the 5th edition of Microsoft DNS challenges with focus on deep speech enhancement achieved by suppressing background noise, reverberation and neighboring talkers and enhancing the signal quality

?
Thank you in advance for the info.

我也遇到了这个疑问


spring starter can now use all sdk autoconfig properties

Also

annotation dependency is included in starter now
explain advanced config with the example of "exclude actuator endpoints from tracing"
explain resource providers

/fix:format

/fix:format

@theletterf Could you please have another look?

Approved!

I'm a little late, but this is great 🎉

I'm a little late, but this is great 🎉

Never too late for that :smile:


support for androidx

Hey @davecoffin , thank you for your contribution! The fix for this is already available in the new version of the plugin available in the ProPlugins NPM registry.
All future work of this plugin will be available exclusively as part of ProPlugins initiative. So if you want to get the latest updates/fixes/features make sure you subscribe to the service. You get all my plugins plus many of the most used and high quality plugins for a small monthly subscription fee.


Message from nc doesn't get sent to HexChat user when more than 1 arg

Probably an iteration issue.

From hexchat the messages come with ":" sign at the beginning and are parsed as single trailing argument starting from ":",
In nc there isn´t obligation to add ":"
I updated sendChannelMessage and sendPrivateMessage functions to concatenate several args into one string before sending.


[R][C++] segfault when writing to ParquetFileWriter after closing

Describe the bug, including details regarding any error messages, version, and platform.
Writing to a closed writer causes a segfault rather than an error. I ran into this while testing something unrelated, evaluating portions of a larger script in a REPL. Writing to a closed output errors as expected so the key here is writer$Close() and the subsequent writer$WriteTable call:
library(arrow)

outfile <- tempfile(fileext = ".parquet")
sink <- FileOutputStream$create(outfile)

my_schema <- schema(letters = string())
writer <- ParquetFileWriter$create(
  schema = my_schema,
  sink,
  properties = ParquetWriterProperties$create(
    column_names = names(my_schema),
    compression = arrow:::default_parquet_compression()
  )
)
tbl_arrow <- as_arrow_table(data.frame(letters=LETTERS), schema = my_schema)
writer$WriteTable(tbl_arrow, chunk_size = 1)

writer$Close()
sink$close()

tbl_arrow <- as_arrow_table(data.frame(letters=LETTERS), schema = my_schema)
writer$WriteTable(tbl_arrow, chunk_size = 1)

Result:
 *** caught segfault ***
address 0x0, cause 'invalid permissions'

Traceback:
 1: parquet___arrow___FileWriter__WriteTable(self, table, chunk_size)
 2: writer$WriteTable(tbl_arrow, chunk_size = 1)
An irrecoverable exception occurred. R is aborting now ...
fish: Job 1, 'Rscript arrow_memorypool_crashe…' terminated by signal SIGSEGV (Address boundary error)

OS/arch: macOS 14.0 (Sonoma), aarch64 (M2)
R: 4.3.1
arrow version: <IP_ADDRESS>

Component(s)
R

Have replicated this on Ubuntu 23.04 with R 4.3.1 and arrow <IP_ADDRESS> too.  Here's gdb output:
Thread 1 "R" received signal SIGSEGV, Segmentation fault.
parquet::ParquetFileWriter::properties (this=0x5555563a04b0)
    at /home/nic/arrow/cpp/src/parquet/file_writer.cc:663
warning: Source file is more recent than executable.
663	  return contents_->properties();

Hi @thisisnic and @amoeba
I can take this ticket if it's unassigned. I'm running local tests to confirm the fix.

Yep, nobody's assigned and there's no open PR, so it's free.  Thanks @quanghgx!


1 QR scan button for all

Make all the QR Scan buttons in AlphaWallet with multi-functions, the same button for:

wallet/contract address, Ask user want to transfer or add a new contract
EIP67 681 831 URL/URI, go to transfer flow
MagicLink, go to MagicLink flow
normal URL, open in DApp Browser

Currently, there are 3 ways to open QR Scanner:

The + button in wallet page
The camera button in send page
The text button in DApp Browser

I like this, but have a dedicated icon next to the '+', something like [+] turned 90 degrees which you see as a standard scan icon. Then do a response as detailed above.

What about this?

Shall we start a new thread in the forum and come back after we finalise it?
(I think James Brown commented that users don't understand what the QR code icon means, which is why we switched to the camera).

Yes, good idea.

This looks good. Do you think the icons stand out more in that 'holo' colour or have them as standard 'light' mode which is dark grey.

 As a user engaging with other wallets/services, I want to quickly access a QR code scanner, so that I can conduct business more readily/intuitively.

Duplicate of #1723


Add instruction for running

Add npm run build to the instructions

Sorry, it should be npm run build, not npm build


Refactor get program definition to throw an exception when not found

Description
After this change ProgramService#getProgramDefinitionAsync may throw exceptions after the future returned completes. The exception can propagate if the future is concatenated with other completable stages.
Checklist

[ ] Created tests which fail without the change (if possible)
[ ] Extended the README / documentation, if necessary

Issue(s)
[Fixes] #<issue_number>

(blocking) the ApplicantProgramsController.edit uses ApplicantService and needs to use exceptionally. There may be others.

Working on this :)


Add kubelogin to the Docker file

To access Private AKS and missing features from Kubectl can we add https://github.com/Azure/kubelogin, please
PR: https://github.com/aztfmod/rover/pull/305

hi @nusrath432, are you able to submit a PR or want me to add to team backlog? :)

@arnaudlh only if you hiring me full-time ;)
https://github.com/aztfmod/rover/pull/305

@arnaudlh only if you hiring me full-time ;)
PR: #305

you overestimate my power @nusrath432 ;) but thanks for the PR :)


Is Ukrainian to ASCII correct?

Example: Is this correct?
"діти" -> "diti"

Can a native speaker, please check the character-replacement, thanks.
https://github.com/voku/portable-ascii/blob/master/src/voku/helper/data/ascii_by_languages.php#L538

Hello, the correct transliteration for "діти" is "dity". Please check my PR with updates according to the official Ukrainian transliteration docs https://zakon.rada.gov.ua/laws/show/55-2010-п?lang=en#Text
#59

Also, I have a question.
In Ukrainian for some letters ('ю', 'я', 'Ї', etc) transliteration is different if it is the first letter in a word.
For example:
'яма' -> 'yama'
'пісня' -> 'pisnia'
The letter 'я' may be 'ya' or 'ia' depending on the letter position.
Do you think it's possible to implement this feature? May it be useful for other languages?


[WIP]:feat(firebase_auth): v1 rework

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.4 out of 5 committers have signed the CLA.:white_check_mark: Salakar:white_check_mark: Ehesp:white_check_mark: kirstywilliams:white_check_mark: helenaford:x: greghespYou have signed the CLA already but the status is still pending? Let us recheck it.


test: enable SeedPeersTest and add MasternodeSeedPeersTest

Signed-off-by: HashEngineering<EMAIL_ADDRESS>

Issue being fixed or feature implemented

Enable some unit tests, now that there are seedPeers specified in NetworkParameters.
What was done?

How Has This Been Tested?

Breaking Changes

Checklist:

[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have added or updated relevant unit/integration/functional/e2e tests
[ ] I have made corresponding changes to the documentation

For repository code-owners and collaborators only

[ ] I have assigned this pull request to a milestone

Codecov Report

Merging #145 (9b5cd23) into master (492ae1a) will decrease coverage by 0.00%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master     #145      +/-   ##
============================================
- Coverage     43.64%   43.64%   -0.01%
- Complexity     4913     4918       +5
============================================
  Files           354      354
  Lines         39287    39287
  Branches       5376     5376
============================================
- Hits          17148    17147       -1
+ Misses        20427    20423       -4
- Partials       1712     1717       +5

Impacted Files
Coverage Δ

.../java/org/bitcoinj/crypto/KeyCrypterException.java
50.00% <0.00%> (-16.67%)
:arrow_down:

...main/java/org/bitcoinj/core/PeerSocketHandler.java
51.08% <0.00%> (-7.61%)
:arrow_down:

...src/main/java/org/bitcoinj/net/BlockingClient.java
75.86% <0.00%> (-6.90%)
:arrow_down:

...main/java/org/bitcoinj/core/TxConfidenceTable.java
72.13% <0.00%> (-1.64%)
:arrow_down:

...in/java/org/bitcoinj/core/TransactionOutPoint.java
72.22% <0.00%> (-1.39%)
:arrow_down:

...e/src/main/java/org/bitcoinj/core/BloomFilter.java
74.65% <0.00%> (-1.37%)
:arrow_down:

...c/main/java/org/bitcoinj/net/NioClientManager.java
60.86% <0.00%> (-1.09%)
:arrow_down:

core/src/main/java/org/bitcoinj/core/ECKey.java
59.05% <0.00%> (-0.44%)
:arrow_down:

...main/java/org/bitcoinj/core/NetworkParameters.java
58.42% <0.00%> (+0.56%)
:arrow_up:

...ain/java/org/bitcoinj/net/discovery/SeedPeers.java
66.66% <0.00%> (+66.66%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 492ae1a...9b5cd23. Read the comment docs.

Based on Jira comments, this has been approved.


Add telemetry module to RPM release script

This checklist is used to make sure that common guidelines for a pull request are followed.

[ ] The PR has modified HISTORY.rst describing any customer-facing, functional changes. Note that this does not include changes only to help content. (see Modifying change log).

[ ] I adhere to the Command Guidelines.

Merge this without approval to prepare for the release.


Update documentation to describe the various outputs

Should have sections to describe the results/ folders:

[x] anatomicals
[x] seg folder
[x] surf folder
[ ] transforms

text descriptions for all (except transforms) now added in PR #28

text descriptions for all (except transforms) now added in PR #28

closing this, since PR #28 merges in these changes (will leave transforms for another issue in future)..


Tacotron plus World vocoder

Hey I am glad to inform you that I have succeeded to merge Tacotron model with World vocoder and generated some evaluation results as follows. The results sound not bad but still not perfect. However it shows another way to train different feature parameters with Tacotron. The World vocoder is an open source project and thus everyone can use it for all. Moreover the quality of resynth results from that vocoder is better than that from Griffin-Lim since the three features (lf0[1], mgc[60] and ap[5]) contain not only magnitude spectrograms but also phase information. Furthermore the depth of the features is low enough that we do not need postnet for Tacotron model. The performance of training can be reduced to 0.7 second per step. The inference can also be quick enough even it only works on CPU. So it really worthes trying.
I would like to share my experimental source code with you as follows. Note that it currently only for Chinese mandarin. You may modify it for other languages:
tacotron-world-vocoder branch
Python-Wrapper-for-World-Vocoder
pysptk merlin-world-vocoder branch
By the way you need use python setup.py install and the copy the so file manually into the system path for pysptk and python wrapper project.
Besides I also would like to provide two Python scripts for World vocoder resynth test.
world_vocoder_resynth_scripts.zip
world_vocoder_demo.zip

I have modified something on both World vocoder and pysptk to make them work on this project. And I have used merlin tool scripts for feature extraction and resynth. It seems the latest version of World vocoder failing to work on this issue. So I open my modified forks here so that everybody can try them.

Note that it currently only for Chinese mandarin. You may modify it for other languages

What would need to be changed for it to work with English? I'm interested in trying it but I don't know anything about creating neural networks. I know just enough to use them. If it's too complicated, are you planning to have it work with English later on?

@isaiah658 You can do it on your own. That is not tough.
First revert the symbols.py
https://github.com/begeekmyfriend/Tacotron-2/commit/6abba13160e0a09bba248cb71119b3681e61996c#diff-7163c9c1f4130c23f589899e114afdbd
Second change the cleaner in hparams.py
cleaners='english_cleaners',

Third choose LJSpeech as the option for train.py.

@begeekmyfriend I appreciate the help! Unfortunately I'm having trouble compiling pyworld. I'm clueless when it comes to compiling stuff, but I think I managed to compile pysptk. It made the .so file. Pyworld is proving more difficult though.
First it wants the "world/synthesis.h" from the world project. So I paste in the world folder with the header files. Then all the other header files refer to "world/macrodefinitions.h" and a couple other files so I remove the "world/" so they refer to the local folder. After that compiling gives several errors starting with pyworld.cpp:3880:33: error: invalid conversion from CheapTrickOption* to int. It also says there are too few arguments on line pyworld.cpp:3880:49. After that it says D4C_coarse and Synthesize were not declared in scope.
Keep in mind that I am using your modified world vocoder project as you said something in a different comment that it won't work with the latest from mmorise.
I'm probably doing something wrong. Please help me if possible. Thanks!

git clone https://begeekmyfriend/pysptk.git
cd pysptk
git submodule update --init
sudo setup.py install
sudo cp build/lib.../...so       /.../python3.6/site-package/pysptk

@begeekmyfriend Thank you very much! I had to make some adjustments in the ljspeech.py file as well. I just looked at the thchs30.py file and copied the needed parts to ljspeech.py. Training is currently working! So after training is done, do I still use the demo_server.py like in the keithito tacotron?

You might try both demo_server.py and eval.py. By the way, I have found that the effect on T2 works even better.

How many steps did it take you to get an alignment with tacotron plus world vocoder?

Well the composition of T1 + World has some problems in fact for when it goes over 100k steps, the loss begin to rise. I have no idea about such issue. So I just take out the results from that under 100k steps. For T2 + World the loss curves seem normal and the results sound better.
biaobai_xizang_48000.zip

Unfortunately after 20k training steps the alignment is not improving. It's the same as it was at 1k steps. It's a bunch of green at the top of the graph. I think it's because my dataset is too small and it's actual speech. It's not someone reading lines from a script. When someone reads a script they generally focus more on enunciation than someone who is just talking. The dataset only has 50 minutes of audio.
It would be interesting to try training on an already trained English speaking model. I've been able to get  good results with keithito's tacotron by training on top of the ljspeech model. Training on top of another model also saves me a lot of time because I don't have a GPU to train with.
Just wanted to share my results. I wasn't expecting it to work with my dataset, but I wanted to try it out and see what would happen.

50 min is too little. 5~10 hour is the least.

Great work @begeekmyfriend . i was wondering that if we want to try WORLD vocoder for tacotron-2 on LJSpeech. What would be the necessary changes for the current Tacotron-2 + WORLD implementation of yours.
And if we fine tune it for speaker adaptation, how efficient will be the results compared to the weird robotic sound it gives on the LJ Dataset or any other speaker

@khanuzair680 See https://github.com/keithito/tacotron/issues/241#issuecomment-450536544

Thank but I am not able to create or find a ".so" file from the python wrapper or pysptk. I was wondering that whether it was created for python 2.7. I have tried using setup.py an its exiting with an error. I am using following commands:
git clone https://begeekmyfriend/python-wrapper... .git
cd python-wrapper
git submodule update --init
sudo setup.py install(error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
)(isnie build file:  lib.linux-x86_64-2.7  temp.linux-x86_64-2.7
)
sudo cp build/lib.../*.so       /.../python3.6/site-package/pysptk
cp cannot stat: No such file or directory
For pysptk, it is unable to run setup.py. returning with import error on numpy
It would be wonderful if you could guide me in the right direction

@khanuzair680
python setup.py install

@begeekmyfriend I ask one more question. In the 'eval_interval' process, the following error occurred.
TypeError: mgc2sp() got an unexpected keyword argument 'order'
In your 'datasets/audio.py', please check the function 'sp_denormalize()'.
Checking the pysptk documentation, the 'mgc2sp()' function does not have an order parameter.

I am facing the similar issue and i am following the instructions for installing python world wrapper and pysptk. Can you guide me a little. I have used all the options deleted and reinstalled pyworld and pysptk

@khanuzair680 Please check out the right branch mentioned on the 1st floor.

@begeekmyfriend Python-wrapper-for-world-vocoder branch is taken from merlin-world while the pysptk is from merlin-world-vocoder. Am i missing something?

Try git update --init for pysptk?

I am trying the following:
git clone https://github.com/begeekmyfriend/pysptk.git
cd pysptk
git submodule update --init
python setup.py install
sudo cp build/.../pysptk/*.so ../anaconda3/.../site-package/pysptk

I have tried uninstall and remove the respective file from python packages an reinstalling them and after that cloning the .so files into them through git process

I am installing pyworld through pip separately, and then copying .so into them after cloning your repository. Is this the right way @begeekmyfriend

It started working after some tweaks, great work @begeekmyfriend.

@begeekmyfriend It seems not word well for 16kHz audio? Have you try 16kHz audio?

it seems that in your https://github.com/begeekmyfriend/Tacotron-2/tree/mandarin-world-vocoder, the hparams.py file does not have cbhg related parameters. Does i did something not correct?

@begeekmyfriend Thank you very much for your hard work. I have a question about hyper parameters in training. When 'num_bap' is set to 5, an error occurs in the model initialization process because 'bap depth' of the preprocessing result is 1025. The error message is as follows.
ValueError: Cannot feed value of shape (32, 1438, 1025) for Tensor 'datafeeder/bap_targets:0', which has shape '(?, ?, 5)'
Is it possible to set 'num_bap' to 1025?
I would appreciate a more detailed explanation of 'bap'. ; )

I've meet the same problem. How did you solve the problem? set 'num_bap' to 1025?

@begeekmyfriend  i also found the training time became much more longer. I haven't changed anything but just replace the data.

Hi everyone, I have upgraded WORLD vocoder into the latest version where we can use havest instead of dio for F0 pitch extraction. The link adress is still on the 1st floor. Any suggestion is welcome! I am now using T2 for mandarin only.

@hyzhan See https://github.com/mozilla/TTS/issues/9#issuecomment-473743232

@begeekmyfriend What is the havest? If I want to apply the latest version, do I need to reinstall pyworld ?

@herenje Of course, please reclone the link on the links on the 1st floor and the resynth scripts are provided as well.

Focus on T2 currently. Any feedback is welcome https://github.com/begeekmyfriend/Tacotron-2/commit/e40a7b73ac31d299d731439fbabe8921b231a739

@begeekmyfriend
did you succeed in aligning your attention using world vocoder on T2?

Of course @herenje

I forgot to tell you that for differnt dataset, adjust the hp.max_frame_num and hp.max_text_length adopted by guided attention as the alignment slope for better convergence.

@begeekmyfriend Thank you. One more question. Do I need to install your pyworld and pysptk repository for using the latest T2 repository?

@herenje You might install the upstream repo. It works for me.

@begeekmyfriend I will try to learn LJSpeech data again in your latest tacotron2-world-vocoder repository. Can you tell me what variables I need to care about in hparams? Thank you.

num_mgc = 60
num_lf0 = 1
num_vuv = 1
num_bap = features.shape[1] - num_mgc - num_lf0 - num_vuv

@begeekmyfriend I am trying to learn LJSpeech data. It is going well, but attention graph seems to be not well aligned. The parameters I used are followings:
	num_mgc = 60
	num_lf0 = 1
	num_vuv = 1
	num_bap = 2
	frame_period = 8

I know that this may require more learning time. Could I get an answer from the parameters? Any comment please let me know :) I always appreciate your help.

Please adapt (reduce) your max_text_length and max_frame_num for the best ratio of N:T.

@begeekmyfriend Considering the attention graph, the appropriate ratio is 1:2. Is it right?

The radio is depends on your dataset.

@begeekmyfriend OK. Thank you. I will try it again :)

@begeekmyfriend Attention is aligned very quickly. But it still sounds squeaky? shrieking? in learning LJSpeech dataset. Can I hear a sample of the results you learned from mandarin data?

The demo has been shown on the floors where there is spectrogram graph. Maybe I need to reduce the frame period to obtain better quality. However I was told that the fidelity of sample is not as good as that from G&L. The quality is better indeed.

By the way when you want to hear complete synthesized samples, please wait until the stop token loss reduced to zero.

@begeekmyfriend Okay. I have just passed 5,000 steps. I will wait and share the results with you. By the way, when I heard it again, the results of mandarin demo are really good !!! :)

Typically female corpus performs better than male one both under WORLD and G&L

Hi, @begeekmyfriend , long time no see!
I heard that LieHuXingKong is using the same way as you did to do the mandarin TTS. I have several questions about the T2+WORLD framework. Could you please give me some word?
First, How's your inference speed of the T2+WORLD framework?
Second, Is there still miss pronunciation or repeat pronunciation exists in the T2+WORLD outputs?
Third, Did you try to use WaveNet condition on world features? And then combine the Tacotron2 with WaveNet to make vocoder much more natural and fast?
Please forgive me for asking so many questions at once. We are also considering to do the same things as you did, but we need some survey indeed.

I have no contact with 猎户星空 Maybe there is coincidence.
I have added a rough inference benchmark for WORLD branch https://github.com/begeekmyfriend/Tacotron-2/commit/70c5a62d2e7c6d64991b50bfa3623adde8df7899. Please check it out on your own machine.
In my test mis pronunciation depends much on your dataset and transcript. If you were using Biaobei corpus, I do not think you would run into such problems.
Wavenet is too slow for me but it might works well for developers possessing bundles of GPUs.
Anyway, thank you for question though I do not remember you through those issues...XD

Is there any comparison on the performance between GL and WORLD?

features

@begeekmyfriend
What's the meaning of " features.shape[1]"?
Thank you!

You might print features.shape and you will get it.

Thank you for your reply!

Is there any comparison on the performance between GL and WORLD?

@begeekmyfriend  Have you compared the performance?~

You might print features.shape and you will get it

@begeekmyfriend
my training data is 16k;
Do you think the hparams is ok?

@peter05010402 You need to adjust max_fram_num and max_text_length according to your dataset.

@begeekmyfriend  Thank you for your quick reply!!!

Here is biaobei mandarin demo from T2 + WORLD. The f0 feature value prediction is tough for this model.
xmly_biaobei_world.zip

Here is biaobei mandarin demo from T2 + WORLD. The f0 feature value prediction is tough for this model.
xmly_biaobei_world.zip

@begeekmyfriend
It sounds good!
but How  to reduce squeaky and  shrieking?

Separate lf0 loss and x10

biaobai_xizang_48000

Hi, @begeekmyfriend
I found this demo sounds best, it's better than xmly_biaobei_world.zip from T2 + WORLD. Do you train your model only with BiaoBei dataset or some private dataset?  Any comment please let me know, Thank you very much!

Give up this solution and turn to WaveRNN. Feel free to reopen this issue. The accuracy of F0 feature is hard for prediction.


Code refactor and cleanup

Refactor some API files to prevent possible empty string errors or NullReferenceEceptions. Created separate files for several classes that were nested inside of the files for other classes. This fixes issue #22 and issue #23

Hi @joshman1019 and @Mirch . If I may suggest a possible improvement for readability, I would suggest using guard clauses for failing fast (thus avoiding else clauses and nested ifs where possible) and keeping the identation at the lowest possible level.
In this case, this could be like:
if (!response.IsSuccessStatusCode)
{
    throw new Exception();
}

var stringResult = await response.Content.ReadAsStringAsync();

if (string.IsNullOrEmpty(stringResult))
{
    throw new Exception();
}

Yes, I agree. I see that you have already made these changes. Thanks!

Yes, I agree. I see that you have already made these changes. Thanks!

That's how I wrote it for the issue I was working on, all other service classes have not been changed yet.

Yes, I agree. I see that you have already made these changes. Thanks!

That's how I wrote it for the issue I was working on, all other service classes have not been changed yet.

Alright I'll get on submitting a pull request with the remainder.


How azure public IP route preference related azure router server?

There is very detailed explanation about azure public IP routing preference in below guide
https://learn.microsoft.com/en-us/azure/virtual-network/ip-services/routing-preference-overview
It is more like a public IP routing selection feature. How does it related with azure router service?

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 830d6000-cba5-c907-b31b-8b2cac930932
Version Independent ID: 91276ed3-b4c6-8823-c3d6-b318a6dab70c
Content: Path selection with Azure Route Server
Content Source: articles/route-server/path-selection.md
Service: route-server
GitHub Login: @halkazwini
Microsoft Alias: halkazwini

@yinoa
Thanks for your feedback!
I've delegated this to content author @halkazwini, who will review it and offer their insightful opinions.

Hi @yinoa
Routing preference and Azure Route Server are two different things.
Route Server simplifies dynamic routing between your virtual network gateway and network virtual appliance (NVA) in a virtual network (so you don't need to use and maintain user-defined routes).
Routing preference is about to use Microsoft global network vs public internet for traffic between Azure resources and your on-premises.

#please-close


build: replace formatter and impsort with spotless

One plugin to rule them all.
Follow-up:
https://github.com/TimefoldAI/timefold-solver-enterprise/pull/64

@triceo, it looks like we need to add the goal check to bind it to the Maven verify phase.

@triceo, it looks like we need to add the goal check to bind it to the Maven verify phase.

Please ignore my comment. The verify phase will also run the goal apply.


Notify HttpSessionListeners and HttpSessionActivationListeners when loading a session from memcached

From martin.grotzke on February 19, 2010 15:07:39
The servlet spec #SRV.7.7.2 "Distributed Environments" says:
"Containers must notify any session attributes implementing the
HttpSessionActivationListener during migration of a session. They must
notify listeners of passivation prior to serialization of a session, and of
activation after deserialization of a session."
This semantics should be supported by the msm-javolution-serializer.
Original issue: http://code.google.com/p/memcached-session-manager/issues/detail?id=38

From martin.grotzke on February 19, 2010 12:51:27
Changed title as this is independent of the actual serialization strategy.
Summary: Notify HttpSessionActivationListeners when loading a session from memcached

From martin.grotzke on February 19, 2010 12:54:48
Passivation does not occur as (right now) sessions are still kept in the local session
map, only activation has to be supported (in the case that a session is loaded from a
tomcat taking over the session from a failed tomcat).

From martin.grotzke on April 01, 2010 00:23:42
Just as a personal note (stumbled over this right now):
https://issues.apache.org/bugzilla/show_bug.cgi?id=48717

From martin.grotzke on April 02, 2010 13:47:51
Fixed. Also javax.servlet.http.HttpSessionListeners are notified (via sessionCreated(HttpSessionEvent)) about the new
active session in this web application.
Summary: Notify HttpSessionListeners and HttpSessionActivationListeners when loading a session from memcached
Status: Fixed
Labels: Milestone-1.3


Add support to create edls.

there should be a config option where is used in process_to_db and a cli command to do them all.

Done! Use the chapter plugin to get the most out this!


Update document production

I updated the document production to use spring-doc-resources, which makes this doc look like Spring's other docs.

Thanks @Buzzardo for the PR. Manually applied the changes against the master branch and leaving the Einstein branch as is. Closing the PR.


TestTemplate.sh does not exit with a non-zero exit code when a test fails

On MacOS, when executing TestTemplate.sh against a template with a failed test (Fails:  1) the TestTemplate.sh file does not exit with a non-zero exit code.
I've also experienced this in a ubuntu docker container and a powershell (Ubuntu) docker container from Microsoft.

s/b fixed in #634 (and an adjacent push to master)


More updates to setup script experience

How to test (s/o @hugh for this):
curl -sL https://raw.githubusercontent.com/sublime-security/sublime-platform/josh.setup_script/install-and-launch.sh | remote_branch=josh.setup_script bash
Note that Github caches the raw content and only updates every `5 minutes. There doesn't seem to be a way to bypass this, except possibly by requesting the commit itself instead of the branch.
Local testing: clone_platform=false bash install-and-launch.sh

@hugh-sublime i got this merged today so i could run through an on boarding with someone, should be pretty straight forward but take a look when you can to be sure everything looks good 🙏

For some reason the original @hugh tag didn't message me in Slack but @hugh-sublime tagged me properly. Everything seems good but we may want to add some newlines in the places we're using printf instead of echo.

i think that original mis tag was my fault. 🙏 thank you! will update


[Hackathon team 2] Bobri dobri.

The PR has been made in order to monitor the process os BeCode hackathon DELL EMC Mongoose task completion for team "Bobri dobri".

Thank you for your help.


[build]: neon-build tool enhancements

[x] We need to delete all of the solution bin and obj folder before a full build.  Doing clean builds doesn't actually clean everything.  We've already run into a couple situations where left over files caused build problems.

[x] Add the neon-build executable to the $/ToolsBin folder and don't build it in the build scripts.

DONE


Fix typos

Just a couple of typos.

Awesome @iimog, that's amazing thanks! Next time you send a PR, consider adding yourself to the list of co-authors:
https://github.com/haesleinhuepf/BioImageAnalysisNotebooks/blob/f7659ddb59ec3afc00ef504b64ed3885a3241997/docs/_config.yml#L2
You curation contribution is definitely welcome!


Removed unused context variables in Page change_view().

Unused after 49c8d4c2905243fb33890f526c334bd39bb350f0 and
785b31a1a0879eddb6385bfba31cf7303c060239.

Coverage remained the same at 82.392% when pulling 9c7cb105c3be3a6d6338ee1837893fabcbd5516c on timgraham:unused-var into 3b43f39953fe50a30bae4c5dcacc2c564e1324b9 on divio:develop.

👍

Thanks @timgraham.
Will land this after 3.4 is released

@timgraham thanks, merging this